Category: Programming

This is one of those posts where I’m setting a marker for when the sh** hits the fan, so to speak. There are no deep dark secrets revealed here, though there is some geek talk after the jump. Don’t say you weren’t warned.

While working on a piece of code, I ran into an unexpected bug- in my program not Python. I’ve since done some investigating and I’ve learned something about how Python deals with class attributes and instance attributes. Following is a little exposition of what I’ve learned.

I have defined a rather simple class for an XML proxy server to facilitate interacting with a blog. Right now, I’ve implemented a WordPress version so I can work with my own blog. Theoretically, it should be possible to implement other classes specific to other blog types thus making my program more general, and useful to others.

What I hadn’t figured out was how to structure the code so as to minimize monkeying around with multiple files when adding the new class. My goal was to come up with a structure that simply called for adding a module to extend the functionality.

Today, I finally came up with something and it utilizes the dynamic module loading capabilities of python.

Having been using open source software for years now, I’ve never really done any thing to try and reciprocate, as it were. Well, I’ve finally gotten my chance and contributed a 3-line bug fix to the python-markdown package.

Here’s the fixi, marked for posterities sake.

I’ve been working on a piece of code to convert a blogpost into Markdown text. Yes, I’m aware of the html2text.py module (mine is html2md, so nyeah!) All I’ll say is how the heck does one learn anything if they keep relying on other people’s work?

Anyway, I’ve got a naive implementation working now (won’t handle more complicated nestings like lists in blockquotes) when I ran into a snag involving unicode. Upon retrieval of a particular post, I got the following error:

UnicodeEncodeError: ‘ascii’ codec can’t encode character u’xa0′ in position 174: ordinal not in range(128)

It came up in the context of passing a string to the iterparse function of the cElementTree module. The character in question is a ‘non-breaking space.’ Frankly, I wasn’t sure how it got there, but I verified it’s presence in the string and then set to figuring a way to deal with it. I believe this is an instance of mixing strings and unicode together, rather than dealing solely in one or the other.

I found the unicodedata module for a solution. There is a function call normalize which will map unicode characters to the local encoding. In this case, ASCII. Now this solution is far from perfect, since special characters (say, from the Russian alphabet or characters with tildes above them) are just converted to rough ASCII equivalents.

The following code additions fixed my problem:

import unicodedata as ud
    .
    .
    .
uthml = ud.normalize('NFKD', unicode(html))

Where html is a string of html directly from the website. I can then pass uhtml to the iterparse function and the error is gone (because the u’xa0′ characters are translated to ASCII space characters) and the rest of the program is able to do it’s thing.

I don’t know if this will be the final solution, but it allows me to continue with the development I had been originally interested in. I was aware of the potential for unicode issues, but had hoped they wouldn’t crop up. This gives me a simple 2 line way to deal with it for the now.

I’m in the process of learning how to use lxml- a fast, powerful XML parser for python that relies on libxml2. I don’t know much in the way of details regarding xml so I got stuck as soon as I got started.

I was passing straight markdown generated XHTML into the various xml parser methods and objects for lxml. All of them were dying with the same error. The only parser that worked was the HTML parser.

The xml parsers kept coughing up the following error:

lxml.etree.XMLSyntaxError: Extra content at the end of the document, line 1, column 222

I really had know clue as to what this might mean. Since the output was from markdown, I reasoned there was little likelihood of malformed XHTML or some such. Besides, I knew that it rendered fine on web pages. Liekly, there was some significant detail I was missing. Unfortunately, my lack of xml knowledge meant I didn’t have anything to fallback on for solving the problem.

Finally, I turned up a comment thread where I learned that xml documents require a “root element.” I had seen the term “root” but only had a vague notion of what it meant. Now, I know exactly what is meant.

Prior to passing the markdown string to the parser, I performed the following operation:

rootedhtml = "<post>%s</post>" % html

Where html was the markdown output. I then passed rootedhtml to the parser and it no longer chokes.

Now I can get back to solving my original problem.

I’ve incorporated markdown into my blog client and in the course of doing so, I saw that it could be extended. So I thought I’d give it a go and see if I could add a ‘strike’ extension to the markdown module.

It turned out to be almost trivially simple thanks to the documentation. I chose to use a double hyphen around a word to create the strike. I looked elsewhere and double tildes seems to be another nice way to do it. The code below shows it for the double hyphen but the code would be identical in either case except for the RE definition.

Here’s the code:

import re

import markdown
from markdown import etree

STRIKE_RE = r'(-{2})(.+?)\2'

class StrikeExtension(markdown.Extension):
    def extendMarkdown(self, md, md_globals):
        md.inlinePatterns.add('strike', markdown.inlinepatterns.SimpleTagPattern(STRIKE_RE, 'strike'), '>strong')

def makeExtension(configs = None):
    return StrikeExtension(configs = configs)

That’s it.

The SimpleTagPattern object is a general purpose object that’s part of the markdown library and it’s used to create text rules involving inline patterns such as emphasis or strong. That’s all my version of the strike rule is.

I implemented my extension as a module, so I had to put it in the ‘extensions’ directory of my markdown library. It’s also possible to just incorporate the extension in project code and make markdown aware of it, but I haven’t tried that yet.

I came up with another nice little piece of python code to solve a problem while working on my blog client.