Category: Python

I’ve recently attained a renewed interest in my blog client blogtool. A big part of that renewal is due to unfinished business- I’d alway meant to release it into the wild but had never taken the time to learn how to package it. I finally took that plunge a few weeks ago. Ever since, I’ve come up with a series of improvements, fine tunings and new ideas to make it a more capable tool and a better piece of software in general.

I haven’t touched the code for the blog client I’d written in quite awhile. This is largely because it works well for my purposes and I haven’t had the need to add further support for other features.

There has been one major shortcoming for it, however, that I hadn’t taken the time to investigate and correct. Often times, when quoting text from an article on the web, I would get a unicode decode error related to the blob of text I’d copied from the browser.

Now, I understood in general terms what the problem was: stray characters within the copied text were not ASCII characters and markdown chokes on those characters. I had an inelegant workaround that kept me from properly dealing with the problem: I’d scan the text for offending characters, typically punctuation, and replace them with reasonable ASCII equivalents. It was a pain, but it worked.

Like all workarounds, this method had limitations. Specifically, certain special letter characters like letters with umlauts, tildes, accent graves or accent aigus over them cannot be duplicated. The fact that I didn’t run into that problem a lot kept me from dealing with it quicker. Also, scanning a block of text for unicode violators is tedious.

What I failed to understand at the time was that the characters on a web page were encoded in some kind of format, like UTF-8 for example. For most of the alpha characters (those without umlauts and the like) UTF-8 and unicode are identical. The problem comes in when characters don’t line up so neatly. What I finally came to understand was that the encoded web page text needed to be decoded into unicode prior to processing. The concept seems so blisteringly obvious, now, that I’m actually perplexed as to how I never grasped it originally.

So I finally fixed the problem. Or, perhaps better put, I came up with a solution with a better set of trade-offs. Because in order to actually “fix” the problem, it would be necessary to always know how text had been encoded. Unfortunately, from the program’s perspective, it can’t be done.

But it can make some educated guesses.

Here’s the basic code that fixes the problem:

for encoding in ['ascii', 'utf-8', 'utf-16', 'iso-8859-1']:
    try:
        xhtml = markdown.convert(text.decode(encoding))
    except (UnicodeDecodeError, UnicodeError):
        continue
    except:
        print "Unexpected Error: %s\n" % sys.exc_info()[0]
        sys.exit(1)
    else:
        return helperfunc(xhtml)

In this case, markdown is an object for marking up markdown formatted text. Prior to passing the text to the markdown object, I decode it using encoding that represent the most likely encodings I’ll run into. If an encoding fails, that a UnicodeDecodeError will get raised, which is caught by the first except clause. That clause merely passes control back to the for loop where the next encoding is selected and tried. Rinse, repeat. When no exception is created, control passes to the else clause where normal program flow continues on the returned xhtml from markdown.

This section of code eliminates, in my case, almost all occurrences my afore explained unicode problems. But that’s because the vast majority of webpages I use are encoded using UTF-8. I’ve since added a command line option to specify the encoding to use for decoding purposes. This should provide a means to cover all other situations that arise. In this instance, when the user specifies the encoding on the command line, the user specification supersedes all other encodings and is used. The presumption is the user knows what they are doing.

The code to support that looks like this:

if charset:
    encodings = [charset]
else
    encodings = ['ascii', 'utf-8', 'utf-16', 'iso-8859-1']

for encoding in encodings:
 .
 .
 .

The rest of the code looks identical to the above snippet.

It was a good exercise for me to muddle through, as I now fully comprehend the unicode problems that can arise and how to deal with them. The basic rules are:

1. Decode text going into the program.
2. Encode text coming out of the program.
3. Use unicode for the string literals within the program.

These should help keep me out of unicode trouble in the future.

This is one of those posts where I’m setting a marker for when the sh** hits the fan, so to speak. There are no deep dark secrets revealed here, though there is some geek talk after the jump. Don’t say you weren’t warned.

While working on a piece of code, I ran into an unexpected bug- in my program not Python. I’ve since done some investigating and I’ve learned something about how Python deals with class attributes and instance attributes. Following is a little exposition of what I’ve learned.

I have defined a rather simple class for an XML proxy server to facilitate interacting with a blog. Right now, I’ve implemented a WordPress version so I can work with my own blog. Theoretically, it should be possible to implement other classes specific to other blog types thus making my program more general, and useful to others.

What I hadn’t figured out was how to structure the code so as to minimize monkeying around with multiple files when adding the new class. My goal was to come up with a structure that simply called for adding a module to extend the functionality.

Today, I finally came up with something and it utilizes the dynamic module loading capabilities of python.

Having been using open source software for years now, I’ve never really done any thing to try and reciprocate, as it were. Well, I’ve finally gotten my chance and contributed a 3-line bug fix to the python-markdown package.

Here’s the fixi, marked for posterities sake.

I’ve been working on a piece of code to convert a blogpost into Markdown text. Yes, I’m aware of the html2text.py module (mine is html2md, so nyeah!) All I’ll say is how the heck does one learn anything if they keep relying on other people’s work?

Anyway, I’ve got a naive implementation working now (won’t handle more complicated nestings like lists in blockquotes) when I ran into a snag involving unicode. Upon retrieval of a particular post, I got the following error:

UnicodeEncodeError: ‘ascii’ codec can’t encode character u’xa0′ in position 174: ordinal not in range(128)

It came up in the context of passing a string to the iterparse function of the cElementTree module. The character in question is a ‘non-breaking space.’ Frankly, I wasn’t sure how it got there, but I verified it’s presence in the string and then set to figuring a way to deal with it. I believe this is an instance of mixing strings and unicode together, rather than dealing solely in one or the other.

I found the unicodedata module for a solution. There is a function call normalize which will map unicode characters to the local encoding. In this case, ASCII. Now this solution is far from perfect, since special characters (say, from the Russian alphabet or characters with tildes above them) are just converted to rough ASCII equivalents.

The following code additions fixed my problem:

import unicodedata as ud
    .
    .
    .
uthml = ud.normalize('NFKD', unicode(html))

Where html is a string of html directly from the website. I can then pass uhtml to the iterparse function and the error is gone (because the u’xa0′ characters are translated to ASCII space characters) and the rest of the program is able to do it’s thing.

I don’t know if this will be the final solution, but it allows me to continue with the development I had been originally interested in. I was aware of the potential for unicode issues, but had hoped they wouldn’t crop up. This gives me a simple 2 line way to deal with it for the now.

I’m in the process of learning how to use lxml- a fast, powerful XML parser for python that relies on libxml2. I don’t know much in the way of details regarding xml so I got stuck as soon as I got started.

I was passing straight markdown generated XHTML into the various xml parser methods and objects for lxml. All of them were dying with the same error. The only parser that worked was the HTML parser.

The xml parsers kept coughing up the following error:

lxml.etree.XMLSyntaxError: Extra content at the end of the document, line 1, column 222

I really had know clue as to what this might mean. Since the output was from markdown, I reasoned there was little likelihood of malformed XHTML or some such. Besides, I knew that it rendered fine on web pages. Liekly, there was some significant detail I was missing. Unfortunately, my lack of xml knowledge meant I didn’t have anything to fallback on for solving the problem.

Finally, I turned up a comment thread where I learned that xml documents require a “root element.” I had seen the term “root” but only had a vague notion of what it meant. Now, I know exactly what is meant.

Prior to passing the markdown string to the parser, I performed the following operation:

rootedhtml = "<post>%s</post>" % html

Where html was the markdown output. I then passed rootedhtml to the parser and it no longer chokes.

Now I can get back to solving my original problem.

I’ve incorporated markdown into my blog client and in the course of doing so, I saw that it could be extended. So I thought I’d give it a go and see if I could add a ‘strike’ extension to the markdown module.

It turned out to be almost trivially simple thanks to the documentation. I chose to use a double hyphen around a word to create the strike. I looked elsewhere and double tildes seems to be another nice way to do it. The code below shows it for the double hyphen but the code would be identical in either case except for the RE definition.

Here’s the code:

import re

import markdown
from markdown import etree

STRIKE_RE = r'(-{2})(.+?)\2'

class StrikeExtension(markdown.Extension):
    def extendMarkdown(self, md, md_globals):
        md.inlinePatterns.add('strike', markdown.inlinepatterns.SimpleTagPattern(STRIKE_RE, 'strike'), '>strong')

def makeExtension(configs = None):
    return StrikeExtension(configs = configs)

That’s it.

The SimpleTagPattern object is a general purpose object that’s part of the markdown library and it’s used to create text rules involving inline patterns such as emphasis or strong. That’s all my version of the strike rule is.

I implemented my extension as a module, so I had to put it in the ‘extensions’ directory of my markdown library. It’s also possible to just incorporate the extension in project code and make markdown aware of it, but I haven’t tried that yet.

I came up with another nice little piece of python code to solve a problem while working on my blog client.