Introducing uberparse

Overview

uberparse is a Python library which reduces the burden of handling command line and configuration file parsing for modules and applications, making it trivial for imported modules to recursively expose configuration options and command line arguments.

In essence, uberparse is a thin wrapper around ConfigParser and optparse, but it presents them in a coherent and very simple way. Options are defined as attributes of a class, and a single function call transparently handles all of the configuration and argument parsing.

A simple example

Below is the simplest possible uberparse example:

import uberparse

class ModuleFlags(uberparse.FlagClass):
test_flag = dict(default="value", help="a test flag", short="-t")

options, args = uberparse.ParseOptions()

print "Test flag:", options["test_flag"]

This gets you the following for free:

$ ./test.py --help
Usage: test.py [options]

Options:
-h, --help show this help message and exit
--help-application
--help-builtin

Options for application:
-t TEST_FLAG, --test_flag=TEST_FLAG
a test flag

Options for builtin:
-c CONFIG, --config=CONFIG
path to configuration file
--genconf dump configuration options and exit

$ ./test.py --test_flag=hello
Test flag: hello

$ ./test.py -t hello
Test flag: hello

It’s also trivial to create custom configuration files:

[application]

test_flag = config value

Builtin flags

You get –help, –help-xxx, –config, and –genconf options for free when using uberparse. The more precise –help options are useful for managing very large, nested sets of flags. The –genconf option will dump a valid configuration file with the current options, comments and values to the console.

Settings

The operation of uberparse can be configured on a per-application basis using another settings class. For example, you could configure the default config file using the following in your module:

class FlagSettings(uberparse.SettingsClass):
config = "test.ini"
include = ('some_test_module',)

Any class name can be used, so long as it inherits from uberparse.SettingsClass.

Module flags

As mentioned briefly earlier, any imported modules will also have their flags parsed. The unified configuration file is capable of configuring options from imported modules as well as the main application. Flags are defined in modules the same way they are defined in the main application.

The include setting from SettingsClass can be used to selectively import flags from specific modules. The default behavior is to use flags defined in any imported module.

Gimme!

You can grab the source code and a more comprehensive example here.

Two logs, one logger.

Though the Python logging documentation doesn’t make it entirely clear, it is possible to log to two different places with a single logger object. It’s even possible to dynamically adjust the log level of each output handler.

The trick is to get a handle on a single logger (you can use a named logger if you don’t want to use “root”. We’ll kick off the process by making sure we have the logging module imported.

import logging

We must set the root logger level to at least as low as all other loggers or it will choke the logging of the other handlers.

root_logger = logging.getLogger("")
root_logger.setLevel(1)

You can then add as many log handlers to that logger as you like, each with its own log level.

file_logger = logging.FileHandler(....)
file_logger.setLevel(logging.DEBUG)
root_logger.addHandler(file_logger)

console_logger = logging.StreamHandler()
console_logger.setLevel(logging.INFO)
root_logger.addHandler(console_logger)

Then elsewhere in your code, you simply use:

logging.info("This will go to both handlers.")
logging.debug("This will only go to the file logger.")

Hopefully that’s been educational. A working example can be found in my boilerplate repository as usual.

Overriding module functions in Python

Overview

Today I will describe a way of overriding functions from imported modules in Python which can be useful for profiling function calls or changing the functionality of a function you’re using without modifying the original source code.

We will be using a real world example for this tutorial. Let’s say you want to keep a count of the number of queries run by SQLObject. This is simple if you’re always calling the query function directly. But if the function is being called deeply within the module (as is the case with SQLObject) you have to get more crafty.

The problem

Let’s say you have the following bit of code:

import sqlobject

class SomeTable(sqlobject.SQLObject):
  some_column = sqlobject.StringCol()

for i in xrange(0, 5):
    s = SomeTable(some_column="Test %d" % i)
    print s.some_column

We know that somewhere behind the scenes, SQLObject is running real SQL queries, but we’re not using that code directly, nor is it presented to us. It would be ideal if SQLObject kept an internal query counter, but it doesn’t so we have to be enterprising and find a way to do it ourselves.

A little bit of digging through the SQLObject code leads us to an interesting bit of code in sqlobject.dbconnection:

class DBAPI:
    def _runWithConnection(self, meth, *args):
        conn = self.getConnection()
        try:
            val = meth(conn, *args)
        finally:
            self.releaseConnection(conn)
        return val

This code gets called each time SQLObject executes a query.

The solution

So we’ve found the code, but now we need a way of keeping track of the number of calls to that function. One way to do this is to simply override that attribute with one that is more to our liking. First let’s see what such a function would look like:

QUERY_COUNT = 0

original_function = sqlobject.dbconnection.DBAPI._runWithConnection

def wrapper(self, meth, *args):
    global QUERY_COUNT
    QUERY_COUNT += 1
    return original_function(self, meth, *args)

sqlobject.dbconnection.DBAPI._runWithConnection = wrapper

This works for most cases, but what if there were subclasses of DBAPI that had their own _runWithConnection methods? We need a way to replace those as well.

You can probably already see where this is going. We need to get all subclasses of the DBAPI class and recursively all of their subclasses, then replace the _runWithConnection method in each of them with our wrapper function.

The roadblock

This is relatively trivial with classes that inherit from ‘object’ (also known as ‘2.4 style’ classes), unfortunately DBAPI is not one of them! To make a long story short, things quickly get more complicated than they need to be. Wouldn’t it be nice if someone had written a module that could do this for us?

Introducing wraptools

You may be pleased to know that someone has in fact created such a module. That person happens to be me, which is why it’s being plugged here 🙂

Let’s go back to the original solution and modify it slightly to use wraptools.

from wraptools import wraps

QUERY_COUNT = 0

@wraps(sqlobject.dbconnection.DBAPI._runWithConnection)
def wrapper(original_function, self, meth, *args):
    global QUERY_COUNT
    QUERY_COUNT += 1
    return original_function(self, meth, *args)

So in this example @wraps does all of the dirty work of replacing every instance of the function (including subclasses) via the sweet syntactic sugar of a decorator. The original function and any arguments it was called with gets passed along to your wrapper, allowing you to munge the input or output to your liking.

Putting it all together

import sqlobject
from wraptools import wraps

class SomeTable(sqlobject.SQLObject):
  some_column = sqlobject.StringCol()

QUERY_COUNT = 0

@wraps(sqlobject.dbconnection.DBAPI._runWithConnection)
def wrapper(original_function, self, meth, *args):
    global QUERY_COUNT
    QUERY_COUNT += 1
    return original_function(self, meth, *args)

for i in xrange(0, 5):
    s = SomeTable(some_column="Test %d" % i)

print "Total queries:", QUERY_COUNT

And there you have it: An elegant, simple way of profiling SQLObject queries! This concept can be easily extended to profile and override SQLAlchemy as well as most other Python modules.

You can access the module documentation here.

The damn code

If you’re psyched, you can grab the source code from my code repository.

Easy GUI applications with Python, GTK+ and Glade

Foreword

This tutorial is not intended to be a detailed guide on using Glade or programming in Python, but rather an introduction to using GTK+/Glade in conjunction with Python to create elegant, maintainable GUI applications for the Gnome desktop.

This guide assumes the capability to hunt down and install packages on your distribution of choice as well as some degree of development experience in Python.

If you want to create GUI driven applications, it’s in your best interest to use an interface builder. For GTK, the best such tool is Glade. Glade is a WYSIWYG interface builder which generates XML files describing the user interface and callbacks for your application. This allows you to completely separate your view (user interface) from the rest of your code and focus on the underlying logic of your program rather than needlessly adjusting the padding of a widget for hours. In most cases, you won’t have to write user interface drawing code at all.

Let’s get started.

Prerequisites

  • Python (2.4 or higher recommended)
  • Python Glade library (python-glade2)
  • Glade interface builder, Glade3 recommended (glade-3)

Building the user interface

Open the Glade interface builder and familiarize yourself with the interface. The ‘Palette’ contains all of your toolbox options, analogous to brush tools in GIMP. The ‘Properties’ panel is your one stop shop for adjusting metadata about your widgets such as display options (‘General’, ‘Common’) and callbacks (‘Signals’). Finally, the main panel is the canvas on which you will be drawing your application GUI.

For this example, we’ll be creating a window with a single button that says “Hello, world!” Pay special attention to the ‘Name’ properties of each widget, as we will be referencing them from the Python code, and they must correlate.

Step one: In the Palette, click the ‘Window’ icon under ‘Top Levels’. It’s the one that looks like a blank window, but you can also mouse over each icon to see its description.

Palette window

This will add a checkered box to your canvas.

Canvas - Window

Step two: In the properties window, set the ‘Window Title’ to what should appear in the title bar (in this case, we’ll use “Hello, world!”) and change the ‘Name’ to ‘main_window’

Properties window

Step three: Switch to the ‘Signals’ tab in the Properties window and set up a handler for the ‘destroy’ signal of the main window. Set the handler to ‘on_main_window_destroy’. This signal will fire when the user closes the application window.

Main window signals

Step four: Back in the Palette, click the ‘Button’ icon under ‘Control and Display’ and then click anywhere in the gray checkered window we created a moment ago. In properties, change the Label to ‘Hello, world!’ and the Name to ‘hello_button’

Canvas
Button properties

Step five: Add a handler for the ‘clicked’ signal of the hello button. Name the callback ‘on_hello_button_clicked’. This signal will fire whenever the user clicks on the button.

Button signals

Now just save the project as ‘gtk_hello_world.glade’ and exit the application. Congratulations, you’ve successfully built a GUI for your application. Give yourself a pat on the back.

Writing the code

The code is rather simple, we don’t have to mess with any specifics about drawing the widgets, we just tell them when to display and what to do when different events are called. If you don’t understand how classes in Python work, go read up on them before continuing.

import gtk
import gtk.glade


GLADE_FILE = 'gtk_hello_world.glade'


# A decorator that defines a function as a callback to a Glade signal
def glade_callback(function):
    function.__glade_callback__ = True
    return function

# Retrieves a dictionary of functions decorated as Glade callbacks
def get_glade_callbacks(input):
    attributes = dir(input)

    result = {}
    for key in attributes:
        attr = getattr(input, key)
        if callable(attr) and hasattr(attr, '__glade_callback__'):
            result[key] = attr

    return result


class HelloWorld(object):
    def run(self):
        self.interface = gtk.glade.XML(GLADE_FILE)

        self.callbacks = get_glade_callbacks(self)
        self.interface.signal_autoconnect(self.callbacks)

        self.main_window = self.interface.get_widget('main_window')
        self.main_window.show()

        gtk.main()


    @glade_callback
    def on_hello_button_clicked(self, data):
        print 'Hello, world!'

    @glade_callback
    def on_main_window_destroy(self, data):
        print 'Main window destroyed, exiting...'
        gtk.main_quit()


if __name__ == "__main__":
    hello = HelloWorld()
    hello.run()

The glade_callback function is a decorator which sets an attribute on a function defining it as a Glade callback. The get_glade_callbacks function looks for all functions with that attribute and returns them as a dictionary. I use these two helper functions in all of my GTK applications.

The run method is where the meat of the action happens. First we tell Glade to load the GUI that we created earlier in the tutorial:

    self.interface = gtk.glade.XML(GLADE_FILE)

Then we connect all of our Glade callbacks (all functions decorated with glade_callback) to their respective Glade signals as we defined them in the interface builder using introspection.

    self.callbacks = get_glade_callbacks(self)
    self.interface.signal_autoconnect(self.callbacks)

Now we get a handle on the main window and tell it to display. The name comes directly from the ‘Name’ property we set in Glade.

    self.main_window = self.interface.get_widget('main_window')
    self.main_window.show()

Finally, we tell GTK to start handling events for the program and passing them off to our custom signal handlers.

    gtk.main()

That little bit of scaffolding takes care of all of the GUI aspects of the application, and we’re now free to just worry about what we do when a user performs a specific action. That’s where our callbacks come in, which are the two functions on_hello_button_clicked which is called whenever the user clicks our button and on_main_window_destroy which is called when the window is closed.

You can grab the full source code and Glade file here.

Further reading material

Hopefully this guide was helpful. Comments or suggestions welcome.

Happy hacking!