geewax.org

coding is the best

Introducing Helipad

First, let me apologize in advance: I really hate reinventing the wheel.

OK, now that that's out of the way...

Helipad is a very light-weight framework built on top of webapp (the framework built-in to Google App Engine) that adds some of the stuff I felt was stalking me across all my projects:
  • Less boilerplate code for handlers (in particular for routes pointing to a file with only one handler)
  • Simplified URL routing (prefixes, un-ordered URLs)
  • Simple cookies
  • Sessions (backed by App Engine's Memcache service)
  • Simple static file serving
  • Easy templating with Jinja2
If you just want to get down and dirty with the code, check it out on GitHub: http://github.com/jgeewax/helipad

Getting Started

Let's start with a simple example. Keeping with convention, we'll call this HelloHelipad.

Start by creating a new application. To get started quickly, check out AppMake and type the following:

After that, create your first handler in hellohelipad/handlers/hellohelipad.py (and don't forget to hook it up in app.yaml) like this:

Cool -- that was pretty easy... 

For comparison, here's the way webapp has you do a HelloWorld application:

Take note of a few things that you didn't have to do:
  1. Re-define the URL mapping in hellohelipad.py
  2. Manually create a WSGI application based on your handler
  3. Define a main method that serves your application
Now that you have the basics, let's take a look at some of the other fun stuff :)

Simpler URL Routing

If you have just a single route in app.yaml pointing at your script, you don't need to redefine the route in the file, but what if you have a wildcard pointing at your script? For example, say your app.yaml file looks like this:

If you were using webapp, this would look like

With Helipad, you can do your routing the same way (with a list of tuples) like this:

or if the order doesn't matter to you (which in this case it doesn't) you can use a simple Python dictionary to map URLs to handlers like this:

Finally, if you have a common prefix across all your URLs (in this example, all of our routes are prefixed with "/about/"), you can define that prefix once and it will be automatically prepended to your URLs. This might make the previous handler look like this:

Finding and opening files

One of the things that I realized I never have in my projects is a consistent way of finding and opening files. When I was working on Django applications I would always create a directory for my templates and add that relative to my project directory using os.path and __file__ which has started to feel like boilerplate code. To deal with this, I threw together a simple way of defining a root module and then two helper methods (find_file and open_file) which will get the file you request relative to your root module. That is, if Helipad is located at /home/jj/helipad.py, and you set the root to the Helipad module, 'path/to/myfile.html' will resolve to '/home/jj/path/to/myfile.html'.

This is extremely useful when you don't care where your project is on the disk, you just care about files relative to your project. For example, say the About Me page needed to read from a file... (we're dropping the About Company page for now)

This will let you read static content that you've uploaded to App Engine and need to display in one way or another, or if you need to render a template you now don't have to worry "where is my template file?". This actually brings us to our next topic...

Serving static files (with fancy URLs)

The example above shows you how to use helipad.root(), helipad.find_file(), and helipad.open_file() to access files you've included with your project on App Engine, but that seems still like boilerplate code that you may end up copying and pasting around a lot. Keeping with the theme of DRY, Helipad provides a few different ways of serving static files without so much typing.

The three wasy to do this are...
  1. Using helipad.Handler.static() inside a regular handler
  2. Using helipad.static() to create a StaticApplication for serving a single file
  3. Using helipad.static() to create a StaticApplication with a URL mapping for many static files
For the first, instead of using helipad.open_file() and self.response.out.write(), you can just use self.static() like this...

If there really is nothing else your handler needs to do besides serve a single static file, you can use helipad.static() to generate the boilerplate handler for you like this:

You can even string together the helipad.root() and helipad.static() calls like this:

If we were to bring back the About Company page, you could map a bunch of URLs to static files just like you'd map a bunch of URLs to handlers. You can do this with a list of tuples or, if order doesn't matter, with a Python dictionary. Additionally, just like with helipad.app(), you can use a URL prefix to be prepended to all of your URLs. Here's an example showing all of that:

Handling templates

I had to make a call on templating and what to include, but I personally really like Jinja2. The syntax is pretty similar to Django's template language except it's a bit less restrictive. Because of my preference for Jinja2, this template language comes baked in to Helipad. Just like helipad.Handler.static(), you can use helipad.Handler.template(). The one extra piece is that if you get tired of defining your template directory over and over, you can define a template_root which is just prepended to all of your template file paths (which is then based on your helipad.root()). Perhaps this is best explained with an example... (see comments in-line)

Cookies

Handling cookies with Helipad is really naive. helipad.Handler has three methods related to cookies: set_cookie(), get_cookie(), and clear_cookie(). These should be self explanatory and take care of all the garbage with headers and whatnot which I got tired of having to reinvent every time I made a project. There may be some subtle bugs in here, but so far everything has been fine for me.

Sessions

Sessions in Helipad are equally simple. helipad.Handler has a property called session which gives you back a Session object which has a few methods for you to use: get(), set(), and delete(). The way this works is the Session generates a random key (stored in a cookie) which then serves as the namespace for Memcache operations (get, set, and delete). Since Memcache is behind all of this, expiration is out of our hands, but hopefully you're not putting anything critical in your sessions anyway...

Conclusion and future plans

So far, Helipad has just been a place where I end up putting stuff I notice needing in every project (such as cookies, sessions, etc). The goal has mainly been to make it simpler to do the common stuff (like serving a static file) but still make it easy to bypass all of that stuff if I need to (for example, you can do anything you want in a handler...) and I think that's worked out pretty well. There must be a few bugs here and there (particularly in the cookie and session area...) so any help fixing those would be awesome, but otherwise Helipad seems to be meeting my needs, and hopefully it meets yours and can help to speed up your webapp development.

If you're interested in helping out at all, you can find all the code on GitHub at http://github.com/jgeewax/helipad

Comments [0]

Introducing AppMake

Managing lots of different App Engine applications gets pretty tricky, so I threw together a little Makefile to make it easy to run the debug server, start the development console, deploy the application, etc.

This Makefile turns running the development server from:

into

The full list of commands is:

The code is open sourced on GitHub at http://github.com/jgeewax/appmake, and to get started, all you have to do is type...

There's more documentation in the README file here:

Comments [0]

Acquired by Google

In the early days of Invite Media we always talked about being acquired but I didn’t really think it would ever happen. Amazingly, back in June we were acquired by Google. (You can read about it in the official DoubleClick blog post.)

We’re all still getting ourselves situated at Google — our NYC engineers are happily settling into Google’s NYCoffice and Google is setting up a new office in Center City for the Philly engineers — but for the most part the work is the same. I’m still trying to find my way around, but everybody I’ve met so far has been really helpful.

Looks like I officially work for Google. Let the excitement begin…

Comments [0]

Default description value for Jira issues

We use Jira for a whole lot of things, but one of the major uses is for Bug reports. That said, sometimes people forget important information when filing a bug so I wanted to have the default description for Bug reports have some prompts for what information we’d need to reproduce the bugs (sort of like Google Code’s bug report interface).

Unfortunately, there’s no nifty UI to edit the default value for the description field in Jira, but there’s a niceKnowledge Base article which describes how you can do it by editing the template file used to render the description field. That file is: /path/to/atlassian-jira/WEB-INF/classes/templates/jira/issue/field/description-edit.vm

It wasn’t entirely clear how to do different things based on the issue type, etc so I had to poke around a bit to figure out what’s different, the section I wrote ended up as:

This makes the entire description-edit.vm file look like this:

Hopefully this helps someone else who wants to provide defaults for system fields like Description. In the meantime, please go vote on the feature request so that this blog post won’t be needed anymore.

Comments [0]

Forking django-chronograph

Keeping track of scripts that run regularly and e-mail the output is pretty simple when only one person is managing the schedules, scripts, and server. When you throw many more people into the mix, keeping track of who wants to see the output of which jobs and when jobs should run, you end up with a nasty crontab, and lots of cooks in the kitchen.

I went looking around for a simple way to throw a GUI ontop of a crontab, and stumbled across the Chronograph Django application. It had most of the stuff that we needed (including some really advanced scheduling options thanks to the dateutil library) so I tried spinning it up. What came out was a really nice user interface where we could see just about everything we wanted.

There were a few tweaks I wanted to make for my own purposes, but one major thing that was missing was a list of e-mail subscribers. This was surprisingly simple to throw in, and works really well since Django does pretty much all of the work for me.

Anyway, I’m hoping these changes can get pulled back into the main repository on Google Code, but if not, I’ve got a fork going on GitHub for anybody that’s interested:

http://github.com/jgeewax/django-chronograph

You can install this via easy install:

$ easy_install http://github.com/downloads/jgeewax/django-chronograph/django-chronograph-0.1...

Here's what the fork looks like:

       
Click here to download:
forking-django-chronograph-rDhAczEIfeBxCxlonDqb.zip (311 KB)

Comments [0]

Agile git Workflow

When we started using git to manage our source code at work, we actually jumped in a little bit too fast. It seems like there is a lot of writing about how you can do lots of really neat things with git, but no real guide about one particular way of using git for your project. This post is going to describe how we use git day to day on a reasonably large “agile-style” project.

Overview

The general overview of how this works is:

  1. Branch off of master to work on the feature
  2. Work, work, work, commit, commit, commit
  3. Pull down any updates to master
  4. Rebase and squash (with discretion) the feature branch to master’s HEAD
  5. Merge the feature branch back to master
  6. Push up to the shared repository

Also in this post are details about using release-candidate and production branches and merging features across those branches.

Working on a feature

Let’s imagine that you need to work on Issue #12. The first thing you’d do is create a feature branch from master for the issue. This keeps your work isolated, so that you can switch what you’re working on very quickly should the need arise.

Now that you’re on the feature branch, you might do all sorts of work towards completing this issue:

Getting feedback on your feature

If you want to share all that work with someone else as a patch (maybe for code review or something), thats is pretty simple:

This is really saying, "compare what’s currently on my branch with what’s on master and print it to standard out.

You can redirect the output of that to a file (ie, git diff master > issue12.diff) if you need.

Packaging up your work for the shared repository

When you’re ready to share this feature with the rest of your team, they probably won’t care about all your interim commits (for example, you added and then changed feature.py). We can package up all the commits into one big commit (sort of like the one big patch against master you used before) by using git’s rebase command.

The -i flag is “interactive” which allows you to specify what you want git to do with each of those interim commits. It should present a text editor that looks like this:

To pack all of these commits into a single commit, change all but the top one to s or squash. This says “keep the change from commit, but roll it into the parent commit”. The default as you can see is “pick” which if left alone would leave the commits separate. If for some reason you want to “undo” a commit that you already had, you can remove that line entirely. This makes will remove entirely the changes introduced in that particular commit.

Keep in mind that you might not want to package everything as a single commit, and anything that you want to keep separate, just leave that line as pick and it will be left alone. For the purposes of this article, we’re assuming that each branch is a completely isolated topic, where the intermediate commits are for your personal use, and not of any use to the rest of your team. This may not always be the case.

To “package” up the branch into a single commit, our editor would look like this:

Once you save that file, you’ll be prompted for a message to go along with the new “re-packaged” commit. Usually it’s best to start fresh from there and go with something about the feature, the prompt I got was:

Which you could change to:

Keeping the shared repository linear

At this point, the “issue-12” branch consists of one single commit. However, if somebody else has pushed any changes to the shared repository, merging in the feature branch will have two commits: one for the feature itself, and another to merge the feature with whatever else was pushed by the rest of the team.

All those “merged my local branch” commits will look like a lot of noise for no good reason, so git allows you to “replay” your commits against the master repository, with the end goal being to make your “re-packaged” commit to appear as though you branched right off the most updated version of the code.

This is sometimes referred to as “changing history” because you are updating the parent of your current commit to be further along in the future than it actually was.

To illustrate how this might work, you can make a small commit on master, and then use rebase to replay the commit against the updated master repository.

Now master has one extra commit, and issue-12 has one extra commit, where each has the same parent. If you were to merge issue-12 into master you’d have the two commits we mentioned before (one for the “re-packaged” feature as a single commit, and another for the recursive merge). To get rid of that second commit that really doesn’t provide much information to the team, we’ll rebase the commit to look like it happened after the typo-fixing commit we just threw into master. This also will give us the chance to resolve any conflicts.

This is one representation before we change the parent:

This is one representation after the rebase changed the parent:

Now we can merge the change back into master with a fast forward:

Now let’s double check that we can still fast-forward on the shared repository:

Since we can fast forward, it’s just a git push away:

And since we don’t need that issue branch anymore, we can delete it:

What if other people push stuff in the meantime?

If somebody else happens to push some code in the time between your rebase and your git remote show origin (you’ll know this because the push won’t be “fast forwardable”) you should first pull down any changes, and then do another rebase. All this will do is (again) replay your commits ontop of the new ones that got pushed so that the shared repository doesn’t have lots of noise from merging your feature branch. Since you’ve already got the commit packaged up the way you like, you can leave off the -i and just do:

The moral of the story is that if you want to keep your shared repository full of only the important stuff, make sure all your pushes are fast-forwards. This is as easy as always remembering to rebase immediately before you push.

Staging and production

Now that you have the concept of feature branches down pat, another thing that we struggled with was how to deal with branches that signified different versions of our code.

Ideally we’d like to have a “trunk” (aka master) and then a branch for the latest release-candidate (say “rc-1.0”) and then the official release branch (say “1.0”).

The basic idea is to treat the release-candidate branch just like master (re-packaging commits, feature branches, etc) and then the production branch as a “merge-only” or “cherry-pick-only” branch. Some people prefer to use tags for this, however using a tag means that anything currently in testing will hold up moving things into production. This is a choice you’ll have to make, a branch gives a bit more independence from the RC than a tag.

Creating the branch

It seems like the first place to start would be to create the RC branch:

Now you have a new release candidate branch that you can push changes to. Follow the same procedure discussed above to make sure that stays linear and everything should work out just fine.

This is what you’re RC branch might look like:

What if somebody else already created the branch?

Maybe someone else on your team already created the branch and you just need to check it out locally. This is pretty simple in git. The format is git checkout -b mybranch -t origin/mybranch. For example, if someone already created the rc-1.0 branch, we could pull it down like this:

Merging between branches

If everything is going fine, there will come a point when it’s time to merge all the changes you made on your RC branch back to the master branch. This is nothing more than a git merge. (Note that this is one place where you won’t be fast-forwarding, but always should have a merge commit to show that you merged the RC branch back to master.)

After things are merged back, your repository will look something like this:

Ready to release

When your release candidate has made it through all the testing it needs, you should be ready to create a production branch. This is just like the previous example where all you need to do is create the remote branch off of the RC branch.

Usually this branch will stay up to date with the RC branch, and as bugs are fixed in the RC they are just merged over into the production branch.

And once somebody verifies that the bug is then fixed:

The same thing applies when you want to pull back this new change into master:

What if I just want one commit?

There may be times when the production branch needs an important fix, but there are other features on the branch that haven’t been adequately tested. This problem is pretty well solved by “cherry-picking”. Merging won’t work because a merge carries with it all the parent commits until a common ancestor, which you definitely don’t want if those parent commits haven’t been tested.

The basic idea cherry-picking is that git will take the change-set, package it up as a different commit, and move it over to another branch, without taking anything else.

Let’s make two small bug fixes on rc-1.0, and then take the last one over with a cherry pick.

If we just wanted to merge over bug fix number 2, the merge would also carry bug fix number 1 along with it since that’s the common parent. We haven’t tested bug fix number 1, so we’ll need to cherry pick number 2 over onto the 1.0 branch by itself.

You now have the change-set from bug fix number 2, but it’s technically a fully different commit (with its own commit hash and everything). This allows you to move things over from the RC branch independently of other things in the branch. Also, when you do decide to merge over the rest of the RC branch, that shouldn’t be a problem at all since I guess git knows that the new commit has the same content and just a different parent.

Help, I broke things and want to start over

Another common thing I find myself doing all the time is having to “try again” after I accidentally merge over the wrong commit, or make a change that just isn’t right. This is where the git reset command comes in handy.

If you just committed before you were ready, the standard git reset HEAD~1 will work just fine. This is git-speak for “rewind things by 1 commit, but don’t change the contents of the files on disk”.

If you do want to undo any changes to files locally, you can use the --hard flag to change that as well: git reset --hard HEAD~1.

The HEAD~1 is notation for the head of the current branch minus 1 commit. So HEAD~4 would mean “four commits ago”.

You can also point out a specific commit by it’s short hash to rewind to that point in time. And remember the --hard flag will bring you back to that point with the exact code from then as well.

I accidentally started my work on master…

Sometimes I’ll make a couple of small changes on master and one thing leads to another and all the sudden I realize I’ve got a lot of changes right on master instead of a feature branch. Let’s get into that situation real quick:

OK, now we are on master, with two commits that we accidentally did right on master instead of a feature branch. The way we’ll get these onto a feature branch, and get master back to normal is with by branching and resetting.

First, we’ll create a branch from master which will have those two commits, then we’ll reset our master branch back to before we accidentally made those commits.

This keeps your changes off on the “feature1” branch, and then moves the “master” pointer back to the master pointer on the shared repository. If you checkout the feature1 branch, you’ll see that the changes you made originally on master are in place over there, and you can easily merge them back into master when the time comes to commit them.

That’s all

That’s all I have for now. If there’s anything missing that you’d like to see feel free to e-mail me at jj@(this website).

Extra thanks to Mark Chadwick for being the inspiration behind this post. Also if you’re going to be working with lots of git repositories and lots of different branches, see my previous post (http://geewax.org/2009/11/15/git-workspace-magic.html) for how to make your PS1 update based on what branch you’re currently on.

Other discussions

There are a few discussions going on over at Reddit and HackerNews, some of the feedback has been incorporated into the article since then, but hopefully the comments on these forums can help you to adapt this workflow to best suit the needs of your team.

You can find these discussions here:

Comments [0]

git Workspace Magic in bash

I have lots of projects in PyDev in Eclipse all of which I navigate through with the terminal in Ubuntu (this blog’s source is one of them), so just recently with the help of a couple co-workers I put together some bash methods for moving around my projects easily.

I found that the common pattern was:

The first thing I wanted was to know which project I was in and which branch in git that I was on. With a couple of helper methods this is actually pretty easy and just involves overwriting the PS1 environment variable.

Now my prompt lets me know pretty acurately where I am in my source:

The next thing I decided to add in was a command called `jumpto` which would deactivate whatever environment I’m in, jump into the proper directory, and then activate the current virtual environment for me to start working. Also, I wanted just `jumpto` with no arguments to bring me back to my home directory and exit out of any virtual environments.

The end result of this was the following:

This made moving around pretty easy:

The next thing I wanted was to do tab-completion based on the projects that I had set up in my `~/workspace/` directory. After some research on how tab completion works in the first place, I ended up with the following:

This lets me do things like `ju[tab] das[tab]` and end up in the dashboard project with the virtual environment all set up and ready to go.

I hope this helps somebody else out who’s working with lots of git projects with lots of different branches and virtual environments for each of them.

Comments [0]

Creating iPhone Ringtones on Ubuntu

I just had a hell of a time trying to turn a very short MP3 file into a ringtone for my iPhone. I ended up using Perl Audio Converter and faac (Freeware Advanced Audio Coder) to turn the MP3 file into an m4a. After that I renamed as m4r and dragged onto the iPhone.

The short version boils down to:

After that, just drag the m4r file onto the iPhone device in iTunes.

Comments [0]

Easy-Installing SOAPpy 0.12.0

I was recently working with virtualenv by Ian Bicking (which is awesome) and I ended up having quite a bit of trouble getting SOAPpy v 0.12.0 installed. It’s a standard EasyInstall package, so it should be no sweat, but it seems like SOAPpy (a deprecated package I believe…) has a few problems which appear to be “packaging” related.

For example, the package does a couple of from __future__ import ... calls, which in Python 2.5, must be the first line of the file. This seems fine, except the author(s) (I assume when they were packing the library up to share with the world) added copyright notices at the top of files as strings instead of comments. This means that importing the files (which EasyInstall does) will throw an Exception about the placement of the __future__imports.

There was also an issue with dependencies, where SOAPpy clearly requires the fpconst library, but doesn’t declare the dependency in it’s setup.py file. This means that you install SOAPpy and then run into an ImportError for a package that EasyInstall should just grab for you.

To remedy these things I repackaged SOAPpy 0.12.0 with the fixed __future__ imports and the proper dependency declarations. In case anyone else has a problem similar to this, you can find the file athttp://static.geewax.org/SOAPpy-0.12.0.zip

If you want to use EasyInstall to do the work, you can do it with:

sudo easy_install http://static.geewax.org/SOAPpy-0.12.0.zip

Comments [0]

GAE Testbed Documentation Released

I’d been looking to get familiar with Sphinx for a while, and couldn’t ever seem to find a project of mine that needed documentation enough to take the leap. This past week, I decided that even though GAE Testbed is a relatively small project, it could definitely use good documentation and finally got around to working with Sphinx.

I must say that I’m really impressed with the package as a whole (running it both on my MacBook and my Windows machine), though it was a bit tricky to get set up and familiar with the layout.

That said, feel free to take a look at the new GAE Testbed documentation which is located athttp://gaetestbed.geewax.org/index.html. For those of you looking to get the code for GAE Testbed , it’s located on Google Code at http://github.com/jgeewax/gaetestbed.

Comments [0]