JJ Geewax

Getting started with tmux

Thu, 21 Mar 2013 21:42:00 -0400

As a long-time user of GNU screen, I knew it would be difficult to switch, but I’ve made the move and am so far pretty happy. tmux does all the same things that screen does, but was a lot easier to customize. I wanted to share my tmux configuration file and other things that made the switch all that much more amazing.

Keyboard bindings

I changed the default keyboard bindings to be more like screen:

Ctrl-b? Forget that. Ctrl-a all the way.
Ctrl-(left/right) arrow to move between windows? Hell yes.
Ctrl-a twice to go “back”? Perfect.
Ctrl-a, r to reload the config file made it very easy to debug.

Status bar at the bottom

I made my status bar have the session name and my machine name on the far left, the various tabs centered in the middle, and the system time and date on the far right. It was very easy to configure, and I’m guessing I’ll end up tweaking it from time to time.

Project-specific configurations (and attach tool)

Here’s the coolest part in my opinion. I forked a repository called “tat” that does some nifty things to make project-specific configurations really easy to set up consistently, and attach to them later on. Oh, and there’s tab completion for existing sessions as well as for your project directory names. Check out my fork to see the details.

With tat you can create a .tmux file in your project directory, which be run with the first argument being the session name. You can use this to configure your project workspace easily and consistently. Here’s one of my example .tmux project files.

Tab completion

Don’t forget to enable tab completion. On Ubuntu I did the following:

$ sudo curl https://raw.github.com/mithro/rcfiles/master/bash/completion/tmux > /usr/share/bash-completion/completions/tmux

Python Dictionary Comprehensions

Thu, 21 Mar 2013 10:43:00 -0400

I’m a little embarrassed to say that I haven’t yet used these in my own code — they seem so obvious now that I look back on it.

>>> d = {'a': 1, 'b': 2', 'c': 3}
>>> keys = ['a', 'c']
>>> { k: d[k] for k in keys }
{'a': 1, 'c': 3}

Task Queue support in App Engine's ext.Testbed

Tue, 12 Mar 2013 13:25:00 -0400

A while back I wrote some (now deprecated) code that allowed you to easily test your Python App Engine applications and their interaction with the App Engine APIs. When we got acquired by Google, I started working with some of the awesome App Engine engineers on making that code part of the official App Engine codebase.

We launched that, but one of the APIs that was noticeably lacking helper-methods was the Task Queue. I added one of the old methods (get_filtered_tasks()) but a bit later I noticed a pretty serious bug where timezones weren’t handled properly.

So that code hid quietly in the App Engine code base, undocumented and technically broken. Sorry :(

I wrote some extra code to fix it, but that didn’t get merged into the main repository for quite a while.

But now it’s here.

If you’re curious about the change, feel free to check out the diff (scroll to the bottom). The method is still called get_filtered_tasks() and takes several arguments as “filters” for various properties of tasks — and it should properly handle timezones when you set the eta or countdown properties when creating tasks.

Sorry for the delay. And enjoy.

PS: Since this isn’t documented on the official App Engine page, the API may change. I don’t intend to change it, but I can’t promise it won’t. Apologies in advance if you use this helper method and have it break out from under you.

Also, check out the official code.

Some example code:

import unittest
from google.appengine.api import taskqueue
from google.appengine.ext import testbed


class TaskQueueTestCase(unittest.TestCase):

  def setUp(self):
    self.testbed = testbed.Testbed()
    self.testbed.activate()
    self.testbed.init_taskqueue_stub()
    self.task_queue_stub = self.testbed.get_stub(testbed.TASKQUEUE_SERVICE_NAME)

  def tearDown(self):
    self.testbed.deactivate()

  def testTaskAdded(self):
    taskqueue.add(url='/path/to/task')

    tasks = self.taskqueue_stub.get_filtered_tasks(url='/path/to/task')
    self.assertEqual(1, len(tasks))
    self.assertEqual('/path/to/task', tasks[0].url)

unittest.main()

Daily Meeting

Sat, 23 Feb 2013 14:41:00 -0500

TLDR: I re-built one of my old side projects. Check it out at dailymeetingapp.com

A long time ago, when I was working with a bunch of awesome guys at Invite Media, we started expanding our engineering team quite a bit. We had two engineering offices (New York City and Philadelphia) that were each doing more and more stuff, which was awesome, but we had a problem:

our daily “stand-up” meeting in two cities was kind of a pain.

To clarify what I mean by this, when I say “stand-up meeting” I mean that we had one meeting, every day atnoon, where we went around the room and told the rest of the team what we did since yesterday, and what we’re doing for the rest of the day. It was some kind of agile-ish thing that our team had deteriorated into. (There are lots of ways people do “Scrum” or “Agile” or “Hip New Methodology”, but let’s not get into that. )

This sounds like an easy meeting, right? 30 seconds per person. Boom. Done.

What about when you have this with two groups of people in different cities?

Early on we tried Skype. That was horrible. Laptop microphones really aren’t made for group conference calls. The camera wasn’t really all that great either.

Next we had a coordless phone. We would put that on speaker-phone and pass it around through the group. The audio was a bit better, but it was still pretty bad.

Then we upgraded to those fancy tripod-looking conference room phones, which were a lot better. Still not quite like being in-person, but they got the job done just fine. One problem solved.

The next issue we had to tackle was that most people would show up to this meeting, and not remember what they did yesterday! “I, um…. worked on… um… what did I do… hmm… one second…. oh yea! I built this awesome thing that does …. “

For a 30-second-per-person meeting, that is a lot of wasted time! What we needed was a reminder for everyone to actually think beforehand about what they did, so they could show up prepared for this meeting.

We can’t force everyone to set an alarm, but even if we did, how would we know if they actually thought out what they were working on? What if we had a little app that sent a reminder e-mail at 11:30 to everyone, and they could reply with what they did for everyone to see?

Sounds like a pain in the ass, right? It was. But it did have some benefits.

It didn’t get in the way as much as you’d think. Reply to the e-mail and that’s it. You’re done.
It was easy for everyone to see which people didn’t post an update. Little red dot next to their name.Busted.
People who missed our stand-up meeting (travel, doctor’s appointment, whatever) could come in later and see who did what. Sweet.
People who checked beforehand could show up prepared with a question or two based on what they read.
Everyone got a digest e-mail at noon with everyone’s notes. Super easy to notice the guy who says every day that he’s going to finish Feature X. What’s going on there…?

Eventually, the app morphed into something that was more about a digital (and shared) work-log than an “online stand-up meeting”, which was pretty cool for me.

Recently, I rebuilt quite a bit of the app and started testing it out with some people — who seem to really like it. It’s still not completely done (as you can see from some lorem-ipsum on the landing page….), but I would love to get some constructive feedback (and feature requests).

Check it out at http://www.dailymeetingapp.com/

Login required should be the default

Sat, 23 Feb 2013 14:38:00 -0500

If you’re building a web-app in Python that requires people to sign in, you are probably pretty familiar with the concept of a @login_required decorator. For those that might not be, it’s pretty straight forward: the decorator wraps a handler method with a check for whether the current request is coming from someone who has been through your sign in process.

When I first saw this however many years ago, I was pretty new to Python decorators and thought it was pretty awesome. I still think the whole concept is pretty awesome. This seems to fit the mold of a “decorator” pretty perfectly… almost as if decorators were built with this in mind…

But what if you forget your @login_required decorator? It’s so easy to do. When you’re testing that “Edit your account” page, you’re authenticated, you assume everything should go to plan. You could (and should) add a functional test to check that an unauthenticated request is redirected, but this is just a quick little side project…

So why don’t we just make login_required the default? I think that it should be, so here’s some code to save somebody else some time.

It works by wrapping all local methods with your login_required method during object creation (aka,__new__). If you want a method to be public, you decorate it with the login_not_required method — which sets a flag on the method saying … that login is not required.

Here’s the BaseHandler (and Meta class) which does the wrapping:

import types

import webapp2

from auth import login_required


class BaseHandlerMeta(type):
  """Meta class for all request handlers.

  This automatically wraps all handler methods with the login_required
  decorator. If something should be exposed publicly, it should be wrapped
  with the login_not_required decorator.
  """

  def __new__(cls, name, bases, local):
    if name != 'BaseHandler':
      for func_name, func in local.iteritems():
        if isinstance(func, types.FunctionType):
          local[func_name] = login_required(func)

    return type.__new__(cls, name, bases, local)

class BaseHandler(webapp2.RequestHandler):
  """Base class for all RequestHandlers."""

  __metaclass__ = BaseHandlerMeta

  def user_is_logged_in(self):
    # Do some magic here to check if someone is logged in
    return False

Here are the login_required and login_not_required decorators:

def login_not_required(handler_method):
  """Allows a user to *not* be logged in.

  The login_required attribute is inspected by BaseHandlerMeta and is used
  as the flag for whether to wrap a method with login_required or not.
  """
  handler_method.login_required = False
  return handler_method

def login_required(handler_method):
  """Requires that a user be logged in."""

  required = getattr(handler_method, 'login_required', True)
  already_wrapped = getattr(handler_method, 'wrapped', False)

  # If the method doesn't require a login, or has already been wrapped,
  # just return the original.
  if not required or already_wrapped:
    return handler_method

  def check_login(self, *args, **kwargs):
    if not self.user_is_logged_in():
      uri = self.uri_for('login', redirect=self.request.path)
      self.redirect(uri, abort=True)
    else:
      return handler_method(self, *args, **kwargs)

  # Let others know that this method is already wrapped to avoid wrapping
  # it more than once...
  check_login.wrapped = True

  return check_login

This would make your request handlers look something like this:

import base
from auth import login_not_required


class MyHandler(base.BaseHandler):
  @login_not_required
  def my_public_method(self):
    self.response.write('Hello world!')

  @login_not_required
  def my_public_method_with_a_problem(self):
    # Calling this should force a redirect to the login page!
    self.my_other_handler_that_is_protected()

  def my_other_handler_that_is_protected(self):
    self.response.write('Hello privately!')

  def protected_by_default(self):
    self.response.write('Once you log in, you can view this!')

Notice that if you are in a public method (@login_not_required) and you call a method that does *not* have that decorator, you’ll get redirected to a login page. If you want something to be public, it and all of the functions it calls (inside the handler) should be explicitly defined as public.

Friendly URL routing in App Engine using webapp2

Sat, 23 Feb 2013 14:36:00 -0500

I used to do crazy things like hard-code URLs in my templates:

<a href="/accounts/create/">Create your account</a>

or

<a href="/accounts/view/?id={{ the_id }}">View account {{ the_id }}</a>

Ever since webapp2 came around, I’ve been really into the URL routing helpers, which make it much easier to keep long lists of URLs organized. I started using a new project layout that makes routing really simple — maybe even fun.

Before I get to the interesting part I need to digress slightly. A while back, I posted a question on Stack Overflow asking whether it was better to break your app apart into separate routes in app.yaml or inside aWSGIApplication. Put simply: should you create one WSGIApplication object and route everything there? Or many separate ones, each with their own routing in app.yaml?

The short answer was: It doesn’t really matter.

The longer answer was: if you’re particularly sensitive about start-up time or memory, you might break things apart.

I decided to use one big app. Having a single app makes unit testing with ext.testbed a bit easier.

In addition to making testing easier, I’ve been using the Python 2.7 runtime which lends itself nicely to a single WSGIApplication: I put my code in main.py and then route /.* to main.app and don’t think about it anymore. But how are you supposed to keep track of all the URLs, Handlers, methods, and routes in your app as it grows larger and larger?

webapp2 to the rescue.

Take a look at routes.py in this gist:

from webapp2_extras.routes import RedirectRoute

from handlers.account import AccountHandler


__all__ = ['application_routes']

application_routes = []

_route_info = [
  ('account.list',   'GET', '/accounts/',                 AccountHandler, 'list'),
  ('account.create', None,  '/accounts/create/',          AccountHandler, 'create'),
  ('account.view',   'GET', '/accounts/<id:\d+>/',        AccountHandler, 'view'),
  ('account.delete', None,  '/accounts/<id:\d+>/delete/', AccountHandler, 'delete'),
  ('account.update', None,  '/accounts/<id:\d+>/update/', AccountHandler, 'update'),
]

for name, methods, pattern, handler_cls, handler_method in _route_info:
  # Allow a single string, but this has to be changed to a list.
  # None here means any method
  if isinstance(methods, basestring):
    methods = [methods]

  # Create the route
  route = RedirectRoute(name=name, template=pattern, methods=methods,
                        handler=handler_cls, handler_method=handler_method)

  # Add the route to the public list
  application_routes.append(route)

There are a few things to notice here:

You could type out all the boiler-plate every time. I don’t think routes have enough distinguishing factors to merit that. I removed the boiler plate and put together a list of tuples.
Each tuple defines a route: (name, methods, URL pattern, HandlerClass, handler_method).
Routes are named as thing.action (thing is singular — not things).
The method can be 'GET', 'POST', None, or any other method (or list of methods). None simply means “I don’t care”.
URL patterns allow fancy matching (<id:\d+>).
Rather that having many handlers (CreateUserHandler, DeleteUserHandler), I use one handler per “thing” and have the methods correspond to the actions (UserHandler.create, UserHandler.delete). If necessary, I can use self.request.method to distinguish between intent (ie, “show me the account creation form” versus “create my account and rediect me somewhere”).
The handler method name (AccountHandler.list) and the “action” part of the route name (‘account.list’) are identical. This makes linking easy — you know the route’s name because it lines up with your code.
I break from PEP8 about lining data up vertically. This is data you’d put in a table. I don’t like jagged tables.

Now to tie this all together, I just put some configuration for webapp2 into config.py:

__all__ = ['webapp2_config']

webapp2_config = {
  'webapp2_extras.sessions': {
    'secret_key': 'Put some magical secret here.', 
  },
  'webapp2_extras.jinja2': {
    'environment_args': {
      'autoescape': True,
    },
  },
}

and hook it all up in main.py:

import webapp2

from config import webapp2_config
from routes import application_routes


app = webapp2.WSGIApplication(routes=application_routes, config=webapp2_config)

After that, your handlers might look like this:

import webapp2
from webapp2_extras import auth
from webapp2_extras import jinja2
from webapp2_extras import sessions


class BaseHandler(webapp2.RequestHandler):
  @webapp2.cached_property
  def jinja2(self):
    return jinja2.get_jinja2(app=self.app)

  @webapp2.cached_property
  def auth_config(self):
    return {'login_url': self.uri_for('login'),
            'logout_url': self.uri_for('logout')}

  @webapp2.cached_property
  def auth(self):
    return auth.get_auth()

  @webapp2.cached_property
  def session_store(self):
    return sessions.get_store(request=self.request)

  def dispatch(self):
    """Override dispatch to persist session data."""
    try:
      super(BaseHandler, self).dispatch()
    finally:
      self.session_store.save_sessions(self.response)

  def render_template(self, template, context=None):
    context = context or {}

    extra_context = {
      'request': self.request,
      'uri_for': self.uri_for,
    }
    
    # Only override extra context stuff if it's not set by the template:
    for key, value in extra_context.items():
      if key not in context:
        context[key] = value

    rendered = self.jinja2.render_template(template, **context)
    self.response.write(rendered)

from handlers import base


class AccountHandler(base.BaseHandler):
  def create(self):
    if self.request.method == 'POST':
      # Create the account
      pass
    
      # Redirect them to the dashboard page
      return self.redirect(self.uri_for('account.view'))
    
    else:
      return self.render_template('account_create.html')

And your templates can now create links that look like this:

<a href="{{ uri_for('account.create', my_param='something') }}">Create your account</a>

or

<a href="{{ uri_for('account.view', id=the_id) }}">View account {{ the_id }}</a>

Extra bonus: If you delete a URL and the template tries to look it up, you’ll get an exception — and so will your unit tests. I guess this could be a scary thing if you don’t have tests, but hopefully it is helpful.

GAE Testbed moving into the Python SDK!

Sat, 23 Feb 2013 14:31:04 -0500

Since we were acquired by Google, I figured that it would be a fitting 20% project to work with some App Engine engineers to move GAE Testbed into the App Engine Python SDK. The benefits are pretty obvious:

No third-party package to install (ie, no more easy_install gaetestbed)
“Official” documentation for how to write your tests
More people working on making it easy for users to test their code
Many maintainers, meaning fewer regressions (ie, no more “whoops, the API changed a bit… GAE Testbed doesn’t work right now…”)

And I’m incredibly happy to say that that the 1.4.3 release includes the first steps toward no longer needing GAE Testbed at all: google.appengine.ext.testbed! The testbed module sets up your environment so that you’re able to verify that API calls you expected to be made are actually made.

For example, you can put something in memcache and then get it back to out to verify your application logic. Of course, this isn’t using an actual memcache instance, but looks and acts mostly the same.

I had very little involvement with the code in this release (just a reviewer), but there’s plenty more to come in the future. Many thanks to Robert Schuppenies and Guido van Rossum for helping get this huge improvement out the door.

P.S. Although there isn’t yet a way to do things like self.assertEmailSent(to=”email@example.com”), I hope that feature parity between google.appengine.ext.testbed and GAE Testbed is only a short while away.

Encoding subtitles into your movie...

Sat, 23 Feb 2013 14:31:00 -0500

So, you have an AVI file and a SRT file, and you want to put them together so that you don’t have to carry around the two files… (and you’re on Ubuntu…)

$ sudo apt-get install mencoder
$ mencoder -oac copy -ovc lavc -sub TheSubtitlesFile.srt \
  -subfont-text-scale 3 -o TheMovieWithSubtitles.avi \
  TheOriginalMovie.avi

(Of course you could add change “-ovc lavc” to “-ovc divx”, all up to you…)

My git workflow -- in code

Sat, 23 Feb 2013 14:23:00 -0500

There have been a lot of blog posts (one of mine included…) that talk about using feature branches to have an “agile” workflow while you code. This is all great to talk about but it seemed to me that a lot of people weren’t doing it. Not because they were against the idea but because it was too tricky to remember or too complicated to do or too annoying to do for that one little feature.

Then there are tools like git flow that do the trick but they seemed like kind of a lot of typing to me, so I threw something together that was three aliases to help make my workflow involve less typing, less remembering, and less confusion.

Note: If you ask me, “what git workflow tool should I use?” my answer will be git flow. git flow is FAR better documented, supported, and tested than the tool that I threw together today…

Now… it is with great pleasure that I introduce git-feature. It’s simple to install (all three commands are git aliases — you know, those things in .git/config?), simple to remember (if you forget arguments, it will prompt you for them), mostly safe (it prompts you before doing anything that changes your repository), and has very little magic (those git aliases are just shell commands, think of this as a few friendly shortcuts…).

So, getting right down to the dirt… here’s how I currently use it:

$ git branch
* master
$ git feature my-new-feature
this will create a feature branch my-new-feature to be merged into master (Y/n): Y
$ git branch
* features/my-new-feature
  master
$ # Do some work here, commit it...
$ git finish
this will integrate my-new-feature into master (Y/n): Y
$ git branch
* master

If you’re actually wondering, “Hmm… maybe I could use this!”, you’ll probably have lots of questions like…

what if I forget arguments? or…
what if I want my feature to based off a different parent? or…
what if I want to integrate my feature into two different parents?

Most of there are answered on the project’s GitHub page: http://github.com/jgeewax/git-feature but if you have one that isn’t answered, I’d guess somebody else had that question, so let me know and I’ll try to come up with an answer for everyone.

Deploying code on App Engine? Be careful with your indexes...

Sat, 23 Feb 2013 14:22:00 -0500

Have you ever deployed a new version of your application on App Engine and end up with this rude error saying something like, “You need an index to run that query”? So you scratch your head for a minute, check index.yaml, and sure enough that index is there just like you thought… After all, most (if not all) of your index.yaml is auto generated by the SDK…

It looks like the SDK is so good about keeping the contents of index.yaml off your mind that you forget about the time it takes to build those indexes!

I’m sure it’s written down somewhere but those of us that are so eager to get code out the door are clearly not the ones reading the fine print in the documentation… The short version of all of this is…

When you have modifications to index.yaml (adding a new index for example), it takes time for App Engine to build those indexes. However, the code you deployed is running right away and that code needs those indexes right away. So you end up with a nasty mean error all because you were a bit too hasty with kicking out your code (and the SDK is too busy pushing your files up to Google’s servers to warn you: “Hey wait a minute! This index is going to take a while to build…”)

Anyway, I figured I’d share the simple way I avoid getting bitten by this little annoyance…

First, I use git. If you’re not using any sort of source control, you should probably start doing that now. Even if you don’t want to share your source with anyone, having a local git repository is great… “Whoops! Let me revert back to where things were 10 minutes ago…”

If you’re using git, this problem is fixed by breaking apart your changes into two pieces: the index changes and the code that needs the index changes. If you’re working on Feature X and there are changes to index.yaml (git diff index.yaml), I commit those changes first (git add index.yaml && git commit -m “Updating indexes”) and then stash the other stuff on a new branch (git checkout -b feature-x && git add -A && git commit -m “Feature X”). Then switch back to your previous branch (git checkout master) and deploy JUST those index changes (make deploy*) and then wait!

Go into the Data store Indexes page on the App Engine admin console and keep an eye on that page. It will show your new indexes building and queued and all that lovely stuff (which could take a while).

Once you see that your new index is “serving”, just merge in your other branch (it should be a simple fast-forward) (git merge feature-x) and deploy again (make deploy). And you’re all set!

I’m sure there are other ways of doing this so don’t take this as the only way to handle the problem, but this way seems to work pretty well for me.

Happy deploying (and indexing).

* Deploying with “make deploy” is thanks to AppMake. It makes deploying really simple…

Introducing Helipad

Sat, 23 Feb 2013 14:21:00 -0500

First, let me apologize in advance: I really hate reinventing the wheel.

OK, now that that’s out of the way…Helipad is a very light-weight framework built on top of webapp (the framework built-in to Google App Engine) that adds some of the stuff I felt was stalking me across all my projects:

Less boilerplate code for handlers (in particular for routes pointing to a file with only one handler)
Simplified URL routing (prefixes, un-ordered URLs)
Simple cookies
Sessions (backed by App Engine’s Memcache service)
Simple static file serving
Easy templating with Jinja2

If you just want to get down and dirty with the code, check it out on GitHub: http://github.com/jgeewax/helipad

Getting Started

Let’s start with a simple example. Keeping with convention, we’ll call this HelloHelipad.

Start by creating a new application. To get started quickly, check out AppMake and type the following:

$ curl -OL --silent http://github.com/jgeewax/appmake/raw/master/Makefile
$ make helipad-project name=hellohelipad

After that, create your first handler in hellohelipad/handlers/hellohelipad.py (and don’t forget to hook it up in app.yaml) like this:

app.yaml:

application: hellohelipad
version: 1
runtime: python
api_version: 1

handlers:
- url: /
  script: hireforge/handlers/hellohelipad.py

handler.py:

import helipad

class HelloHelipadHandler(helipad.Handler):
  def get(self):
    self.response.out.write('Hello, Helipad!')

main, application = helipad.app(HelloHelipadHandler)

if __name__ == "__main__":
  main()

Cool — that was pretty easy…

For comparison, here’s the way webapp has you do a HelloWorld application:

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

class MainPage(webapp.RequestHandler):
  def get(self):
    self.response.headers['Content-Type'] = 'text/plain'
    self.response.out.write('Hello, webapp World!')

application = webapp.WSGIApplication([('/', MainPage)],
                                     debug=True)

def main():
  run_wsgi_app(application)

if __name__ == "__main__":
  main()

Take note of a few things that you didn’t have to do:

Re-define the URL mapping in hellohelipad.py
Manually create a WSGI application based on your handler
Define a main method that serves your application

Now that you have the basics, let’s take a look at some of the other fun stuff :)

Simpler URL Routing

If you have just a single route in app.yaml pointing at your script, you don’t need to redefine the route in the file, but what if you have a wildcard pointing at your script? For example, say your app.yaml file looks like this:

application: hellohelipad
version: 1
runtime: python
api_version: 1

handlers:
- url: /about/.*
  script: hellohelipad/handlers/about.py

If you were using webapp, this would look like

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

class AboutMeHandler(webapp.RequestHandler):
  def get(self):
    self.response.out.write('About Me!')

class AboutCompanyHandler(webapp.RequestHandler):
  def get(self):
    self.response.out.write('About Company!')

application = webapp.WSGIApplication([('/about/me/', AboutMeHandler),
                                      ('/about/company/', AboutCompanyHander)],
                                     debug=True)

def main():
  run_wsgi_app(application)

if __name__ == "__main__":
  main()

With Helipad, you can do your routing the same way (with a list of tuples) like this:

import helipad

class AboutMeHandler(helipad.Handler):
  def get(self):
    self.response.out.write('About Me!')

class AboutCompanyHandler(helipad.Handler):
  def get(self):
    self.response.out.write('About Company!')

main, application = helipad.app([
  ('/about/me/', AboutMeHandler),
  ('/about/company/', AboutCompanyHandler),
])

if __name__ == "__main__":
  main()

or if the order doesn’t matter to you (which in this case it doesn’t) you can use a simple Python dictionary to map URLs to handlers like this:

import helipad

class AboutMeHandler(helipad.Handler):
  def get(self):
    self.response.out.write('About Me!')

class AboutCompanyHandler(helipad.Handler):
  def get(self):
    self.response.out.write('About Company!')

main, application = helipad.app({
  '/about/me/': AboutMeHandler,
  '/about/company/': AboutCompanyHandler,
})

if __name__ == "__main__":
  main()

Finally, if you have a common prefix across all your URLs (in this example, all of our routes are prefixed with “/about/”), you can define that prefix once and it will be automatically prepended to your URLs. This might make the previous handler look like this:

import helipad

class AboutMeHandler(helipad.Handler):
  def get(self):
    self.response.out.write('About Me!')

class AboutCompanyHandler(helipad.Handler):
  def get(self):
    self.response.out.write('About Company!')

main, application = helipad.app('/about/', {
  'me/': AboutMeHandler,
  'company/': AboutCompanyHandler,
})

if __name__ == "__main__":
  main()

Finding and opening files

One of the things that I realized I never have in my projects is a consistent way of finding and opening files. When I was working on Django applications I would always create a directory for my templates and add that relative to my project directory using os.path and __file__ which has started to feel like boilerplate code. To deal with this, I threw together a simple way of defining a root module and then two helper methods (find_file and open_file) which will get the file you request relative to your root module. That is, if Helipad is located at /home/jj/helipad.py, and you set the root to the Helipad module, ‘path/to/myfile.html’ will resolve to ‘/home/jj/path/to/myfile.html’.

This is extremely useful when you don’t care where your project is on the disk, you just care about files relative to your project. For example, say the About Me page needed to read from a file… (we’re dropping the About Company page for now)

import helipad

class AboutMeHandler(helipad.Handler):
  def get(self):
    about_me = helipad.open_file('templates/about_me.html')
    self.response.out.write(about_me.read())

# This sets the root to our project module
helipad.root('hellohelipad')

main, application = helipad.app(AboutMeHandler)

if __name__ == "__main__":
  main()

This will let you read static content that you’ve uploaded to App Engine and need to display in one way or another, or if you need to render a template you now don’t have to worry “where is my template file?”. This actually brings us to our next topic…

Serving static files (with fancy URLs)

The example above shows you how to use helipad.root(), helipad.find_file(), and helipad.open_file() to access files you’ve included with your project on App Engine, but that seems still like boilerplate code that you may end up copying and pasting around a lot. Keeping with the theme of DRY, Helipad provides a few different ways of serving static files without so much typing.

The three ways to do this are…

Using helipad.Handler.static() inside a regular handler
Using helipad.static() to create a StaticApplication for serving a single file
Using helipad.static() to create a StaticApplication with a URL mapping for many static files

For the first, instead of using helipad.open_file() and self.response.out.write(), you can just use self.static() like this…

import helipad

class AboutMeHandler(helipad.Handler):
  def get(self):
    self.static('templates/about_me.html')

helipad.root('hellohelipad')

main, application = helipad.app(AboutMeHandler)

if __name__ == "__main__":
  main()

If there really is nothing else your handler needs to do besides serve a single static file, you can use helipad.static() to generate the boilerplate handler for you like this:

import helipad

helipad.root('hellohelipad')

main, application = helipad.static('templates/about_me.html')

if __name__ == "__main__":
  main()

You can even string together the helipad.root() and helipad.static() calls like this:

import helipad

main, application = helipad.root('hellohelipad').static('templates/about_me.html')

if __name__ == "__main__":
  main()

If we were to bring back the About Company page, you could map a bunch of URLs to static files just like you’d map a bunch of URLs to handlers. You can do this with a list of tuples or, if order doesn’t matter, with a Python dictionary. Additionally, just like with helipad.app(), you can use a URL prefix to be prepended to all of your URLs. Here’s an example showing all of that:

import helipad

main, application = helipad.root('hellohelipad').static('/about/', {
  'me/': 'templates/about_me.html',
  'company/': 'templates/about_company.html',
})

if __name__ == "__main__":
  main()

Handling templates

I had to make a call on templating and what to include, but I personally really like Jinja2. The syntax is pretty similar to Django’s template language except it’s a bit less restrictive. Because of my preference for Jinja2, this template language comes baked in to Helipad. Just like helipad.Handler.static(), you can use helipad.Handler.template(). The one extra piece is that if you get tired of defining your template directory over and over, you can define a template_root which is just prepended to all of your template file paths (which is then based on your helipad.root()). Perhaps this is best explained with an example… (see comments in-line)

import helipad

helipad.root('hellohelipad').template_root('templates/')

class AboutMeHandler(helipad.Handler):
  def get(self):
    # This will render the template hellohelipad/templates/template.html
    # Where hellohelipad is wherever the hellohelipad module is located.
    return self.template('template.html', {
      'my_template_variable': 'This is a template variable!',
    })

main, application = helipad.app(AboutMeHandler)

if __name__ == '__main__':
  main()

Cookies

Handling cookies with Helipad is really naive. helipad.Handler has three methods related to cookies:set_cookie(), get_cookie(), and clear_cookie(). These should be self explanatory and take care of all the garbage with headers and whatnot which I got tired of having to reinvent every time I made a project. There may be some subtle bugs in here, but so far everything has been fine for me.

Sessions

Sessions in Helipad are equally simple. helipad.Handler has a property called session which gives you back a Session object which has a few methods for you to use: get(), set(), and delete(). The way this works is the Session generates a random key (stored in a cookie) which then serves as the namespace for Memcache operations (get, set, and delete). Since Memcache is behind all of this, expiration is out of our hands, but hopefully you’re not putting anything critical in your sessions anyway…

Conclusion and future plans

So far, Helipad has just been a place where I end up putting stuff I notice needing in every project (such as cookies, sessions, etc). The goal has mainly been to make it simpler to do the common stuff (like serving a static file) but still make it easy to bypass all of that stuff if I need to (for example, you can do anything you want in a handler…) and I think that’s worked out pretty well. There must be a few bugs here and there (particularly in the cookie and session area…) so any help fixing those would be awesome, but otherwise Helipad seems to be meeting my needs, and hopefully it meets yours and can help to speed up your webapp development.

If you’re interested in helping out at all, you can find all the code on GitHub at http://github.com/jgeewax/helipad

Introducing AppMake

Sat, 23 Feb 2013 14:17:12 -0500

Managing lots of different App Engine applications gets pretty tricky, so I threw together a little Makefile to make it easy to run the debug server, start the development console, deploy the application, etc.

This Makefile turns running the development server from:

$ python /usr/local/google_appengine/dev_appserver.py -a 0.0.0.0 -p 9091 --datastore_path=./datastore --disable_static_caching .

into

$ make serve

The full list of commands is:

$ make help
AppEngine make file. Options are:
 test       Runs the test suite
 coverage   Runs the test suite and prints a coverage report
 deploy     Deploys the current project to AppEngine
 rollback   Rolls back a unclosed update to the application
 serve      Runs the development web server
 console    Opens a development console to your remote application
            (Only works if you've enabled the /remote_api URL)

The code is open sourced on GitHub at http://github.com/jgeewax/appmake, and to get started, all you have to do is type…

curl -OL --silent http://github.com/jgeewax/appmake/raw/master/Makefile

There’s more documentation in the README file here:

http://github.com/jgeewax/appmake/blob/master/README.markdown

Acquired by Google

Sat, 23 Feb 2013 14:16:00 -0500

In the early days of Invite Media we always talked about being acquired but I didn’t really think it would ever happen. Amazingly, back in June we were acquired by Google. (You can read about it in the official DoubleClick blog post.)

We’re all still getting ourselves situated at Google — our NYC engineers are happily settling into Google’s NYC office and Google is setting up a new office in Center City for the Philly engineers — but for the most part the work is the same. I’m still trying to find my way around, but everybody I’ve met so far has been really helpful.

Looks like I officially work for Google. Let the excitement begin…

Default description value for Jira issues

Sat, 23 Feb 2013 14:15:35 -0500

We use Jira for a whole lot of things, but one of the major uses is for Bug reports. That said, sometimes people forget important information when filing a bug so I wanted to have the default description for Bug reports have some prompts for what information we’d need to reproduce the bugs (sort of like Google Code’s bug report interface).

Unfortunately, there’s no nifty UI to edit the default value for the description field in Jira, but there’s a nice Knowledge Base article which describes how you can do it by editing the template file used to render the description field.

That file is: /path/to/atlassian-jira/WEB-INF/classes/templates/jira/issue/field/description-edit.vm

It wasn’t entirely clear how to do different things based on the issue type, etc so I had to poke around a bit to figure out what’s different, the section I wrote ended up as:

#if ($description == '' && $issue.getIssueType().getString('name') == 'Bug')
#set ($description = "Steps taken to produce this bug:\
\
\
Expected Results:\
\
\
Actual Results:\
\
\
Extra Information:\
\
")
#set ($description = $description.replace('\', ''))
#end

This makes the entire description-edit.vm file look like this:

jira@jira1:~$ cat ~jira/atlassian-jira/WEB-INF/classes/templates/jira/issue/field/description-edit.vm
#controlHeader ($action $field.id $i18n.getText($field.nameKey) $fieldLayoutItem.required $displayParameters.get('noHeader'))

## setup some additional parameters
$!rendererParams.put("rows", "12")
$!rendererParams.put("wrap", "virtual")

## Add our default value
#if ($description == '' && $issue.getIssueType().getString('name') == 'Bug')
#set ($description = "Steps taken to produce this bug:\
\
\
Expected Results:\
\
\
Actual Results:\
\
\
Extra Information:\
\
")
#set ($description = $description.replace('\', ''))
#end

## let the renderer display the edit component
$rendererDescriptor.getEditVM($!description, $!issue.key, $!fieldLayoutItem.rendererType, $!field.id, $!field.name, $rendererParams, false)

#controlFooter ($action $fieldLayoutItem.getFieldDescription() $displayParameters.get('noHeader'))

Hopefully this helps someone else who wants to provide defaults for system fields like Description. In the meantime, please go vote on the feature request so that this blog post won’t be needed anymore.

Forking django-chronograph

Sat, 23 Feb 2013 14:13:21 -0500

Keeping track of scripts that run regularly and e-mail the output is pretty simple when only one person is managing the schedules, scripts, and server. When you throw many more people into the mix, keeping track of who wants to see the output of which jobs and when jobs should run, you end up with a nasty crontab, and lots of cooks in the kitchen.

I went looking around for a simple way to throw a GUI ontop of a crontab, and stumbled across the Chronograph Django application. It had most of the stuff that we needed (including some really advanced scheduling options thanks to the dateutil library) so I tried spinning it up. What came out was a really nice user interface where we could see just about everything we wanted.

There were a few tweaks I wanted to make for my own purposes, but one major thing that was missing was a list of e-mail subscribers. This was surprisingly simple to throw in, and works really well since Django does pretty much all of the work for me.

Anyway, I’m hoping these changes can get pulled back into the main repository on Google Code, but if not, I’ve got a fork going on GitHub for anybody that’s interested:

http://github.com/jgeewax/django-chronograph

You can install this via easy install:

$ easy_install http://github.com/downloads/jgeewax/django-chronograph/django-chronograph-0.1…

Here’s what the fork looks like:

Agile git Workflow

Sat, 23 Feb 2013 14:10:00 -0500

When we started using git to manage our source code at work, we actually jumped in a little bit too fast. It seems like there is a lot of writing about how you can do lots of really neat things with git, but no real guide about one particular way of using git for your project. This post is going to describe how we use git day to day on a reasonably large “agile-style” project.

Overview

The general overview of how this works is:

Branch off of master to work on the feature
Work, work, work, commit, commit, commit
Pull down any updates to master
Rebase and squash (with discretion) the feature branch to master’s HEAD
Merge the feature branch back to master
Push up to the shared repository

Also in this post are details about using release-candidate and production branches and merging features across those branches.

Working on a feature

Let’s imagine that you need to work on Issue #12. The first thing you’d do is create a feature branch from master for the issue. This keeps your work isolated, so that you can switch what you’re working on very quickly should the need arise.

(master)jj@im-jj:~/demo$ git checkout -b issue-12
Switched to a new branch 'issue-12'
(issue-12)jj@im-jj:~/demo$

Now that you’re on the feature branch, you might do all sorts of work towards completing this issue:

(issue-12)jj@im-jj:~/demo$ echo "print 'hi'" >> feature.py
(issue-12)jj@im-jj:~/demo$ git add feature.py
(issue-12)jj@im-jj:~/demo$ git commit -m "Added feature"
[issue-12 765a6c0] Added feature
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 feature.py
(issue-12)jj@im-jj:~/demo$ git rm README.txt
rm 'README.txt'
(issue-12)jj@im-jj:~/demo$ git commit -m "Removed the README file"
[issue-12 e7829cb] Removed the README file
 0 files changed, 0 insertions(+), 0 deletions(-)
 delete mode 100644 README.txt
(issue-12)jj@im-jj:~/demo$ echo "this is a test file!" >> testfile.txt
(issue-12)jj@im-jj:~/demo$ git add testfile.txt
(issue-12)jj@im-jj:~/demo$ git commit -m "Added testfile.txt"
[issue-12 f9753df] Added testfile.txt
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 testfile.txt
(issue-12)jj@im-jj:~/demo$ echo "print 'hi again'" >> feature.py
(issue-12)jj@im-jj:~/demo$ git add feature.py
(issue-12)jj@im-jj:~/demo$ git commit -m "Updated feature"
[issue-12 4e39fc7] Updated feature
 1 files changed, 3 insertions(+), 1 deletions(-)

Getting feedback on your feature

If you want to share all that work with someone else as a patch (maybe for code review or something), that is pretty simple:

(issue-12)jj@im-jj:~/demo$ git diff master

This is really saying, “compare what’s currently on my branch with what’s on master and print it to standard out.

You can redirect the output of that to a file (ie, git diff master > issue12.diff) if you need.

Packaging up your work for the shared repository

When you’re ready to share this feature with the rest of your team, they probably won’t care about all your interim commits (for example, you added and then changed feature.py). We can package up all the commits into one big commit (sort of like the one big patch against master you used before) by using git’s rebase command.

(issue-12)jj@im-jj:~/demo$ git rebase -i master

The -i flag is “interactive” which allows you to specify what you want git to do with each of those interim commits. It should present a text editor that looks like this:

pick 765a6c0 Added feature
pick e7829cb Removed the README file
pick f9753df Added testfile.txt
pick 4e39fc7 Updated feature

# Rebase 309291d..4e39fc7 onto 309291d
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

To pack all of these commits into a single commit, change all but the top one to s or squash. This says “keep the change from commit, but roll it into the parent commit”. The default as you can see is “pick” which if left alone would leave the commits separate. If for some reason you want to “undo” a commit that you already had, you can remove that line entirely. This makes will remove entirely the changes introduced in that particular commit.

Keep in mind that you might not want to package everything as a single commit, and anything that you want to keep separate, just leave that line as pick and it will be left alone. For the purposes of this article, we’re assuming that each branch is a completely isolated topic, where the intermediate commits are for your personal use, and not of any use to the rest of your team. This may not always be the case.

To “package” up the branch into a single commit, our editor would look like this:

pick 765a6c0 Added feature
s e7829cb Removed the README file
s f9753df Added testfile.txt
s 4e39fc7 Updated feature

# Rebase 309291d..4e39fc7 onto 309291d
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

Once you save that file, you’ll be prompted for a message to go along with the new “re-packaged” commit. Usually it’s best to start fresh from there and go with something about the feature, the prompt I got was:

# This is a combination of 4 commits.
# The first commit's message is:
Added feature

# This is the 2nd commit message:

Removed the README file

# This is the 3rd commit message:

Added testfile.txt

# This is the 4th commit message:

Updated feature

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Not currently on any branch.
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       deleted:    README.txt
#       new file:   feature.py
#       new file:   testfile.txt
#

Which you could change to:

Issue 12 - Added new feature XYZ

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Not currently on any branch.
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       deleted:    README.txt
#       new file:   feature.py
#       new file:   testfile.txt
#

Keeping the shared repository linear

At this point, the “issue-12” branch consists of one single commit. However, if somebody else has pushed any changes to the shared repository, merging in the feature branch will have two commits: one for the feature itself, and another to merge the feature with whatever else was pushed by the rest of the team.

All those “merged my local branch” commits will look like a lot of noise for no good reason, so git allows you to “replay” your commits against the master repository, with the end goal being to make your “re-packaged” commit to appear as though you branched right off the most updated version of the code.

This is sometimes referred to as “changing history” because you are updating the parent of your current commit to be further along in the future than it actually was.

To illustrate how this might work, you can make a small commit on master, and then use rebase to replay the commit against the updated master repository.

(issue-12)jj@im-jj:~/demo$ git checkout master 
Switched to branch 'master'
(master)jj@im-jj:~/demo$ echo "there was a typo" >> fixed_typo.txt
(master)jj@im-jj:~/demo$ git add fixed_typo.txt
(master)jj@im-jj:~/demo$ git commit -m "Added fixed typo"
[master c348893] Added fixed typo
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 fixed_typo.txt
(master)jj@im-jj:~/demo$

Now master has one extra commit, and issue-12 has one extra commit, where each has the same parent. If you were to merge issue-12 into master you’d have the two commits we mentioned before (one for the “re-packaged” feature as a single commit, and another for the recursive merge). To get rid of that second commit that really doesn’t provide much information to the team, we’ll rebase the commit to look like it happened after the typo-fixing commit we just threw into master. This also will give us the chance to resolve any conflicts.

This is one representation before we change the parent:

(master)jj@im-jj:~/demo$ git checkout issue-12 
Switched to branch 'issue-12'
(issue-12)jj@im-jj:~/demo$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: Issue 12 - Added new feature XYZ

This is one representation after the rebase changed the parent:

Now we can merge the change back into master with a fast forward:

(issue-12)jj@im-jj:~/demo$ git checkout master 
Switched to branch 'master'
Your branch is ahead of 'origin/master' by 1 commit. # <-- Always be 1 commit ahead.
(master)jj@im-jj:~/demo$ git merge issue-12 
Updating c348893..d0e9912
Fast forward
 feature.py   |    3 +++
 testfile.txt |    1 +
 2 files changed, 4 insertions(+), 0 deletions(-)
 delete mode 100644 README.txt
 create mode 100644 feature.py
 create mode 100644 testfile.txt

Now let’s double check that we can still fast-forward on the shared repository:

(master)jj@im-jj:~/demo$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 2 commits. # <-- 1 for issue-12, 1 for the typo-fix.
#
nothing to commit (working directory clean)
(master)jj@im-jj:~/demo$ git remote show origin 
* remote origin
  URL: my-server:/git/demo.git
  HEAD branch: master
  Remote branch:
    master tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (fast forwardable) # <-- Always be able to fast-forward.

Since we can fast forward, it’s just a git push away:

(master)jj@im-jj:~/demo$ git push origin
Counting objects: 8, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (7/7), 699 bytes, done.
Total 7 (delta 0), reused 0 (delta 0)
To my-server:/git/demo.git
   309291d..d0e9912  HEAD -> master

And since we don’t need that issue branch anymore, we can delete it:

(master)jj@im-jj:~/demo$ git branch -d issue-12 
Deleted branch issue-12 (was d0e9912).

What if other people push stuff in the meantime?

If somebody else happens to push some code in the time between your rebase and your git remote show origin (you’ll know this because the push won’t be “fast forwardable”) you should first pull down any changes, and then do another rebase. All this will do is (again) replay your commits ontop of the new ones that got pushed so that the shared repository doesn’t have lots of noise from merging your feature branch. Since you’ve already got the commit packaged up the way you like, you can leave off the -i and just do:

(issue-12)jj@im-jj:~/demo$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: Issue 12 - Added new feature XYZ

The moral of the story is that if you want to keep your shared repository full of only the important stuff, make sure all your pushes are fast-forwards. This is as easy as always remembering to rebase immediately before you push.

Staging and production

Now that you have the concept of feature branches down pat, another thing that we struggled with was how to deal with branches that signified different versions of our code.

Ideally we’d like to have a “trunk” (aka master) and then a branch for the latest release-candidate (say “rc-1.0”) and then the official release branch (say “1.0”).

The basic idea is to treat the release-candidate branch just like master (re-packaging commits, feature branches, etc) and then the production branch as a “merge-only” or “cherry-pick-only” branch. Some people prefer to use tags for this, however using a tag means that anything currently in testing will hold up moving things into production. This is a choice you’ll have to make, a branch gives a bit more independence from the RC than a tag.

Creating the branch

It seems like the first place to start would be to create the RC branch:

(master)jj@im-jj:~/demo$ git checkout -b rc-1.0
Switched to a new branch 'rc-1.0'
(rc-1.0)jj@im-jj:~/demo$ git push origin rc-1.0
Total 0 (delta 0), reused 0 (delta 0)
To my-server:/git/demo.git
 * [new branch]      rc-1.0 -> rc-1.0
(rc-1.0)jj@im-jj:~/demo$ git remote show origin 
* remote origin
  URL: my-server:/git/demo.git
  HEAD branch (remote HEAD is ambiguous, may be one of the following):
    master
    rc-1.0
  Remote branches:
    master tracked
    rc-1.0 tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local refs configured for 'git push':
    master pushes to master (up to date)
    rc-1.0 pushes to rc-1.0 (up to date)

Now you have a new release candidate branch that you can push changes to. Follow the same procedure discussed above to make sure that stays linear and everything should work out just fine.

This is what you’re RC branch might look like:

What if somebody else already created the branch?

Maybe someone else on your team already created the branch and you just need to check it out locally. This is pretty simple in git. The format is git checkout -b mybranch -t origin/mybranch. For example, if someone already created the rc-1.0 branch, we could pull it down like this:

(master)jj@im-jj:~/demo$ git checkout -b rc-1.0 -t origin/rc-1.0 
Branch rc-1.0 set up to track remote branch rc-1.0 from origin.
Switched to a new branch 'rc-1.0'
(rc-1.0)jj@im-jj:~/demo$ git remote show origin 
* remote origin
  URL: my-server:/git/demo.git
  HEAD branch: master
  Remote branches:
    master tracked
    rc-1.0 tracked
  Local branches configured for 'git pull':
    master merges with remote master
    rc-1.0 merges with remote rc-1.0
  Local refs configured for 'git push':
    master pushes to master (up to date)
    rc-1.0 pushes to rc-1.0 (up to date)

Merging between branches

If everything is going fine, there will come a point when it’s time to merge all the changes you made on your RC branch back to the master branch. This is nothing more than a git merge. (Note that this is one place where you won’t be fast-forwarding, but always should have a merge commit to show that you merged the RC branch back to master.)

(rc-1.0)jj@im-jj:~/demo$ git checkout master
Switched to branch 'master'
(master)jj@im-jj:~/demo$ git merge rc-1.0 
Merge made by recursive.
 mynewfile.txt  |    1 +
 mynewfile2.txt |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 mynewfile.txt
 create mode 100644 mynewfile2.txt
(master)jj@im-jj:~/demo$ git push
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 331 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
To my-server:/git/demo.git
   af218a9..b4c7ac5  HEAD -> master

After things are merged back, your repository will look something like this:

Ready to release

When your release candidate has made it through all the testing it needs, you should be ready to create a production branch. This is just like the previous example where all you need to do is create the remote branch off of the RC branch.

(master)jj@im-jj:~/demo$ git checkout rc-1.0 
Switched to branch 'rc-1.0'
(rc-1.0)jj@im-jj:~/demo$ git checkout -b 1.0
Switched to a new branch '1.0'
(1.0)jj@im-jj:~/demo$ git push origin 1.0
Total 0 (delta 0), reused 0 (delta 0)
To my-server:/git/demo.git
 * [new branch]      1.0 -> 1.0

Usually this branch will stay up to date with the RC branch, and as bugs are fixed in the RC they are just merged over into the production branch.

(1.0)jj@im-jj:~/demo$ git checkout rc-1.0 
Switched to branch 'rc-1.0'
(rc-1.0)jj@im-jj:~/demo$ echo 'print "fixing a bug"' >> feature.py
(rc-1.0)jj@im-jj:~/demo$ git add -A && git commit -m "Added bug fix"
[rc-1.0 52a6d49] Added bug fix
 1 files changed, 1 insertions(+), 0 deletions(-)
(rc-1.0)jj@im-jj:~/demo$ git push
Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 338 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
To my-server:/git/demo.git
   97a0f2a..52a6d49  HEAD -> rc-1.0

And once somebody verifies that the bug is then fixed:

(rc-1.0)jj@im-jj:~/demo$ git checkout 1.0 
Switched to branch '1.0'
(1.0)jj@im-jj:~/demo$ git merge rc-1.0 
Updating 97a0f2a..52a6d49
Fast forward
 feature.py |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
(1.0)jj@im-jj:~/demo$ git push
Total 0 (delta 0), reused 0 (delta 0)
To my-server:/git/demo.git
   97a0f2a..52a6d49  HEAD -> 1.0

The same thing applies when you want to pull back this new change into master:

(1.0)jj@im-jj:~/demo$ git checkout master
Switched to branch 'master'
(master)jj@im-jj:~/demo$ git merge rc-1.0 
Merge made by recursive.
 feature.py |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
(master)jj@im-jj:~/demo$ git push
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 285 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
To my-server:/git/demo.git
   b4c7ac5..4834481  HEAD -> master

What if I just want one commit?

There may be times when the production branch needs an important fix, but there are other features on the branch that haven’t been adequately tested. This problem is pretty well solved by “cherry-picking”. Merging won’t work because a merge carries with it all the parent commits until a common ancestor, which you definitely don’t want if those parent commits haven’t been tested.

The basic idea cherry-picking is that git will take the change-set, package it up as a different commit, and move it over to another branch, without taking anything else.

Let’s make two small bug fixes on rc-1.0, and then take the last one over with a cherry pick.

(master)jj@im-jj:~/demo$ git checkout rc-1.0 
Switched to branch 'rc-1.0'
(rc-1.0)jj@im-jj:~/demo$ echo '# bugfix1!' >> feature.py 
(rc-1.0)jj@im-jj:~/demo$ git add -A && git commit -m "Added bugfix number 1"
[rc-1.0 92ab376] Added bugfix number 1
 1 files changed, 1 insertions(+), 0 deletions(-)
(rc-1.0)jj@im-jj:~/demo$ echo '# bugfix2!' >> feature2.py 
(rc-1.0)jj@im-jj:~/demo$ git add -A && git commit -m "Added bugfix number 2"
[rc-1.0 1dec2d3] Added bugfix number 2
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 feature2.py
(rc-1.0)jj@im-jj:~/demo$ git push
Counting objects: 8, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 629 bytes, done.
Total 6 (delta 3), reused 0 (delta 0)
To my-server:/git/demo.git
   52a6d49..1dec2d3  HEAD -> rc-1.0

If we just wanted to merge over bug fix number 2, the merge would also carry bug fix number 1 along with it since that’s the common parent. We haven’t tested bug fix number 1, so we’ll need to cherry pick number 2 over onto the 1.0 branch by itself.

(rc-1.0)jj@im-jj:~/demo$ git log --pretty=oneline --abbrev-commit
1dec2d3 Added bugfix number 2 # <-- This is the commit we want, note the short-hash.
92ab376 Added bugfix number 1
52a6d49 Added bug fix
97a0f2a Minor commit
099214d Minor commit
d0e9912 Issue 12 - Added new feature XYZ
c348893 Added fixed typo
309291d Initial Commit
(rc-1.0)jj@im-jj:~/demo$ git checkout 1.0 
Switched to branch '1.0'
(1.0)jj@im-jj:~/demo$ git cherry-pick 1dec2d3
Finished one cherry-pick.
[1.0 8e3a5f9] Added bugfix number 2
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 feature2.py
 (1.0)jj@im-jj:~/demo$ git push
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 289 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
To my-server:/git/demo.git
   52a6d49..8e3a5f9  HEAD -> 1.0

You now have the change-set from bug fix number 2, but it’s technically a fully different commit (with its own commit hash and everything). This allows you to move things over from the RC branch independently of other things in the branch. Also, when you do decide to merge over the rest of the RC branch, that shouldn’t be a problem at all since I guess git knows that the new commit has the same content and just a different parent.

Help, I broke things and want to start over

Another common thing I find myself doing all the time is having to “try again” after I accidentally merge over the wrong commit, or make a change that just isn’t right. This is where the git reset command comes in handy.

If you just committed before you were ready, the standard git reset HEAD~1 will work just fine. This is git-speak for “rewind things by 1 commit, but don’t change the contents of the files on disk”.

If you do want to undo any changes to files locally, you can use the --hard flag to change that as well: git reset --hard HEAD~1.

The HEAD~1 is notation for the head of the current branch minus 1 commit. So HEAD~4 would mean “four commits ago”.

You can also point out a specific commit by it’s short hash to rewind to that point in time. And remember the --hard flag will bring you back to that point with the exact code from then as well.

(1.0)jj@im-jj:~/demo$ git log --abbrev-commit --pretty=oneline
8e3a5f9 Added bugfix number 2
52a6d49 Added bug fix
97a0f2a Minor commit
099214d Minor commit
d0e9912 Issue 12 - Added new feature XYZ
c348893 Added fixed typo
309291d Initial Commit
(1.0)jj@im-jj:~/demo$ git reset --hard 52a6d49
HEAD is now at 52a6d49 Added bug fix
(1.0)jj@im-jj:~/demo$ git log --abbrev-commit --pretty=oneline
52a6d49 Added bug fix
97a0f2a Minor commit
099214d Minor commit
d0e9912 Issue 12 - Added new feature XYZ
c348893 Added fixed typo
309291d Initial Commit

I accidentally started my work on master…

Sometimes I’ll make a couple of small changes on master and one thing leads to another and all the sudden I realize I’ve got a lot of changes right on master instead of a feature branch. Let’s get into that situation real quick:

(rc-1.0)jj@im-jj:~/demo$ git checkout master
Switched to branch 'master'
(master)jj@im-jj:~/demo$ touch myothernewfile.txt
(master)jj@im-jj:~/demo$ git add myothernewfile.txt
(master)jj@im-jj:~/demo$ git commit -m "Added another new file"
[master 617f46d] Added another new file
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 myothernewfile.txt
(master)jj@im-jj:~/demo$ touch onemorefile.txt
(master)jj@im-jj:~/demo$ git add onemorefile.txt
(master)jj@im-jj:~/demo$ git commit -m "Added one more new file"
[master 23290ed] Added one more new file
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 onemorefile.txt

OK, now we are on master, with two commits that we accidentally did right on master instead of a feature branch. The way we’ll get these onto a feature branch, and get master back to normal is with by branching and resetting.

First, we’ll create a branch from master which will have those two commits, then we’ll reset our master branch back to before we accidentally made those commits.

(master)jj@im-jj:~/demo$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 2 commits.
#
nothing to commit (working directory clean)
(master)jj@im-jj:~/demo$ git branch feature1
(master)jj@im-jj:~/demo$ git reset --hard origin/master 
HEAD is now at 4834481 Merge branch 'rc-1.0'
(master)jj@im-jj:~/demo$ git status
# On branch master
nothing to commit (working directory clean)

This keeps your changes off on the “feature1” branch, and then moves the “master” pointer back to the master pointer on the shared repository. If you checkout the feature1 branch, you’ll see that the changes you made originally on master are in place over there, and you can easily merge them back into master when the time comes to commit them.

That’s all

That’s all I have for now. If there’s anything missing that you’d like to see feel free to e-mail me at jj@(this website).

Extra thanks to Mark Chadwick for being the inspiration behind this post. Also if you’re going to be working with lots of git repositories and lots of different branches, see my previous post (http://geewax.org/2009/11/15/git-workspace-magic.html) for how to make your PS1 update based on what branch you’re currently on.

Other discussions

There are a few discussions going on over at Reddit and HackerNews, some of the feedback has been incorporated into the article since then, but hopefully the comments on these forums can help you to adapt this workflow to best suit the needs of your team.

You can find these discussions here:

git Workspace Magic in bash

Sat, 23 Feb 2013 13:56:40 -0500

I have lots of projects in PyDev in Eclipse all of which I navigate through with the terminal in Ubuntu (this blog’s source is one of them), so just recently with the help of a couple co-workers I put together some bash methods for moving around my projects easily.

I found that the common pattern was:

jj@im-jj:~$ cd workspace/dashboard/src/
jj@im-jj:~/workspace/dashboard/src$ source environment/bin/activate
(environment)jj@im-jj:~/workspace/dashboard/src$ deactivate 
jj@im-jj:~/workspace/dashboard/src$ cd ../../reportingdb/src/
jj@im-jj:~/workspace/reportingdb/src$ source environment/bin/activate
(environment)jj@im-jj:~/workspace/reportingdb/src$

The first thing I wanted was to know which project I was in and which branch in git that I was on. With a couple of helper methods this is actually pretty easy and just involves overwriting the PS1 environment variable.

# Git stuff:
function get_repository {
    pwd | grep $HOME'/workspace/.*/src.*' | awk -F/ '{print "("$5")"}'
}
function get_branch {
    git branch 2> /dev/null | grep \* | awk '{print "("$2")"}'
}

# Update PS1
PS1="\[\033[01;34m\]\$(get_repository)\[\033[31m\]\$(get_branch)\[\033[37m\]\[\033[00m\]\[\033[38m\]\u@\h:\w$ "

Now my prompt lets me know pretty acurately where I am in my source:

(environment)(dashboard)(master)jj@im-jj:~/workspace/dashboard/src$ git branch test
(environment)(dashboard)(master)jj@im-jj:~/workspace/dashboard/src$ git checkout test
Switched to branch 'test'
(environment)(dashboard)(test)jj@im-jj:~/workspace/dashboard/src$

The next thing I decided to add in was a command called `jumpto` which would deactivate whatever environment I’m in, jump into the proper directory, and then activate the current virtual environment for me to start working. Also, I wanted just `jumpto` with no arguments to bring me back to my home directory and exit out of any virtual environments.

The end result of this was the following:

function jumpto {
    if [ -z "$1" ]; then
        deactivate 2>/dev/null
        cd $HOME
        return 0
    fi

    local DIRECTORY=$HOME/workspace/$1/src

    if [ -d $DIRECTORY ]; then
        deactivate 2>/dev/null
        cd $DIRECTORY
        if [ -d $DIRECTORY/environment ]; then
            source $DIRECTORY/environment/bin/activate
        fi
    else
        echo "Invalid workspace! ($1)"
    fi
}

This made moving around pretty easy:

jj@im-jj:~$ jumpto dashboard
(environment)(dashboard)(master)jj@im-jj:~/workspace/dashboard/src$ jumpto reportingdb
(environment)(reportingdb)(master)jj@im-jj:~/workspace/reportingdb/src$ jumpto 
jj@im-jj:~$

The next thing I wanted was to do tab-completion based on the projects that I had set up in my `~/workspace/` directory. After some research on how tab completion works in the first place, I ended up with the following:

_jumpto() {
    local IFS=$'\t\n'
    local dirs=("$HOME/workspace/")

    COMPREPLY=( $(
        for dir in "${dirs}"; do
            cd "$dir" 2 >/dev/null &&
            compgen -d -- "${COMP_WORDS[COMP_CWORD]}"
        done
    ) )
    return 0
}

complete -o filenames -o nospace -F _jumpto jumpto

This lets me do things like `ju[tab] das[tab]` and end up in the dashboard project with the virtual environment all set up and ready to go.

I hope this helps somebody else out who’s working with lots of git projects with lots of different branches and virtual environments for each of them.

Creating iPhone Ringtones on Ubuntu

Sat, 23 Feb 2013 13:54:34 -0500

I just had a hell of a time trying to turn a very short MP3 file into a ringtone for my iPhone. I ended up using Perl Audio Converter and faac (Freeware Advanced Audio Coder) to turn the MP3 file into an m4a. After that I renamed as m4r and dragged onto the iPhone.

The short version boils down to:

$ sudo apt-get install pacpl faac
$ pacpl --to m4a yourfile
$ mv yourfile.m4a yourfile.m4r

After that, just drag the m4r file onto the iPhone device in iTunes.

Easy-Installing SOAPpy 0.12.0

Sat, 23 Feb 2013 13:53:48 -0500

I was recently working with virtualenv by Ian Bicking (which is awesome) and I ended up having quite a bit of trouble getting SOAPpy v 0.12.0 installed. It’s a standard EasyInstall package, so it should be no sweat, but it seems like SOAPpy (a deprecated package I believe…) has a few problems which appear to be “packaging” related.

For example, the package does a couple of from __future__ import ... calls, which in Python 2.5, must be the first line of the file. This seems fine, except the author(s) (I assume when they were packing the library up to share with the world) added copyright notices at the top of files as strings instead of comments. This means that importing the files (which EasyInstall does) will throw an Exception about the placement of the __future__imports.

There was also an issue with dependencies, where SOAPpy clearly requires the fpconst library, but doesn’t declare the dependency in it’s setup.py file. This means that you install SOAPpy and then run into an ImportError for a package that EasyInstall should just grab for you.

To remedy these things I repackaged SOAPpy 0.12.0 with the fixed __future__ imports and the proper dependency declarations. In case anyone else has a problem similar to this, you can find the file at http://static.geewax.org/SOAPpy-0.12.0.zip

If you want to use EasyInstall to do the work, you can do it with:

sudo easy_install http://static.geewax.org/SOAPpy-0.12.0.zip

GAE Testbed Documentation Released

Sat, 23 Feb 2013 13:53:05 -0500

I’d been looking to get familiar with Sphinx for a while, and couldn’t ever seem to find a project of mine that needed documentation enough to take the leap. This past week, I decided that even though GAE Testbed is a relatively small project, it could definitely use good documentation and finally got around to working with Sphinx.

I must say that I’m really impressed with the package as a whole (running it both on my MacBook and my Windows machine), though it was a bit tricky to get set up and familiar with the layout.

That said, feel free to take a look at the new GAE Testbed documentation which is located at http://gaetestbed.geewax.org/index.html. For those of you looking to get the code for GAE Testbed , it’s located on Google Code at http://github.com/jgeewax/gaetestbed.