Getting started with tmux

As a long-time user of GNU screen, I knew it would be difficult to switch, but I’ve made the move and am so far pretty happy. tmux does all the same things that screen does, but was a lot easier to customize. I wanted to share my tmux configuration file and other things that made the switch all that much more amazing.

Keyboard bindings

I changed the default keyboard bindings to be more like screen:

• Ctrl-b? Forget that. Ctrl-a all the way.
• Ctrl-(left/right) arrow to move between windows? Hell yes.
• Ctrl-a twice to go “back”? Perfect.
• Ctrl-a, r to reload the config file made it very easy to debug.

Status bar at the bottom

I made my status bar have the session name and my machine name on the far left, the various tabs centered in the middle, and the system time and date on the far right. It was very easy to configure, and I’m guessing I’ll end up tweaking it from time to time.

Project-specific configurations (and attach tool)

Here’s the coolest part in my opinion. I forked a repository called “tat” that does some nifty things to make project-specific configurations really easy to set up consistently, and attach to them later on. Oh, and there’s tab completion for existing sessions as well as for your project directory names. Check out my fork to see the details.

With tat you can create a .tmux file in your project directory, which be run with the first argument being the session name. You can use this to configure your project workspace easily and consistently. Here’s one of my example .tmux project files.

Tab completion

Don’t forget to enable tab completion. On Ubuntu I did the following:

$ sudo curl https://raw.github.com/mithro/rcfiles/master/bash/completion/tmux > /usr/share/bash-completion/completions/tmux

Python Dictionary Comprehensions

I’m a little embarrassed to say that I haven’t yet used these in my own code — they seem so obvious now that I look back on it.

>>> d = {'a': 1, 'b': 2', 'c': 3}
>>> keys = ['a', 'c']
>>> { k: d[k] for k in keys }
{'a': 1, 'c': 3}

Task Queue support in App Engine’s ext.Testbed

A while back I wrote some (now deprecated) code that allowed you to easily test your Python App Engine applications and their interaction with the App Engine APIs. When we got acquired by Google, I started working with some of the awesome App Engine engineers on making that code part of the official App Engine codebase.

We launched that, but one of the APIs that was noticeably lacking helper-methods was the Task Queue. I added one of the old methods (get_filtered_tasks()) but a bit later I noticed a pretty serious bug where timezones weren’t handled properly.

So that code hid quietly in the App Engine code base, undocumented and technically broken. Sorry :(

I wrote some extra code to fix it, but that didn’t get merged into the main repository for quite a while.

But now it’s here.

If you’re curious about the change, feel free to check out the diff (scroll to the bottom). The method is still called get_filtered_tasks() and takes several arguments as “filters” for various properties of tasks — and it should properly handle timezones when you set the eta or countdown properties when creating tasks.

Sorry for the delay. And enjoy.

PS: Since this isn’t documented on the official App Engine page, the API may change. I don’t intend to change it, but I can’t promise it won’t. Apologies in advance if you use this helper method and have it break out from under you.

Also, check out the official code.

Some example code:

import unittest
from google.appengine.api import taskqueue
from google.appengine.ext import testbed


class TaskQueueTestCase(unittest.TestCase):

  def setUp(self):
    self.testbed = testbed.Testbed()
    self.testbed.activate()
    self.testbed.init_taskqueue_stub()
    self.task_queue_stub = self.testbed.get_stub(testbed.TASKQUEUE_SERVICE_NAME)

  def tearDown(self):
    self.testbed.deactivate()

  def testTaskAdded(self):
    taskqueue.add(url='/path/to/task')

    tasks = self.taskqueue_stub.get_filtered_tasks(url='/path/to/task')
    self.assertEqual(1, len(tasks))
    self.assertEqual('/path/to/task', tasks[0].url)

unittest.main()

Daily Meeting

TLDR: I re-built one of my old side projects. Check it out at dailymeetingapp.com

A long time ago, when I was working with a bunch of awesome guys at Invite Media, we started expanding our engineering team quite a bit. We had two engineering offices (New York City and Philadelphia) that were each doing more and more stuff, which was awesome, but we had a problem:

our daily “stand-up” meeting in two cities was kind of a pain.

To clarify what I mean by this, when I say “stand-up meeting” I mean that we had one meeting, every day atnoon, where we went around the room and told the rest of the team what we did since yesterday, and what we’re doing for the rest of the day. It was some kind of agile-ish thing that our team had deteriorated into. (There are lots of ways people do “Scrum” or “Agile” or “Hip New Methodology”, but let’s not get into that. )

This sounds like an easy meeting, right? 30 seconds per person. Boom. Done.

What about when you have this with two groups of people in different cities?

Early on we tried Skype. That was horrible. Laptop microphones really aren’t made for group conference calls. The camera wasn’t really all that great either.

Next we had a coordless phone. We would put that on speaker-phone and pass it around through the group. The audio was a bit better, but it was still pretty bad.

Then we upgraded to those fancy tripod-looking conference room phones, which were a lot better. Still not quite like being in-person, but they got the job done just fine. One problem solved.

The next issue we had to tackle was that most people would show up to this meeting, and not remember what they did yesterday! “I, um…. worked on… um… what did I do… hmm… one second…. oh yea! I built this awesome thing that does …. “

For a 30-second-per-person meeting, that is a lot of wasted time! What we needed was a reminder for everyone to actually think beforehand about what they did, so they could show up prepared for this meeting.

We can’t force everyone to set an alarm, but even if we did, how would we know if they actually thought out what they were working on? What if we had a little app that sent a reminder e-mail at 11:30 to everyone, and they could reply with what they did for everyone to see?

Sounds like a pain in the ass, right? It was. But it did have some benefits.

1. It didn’t get in the way as much as you’d think. Reply to the e-mail and that’s it. You’re done.
2. It was easy for everyone to see which people didn’t post an update. Little red dot next to their name.Busted.
3. People who missed our stand-up meeting (travel, doctor’s appointment, whatever) could come in later and see who did what. Sweet.
4. People who checked beforehand could show up prepared with a question or two based on what they read.
5. Everyone got a digest e-mail at noon with everyone’s notes. Super easy to notice the guy who says every day that he’s going to finish Feature X. What’s going on there…?

Eventually, the app morphed into something that was more about a digital (and shared) work-log than an “online stand-up meeting”, which was pretty cool for me.

Recently, I rebuilt quite a bit of the app and started testing it out with some people — who seem to really like it. It’s still not completely done (as you can see from some lorem-ipsum on the landing page….), but I would love to get some constructive feedback (and feature requests).

Check it out at http://www.dailymeetingapp.com/

Login required should be the default

If you’re building a web-app in Python that requires people to sign in, you are probably pretty familiar with the concept of a @login_required decorator. For those that might not be, it’s pretty straight forward: the decorator wraps a handler method with a check for whether the current request is coming from someone who has been through your sign in process.

Friendly URL routing in App Engine using webapp2

I used to do crazy things like hard-code URLs in my templates:

<a href="/accounts/create/">Create your account</a>

or

<a href="/accounts/view/?id={{ the_id }}">View account {{ the_id }}</a>

Ever since webapp2 came around, I’ve been really into the URL routing helpers, which make it much easier to keep long lists of URLs organized. I started using a new project layout that makes routing really simple — maybe even fun.

GAE Testbed moving into the Python SDK!

Since we were acquired by Google, I figured that it would be a fitting 20% project to work with some App Engine engineers to move GAE Testbed into the App Engine Python SDK. The benefits are pretty obvious:

1. No third-party package to install (ie, no more easy_install gaetestbed)
2. “Official” documentation for how to write your tests
3. More people working on making it easy for users to test their code
4. Many maintainers, meaning fewer regressions (ie, no more “whoops, the API changed a bit… GAE Testbed doesn’t work right now…”)

And I’m incredibly happy to say that that the 1.4.3 release includes the first steps toward no longer needing GAE Testbed at all: google.appengine.ext.testbed! The testbed module sets up your environment so that you’re able to verify that API calls you expected to be made are actually made.

For example, you can put something in memcache and then get it back to out to verify your application logic. Of course, this isn’t using an actual memcache instance, but looks and acts mostly the same.

I had very little involvement with the code in this release (just a reviewer), but there’s plenty more to come in the future. Many thanks to Robert Schuppenies and Guido van Rossum for helping get this huge improvement out the door.

P.S. Although there isn’t yet a way to do things like self.assertEmailSent(to=”email@example.com”), I hope that feature parity between google.appengine.ext.testbed and GAE Testbed is only a short while away.

Encoding subtitles into your movie…

So, you have an AVI file and a SRT file, and you want to put them together so that you don’t have to carry around the two files… (and you’re on Ubuntu…)

$ sudo apt-get install mencoder
$ mencoder -oac copy -ovc lavc -sub TheSubtitlesFile.srt \
  -subfont-text-scale 3 -o TheMovieWithSubtitles.avi \
  TheOriginalMovie.avi

(Of course you could add change “-ovc lavc” to “-ovc divx”, all up to you…)

My git workflow — in code

There have been a lot of blog posts (one of mine included…) that talk about using feature branches to have an “agile” workflow while you code. This is all great to talk about but it seemed to me that a lot of people weren’t doing it. Not because they were against the idea but because it was too tricky to remember or too complicated to do or too annoying to do for that one little feature.

Then there are tools like git flow that do the trick but they seemed like kind of a lot of typing to me, so I threw something together that was three aliases to help make my workflow involve less typing, less remembering, and less confusion.

Note: If you ask me, “what git workflow tool should I use?” my answer will be git flow. git flow is FAR better documented, supported, and tested than the tool that I threw together today…

Now… it is with great pleasure that I introduce git-feature. It’s simple to install (all three commands are git aliases — you know, those things in .git/config?), simple to remember (if you forget arguments, it will prompt you for them), mostly safe (it prompts you before doing anything that changes your repository), and has very little magic (those git aliases are just shell commands, think of this as a few friendly shortcuts…).

So, getting right down to the dirt… here’s how I currently use it:

$ git branch
* master
$ git feature my-new-feature
this will create a feature branch my-new-feature to be merged into master (Y/n): Y
$ git branch
* features/my-new-feature
  master
$ # Do some work here, commit it...
$ git finish
this will integrate my-new-feature into master (Y/n): Y
$ git branch
* master

Deploying code on App Engine? Be careful with your indexes…

Have you ever deployed a new version of your application on App Engine and end up with this rude error saying something like, “You need an index to run that query”? So you scratch your head for a minute, check index.yaml, and sure enough that index is there just like you thought… After all, most (if not all) of your index.yaml is auto generated by the SDK…

It looks like the SDK is so good about keeping the contents of index.yaml off your mind that you forget about the time it takes to build those indexes!

I’m sure it’s written down somewhere but those of us that are so eager to get code out the door are clearly not the ones reading the fine print in the documentation… The short version of all of this is…

When you have modifications to index.yaml (adding a new index for example), it takes time for App Engine to build those indexes. However, the code you deployed is running right away and that code needs those indexes right away. So you end up with a nasty mean error all because you were a bit too hasty with kicking out your code (and the SDK is too busy pushing your files up to Google’s servers to warn you: “Hey wait a minute! This index is going to take a while to build…”)

Anyway, I figured I’d share the simple way I avoid getting bitten by this little annoyance…

First, I use git. If you’re not using any sort of source control, you should probably start doing that now. Even if you don’t want to share your source with anyone, having a local git repository is great… “Whoops! Let me revert back to where things were 10 minutes ago…”

If you’re using git, this problem is fixed by breaking apart your changes into two pieces: the index changes and the code that needs the index changes. If you’re working on Feature X and there are changes to index.yaml (git diff index.yaml), I commit those changes first (git add index.yaml && git commit -m “Updating indexes”) and then stash the other stuff on a new branch (git checkout -b feature-x && git add -A && git commit -m “Feature X”). Then switch back to your previous branch (git checkout master) and deploy JUST those index changes (make deploy*) and then wait!

Go into the Data store Indexes page on the App Engine admin console and keep an eye on that page. It will show your new indexes building and queued and all that lovely stuff (which could take a while). 

Once you see that your new index is “serving”, just merge in your other branch (it should be a simple fast-forward) (git merge feature-x) and deploy again (make deploy). And you’re all set!

I’m sure there are other ways of doing this so don’t take this as the only way to handle the problem, but this way seems to work pretty well for me.

Happy deploying (and indexing).

* Deploying with “make deploy” is thanks to AppMake. It makes deploying really simple…