Emacs for Python Programmers #2: Virtualenv, IPython & Daemon Mode

In the previous post of this series, I made an attempt at a quick, very basic introduction to using Emacs for programming in Python. We covered the basics of setting up an editing mode and doing some simple customizations. Customizing an editing environment is pretty personal, like deciding what color to paint your bikeshed or how many buttons you like on your suits.

But there are some aspects that are common amongst many developers. When it comes to Python programming, I suspect one of these is use of a virtual environment tool like virtualenv. I love virtualenv, but I also love running my Python interpreter inside of Emacs. And I also love running the IPython enhanced Python interpreter. Our goal here is to combine Emacs, virtualenv, and IPython (if available/desired) and make them work in concert without leaving the editor.

Emacs Virtualenv Functions

First up, we need some Emacs commands that will allow us to switch around our virtual environments. I've implemented a set of Emacs Lisp functions to do this, called virtualenv.el. These functions are designed to work with the python.el editing major mode (see part 1). If you're using python-mode.el, these can easily be modified to work (often you just have to change any symbols starting with python- to py-, but dig into python-mode.el to be certain).

You can install this file into your emacs load-path and setup your python-mode hook to automatically activate the virtualenv extension. This file is relatively simple, it could probably be converted to a derived mode or possibly Emacs minor mode, but for the moment my use involves loading it in this crude manner.

(add-hook 'python-mode-hook '(lambda () (require 'virtualenv)))

Inferior Python Interpreters

When using the Python major mode (python.el) and editing a Python source file, you can run M-x python-shell to get a Python interpreter inside of Emacs. This is handy for many reasons. Often I'll be testing something in the shell and want to save the commands for my test to a buffer or file so that I can implement them as official unit tests.

Python mode will also allow you to send a region of code (C-c C-r) or even a single def or class (C-M-x) directly to the interpreter buffer. These are really handy as I often launch a Python interpreter in an Emacs buffer and send code to it periodically throughout a development session. It saves time by avoiding task switching in the operating system, launching python, etc. You can also use standard Emacs search commands to search across Python interpreter output.

Python-shell & Virtualenv

So all this sounds great, how do we make it work with virtualenv? The virtualenv.el code manages the virtual environments inside of Emacs. It works by taking advantage of several customizable variables in the Python major mode, primarily python-python-command. This is an absolute filesystem path to the interpreter python-mode will run when we call M-x python-shell.

The virtualenv-set-interpreter function updates the python-python-command variable to our virtual environment's interpreter. I'm using virtualenvwrapper, so my virtualenvs live in ~/.virtualenvs. This is the virtual environment root directory, stored in virtualenv.el under the variable virtualenv-root-dir. You will need to update this if you store your virtual environments elsewhere.

When setting the Python interpeter, virtualenv-set-interpreter calls virtualenv-get-interpreter to find the correct Python interpreter for the virtual environment. Right now this means deciding between ipython or python, depending on what is available. This allows support for virtual environments that have ipython installed, but default to regular python if not. You can set the custom variable virtualenv-use-ipython to nil and ipython will never be used.

All of the functions discussed so far are pretty low-level. What we're really interested in is virtualenv-activate and virtualenv-deactivate. These are the interactive functions for switching virtual environments. Issuing M-x virtualenv-activate will prompt you for a virtual environment to use. Your input will be appended to the root directory mentioned earlier.

The Python major mode is now reconfigured for your virtual environment. Running M-x python-shell will launch either python or ipython as appropriate and inspecting sys.path should reveal your virtualenv's path and packages. The next step is to automate this from the command line.

Virtualenvwrapper

Virtualenvwrapper is a set of extensions for virtualenv written by Doug Hellmann. In addition to automatically organizing your virtual environments, it includes several bash commands that simplify creating and working with them. These commands also come with hooks that allow us to run scripts at appropriate times during the workflow.

The hook that's most interesting for integration with Emacs is postactivate. This comes in two flavors: a global postactivate hook runs whenever a new environment is loaded and an environment specific postactivate hook that can be customized on a per-environment basis.

Doug has an awesome post about how to utilize these hooks with Emacs to switch desktops using Emacs's desktop-mode. This is a minor mode that tracks buffers on a per-project basis. It does this by storing a desktop file in your project directories that contains state information, including which files you were editing when you last worked on the project. He is using Emacs in server mode to communicate from the command line (using a postactivate hook and emacsclient) whenever a new environment is activated with workon.

We can extend this functionality to include switching our Emacs virtual environment in a very similar way. You can follow the instructions over at Doug's site if you're interested in using desktop-mode. We will employ the same technique to call our virtualenv-activate Emacs command. But first we'll take a quick look at server mode.

Emacs Server Mode

Emacs can be run in "server mode" in several ways: issuing M-x server-start, adding (server-start) to your .emacs initialization file, or by using daemon mode. Daemon mode involves running emacs --daemon from the command line or setting it up to autostart in your operating system's init scripts.

I've had some success using Mac OS X's launchd to auto-start an Emacs daemon at system start-up using GNU Emacs 23 builds from emacsformacosx.com. There is a basic set of instructions for doing this at emacswiki.org. The simpler approach, especially if you just want to try things out, is to do one of the other two options. Just remember, if you use M-x server-start or add (server-start) to .emacs, your Emacs server will disappear when you leave the editor.

With the Emacs server running, we can use the emacsclient command line tool and -e to issue arbitrary elisp commands to our open editor. It is important to use the correct emacsclient for your environment. For example, my Macbook included GNU Emacs 22.1.1 with Snow Leopard in /usr/bin/emacs and /usr/bin/emacsclient. I'm using the Cocoa version of GNU Emacs 23, however, installed in my /Applications folder. To access this emacsclient from the command line, I navigate inside of the Emacs.app file to: /Applications/Emacs.app/Contents/MacOS/bin/emacsclient.

The Emacs server is documented in the Emacs manual.

Postactivate Function

With our virtualenv functions written, virtualenv_wrapper setup and daemon mode running, we can wire up all three to activate the virtual environment in our Emacs editor whenever we type workon at the command line.

We will need to write a short helper function in our .emacs configuration that will be called every time a new environment is activated. This will take advantage of the virtualenv functions we wrote earlier and can perform any other setup we like (such as switching desktop-mode). Call this function workon-postactivate and it looks like this:

(defun workon-postactivate (virtualenv)
  (require 'virtualenv)
  (virtualenv-activate-environment virtualenv)
  (desktop-change-dir virtualenv))

To our virtual environment's postactivate hook, stored in ~/.virtualenvs/myenv/bin/postactivate by default, we just add one line:

/path/to/emacsclient -e "(workon-postactivate \"$VIRTUAL_ENV\")">/dev/null

This calls our emacs lisp postactivate function, which automatically activates the virtual environment for use in our Emacs Python shells.

I've searched all over the web for information on how to incorporate Virtualenv into Emacs like this, but the lack of any well documented approach has lead me to do it myself. It works very well for me, but I would be very interested in hearing other opinions and possibilities as to how this could be improved.

blog comments powered by Disqus