Why you should not use Python’s easy_install carelessly on Debian

(Hint: This article talks about the Pylons web framework which is essentially dead. But the warning about easy_install and pip is still valid.)

This article tries to make clear why blindly running setuptools/ez_setup on Debian/Ubuntu is dangerous and will happily break your operating system.

I loved the Pylons web framework. A significatnt number of developers was using it. It could be installed easily through a simple – though not sophisticated – tool called easy_install. Easy_install is a wrapper/bootstrapper around the Python setuptools.

Setuptools isn’t terribly bad per se

Pylons depends on a number of Python modules. It would be a real pain to download and install all the right versions of these modules from different web sites just to run Pylons. Fortunately there are the setuptools that download the dependencies and install them on your system. It uses the PyPi Python package index that lists information on available packages and where setuptools can get them from. If your operating system has no sane way of tracking the installed software (like Windows or UNIX flavors that do not have a package management) then this approach is fine.

Setuptools is not a decent software package management

Unfortunately most modern operating systems like Linux distributions have come with a sophisticated package management. Its job is to track what software is installed and what files belong to what package. You can install and uninstall packages (without leaving debris on your system), upgrade them all easily, install security patches, keep your configuration upon upgrades and track dependencies and conflicts. So it’s similar to setuptools but does a whole lot more than that. Setuptools does not even allow you to uninstall a module. After all we are not in the dark ages of operating systems any more where you just install software and the regular way of tidying up your operating system is by reinstalling it because it breaks after a few months anyway. I didn’t have to reinstall Debian for years on my system but installed tons of software for playing around with it and removed it later. Actually the only explanation I have why setuptools is so widespread is perhaps that many Python developers use operating systems without software management. And don’t get me started about its version numbers. Anyway setuptools is a quick’n’dirty way to install Python packages.

Apt versus Setuptools

Debian has a large number of Python modules available as Debian packages through the advanced package tool (apt). Installing Pylons is just a matter of running aptitude install python-pylons. Do you smell the trouble already? Apt does not care for Python modules installed via Setuptools and vice versa. So it’s possible to install a certain module twice in different locations. Depending on the order of your PYTHONPATH you may use different versions of the same module. Some people argue that they know which Python modules they install through “apt” and which are installed through “setuptools”. That’s a dangerous game to play. Since both know how to install dependent packages/modules you will quickly install software you are not aware of. Welcome to the mess.

Debian’s modules are always outdated

Yeah, right. That is the old rumor. People who say that have still not understood the difference of “stable” and “unstable” branches of Debian. Debian consists of pretty up-to-date packages. Whatever is available via setuptools is likely available as a Debian package within days, too. So you will not miss anything. Debian releases a “stable” version roughly every two years though. Of course there is a lot of change happening within two years. But many people favor stability over new features. They are happy that their desktops will work smoothly for two years. If you prefer the newest software then just use the “unstable” branch instead of the “stable” one. The name “unstable” is misleading. It doesn’t mean that your system will constantly be broken if you use it. It contains the newest software though that couldn’t be tested thoroughly. So my personal recommendation to Debian users is: use “unstable” if you like to stay up-to-date.

Jailed by virtualenv

If you do not like to bring your system to “unstable” and still want to run Pylons applications you can create a virtual environment (virtualenv). It will debootstrap a Python environment into a directory that does not conflict with the rest of your operating system. I wouldn’t recommend that for daily work. But it allows to use setuptools to install the newest software. And if you get fed up or do not need it any more then just remove that special directory and no harm is done. An additional advantage is that you can run different projects that each require different versions of software. Perhaps one project does only work with a package of version <= 0.3 while another project needs at least >= 0.4. In this case you just give each project what it needs.

The drawback is that you do not get security updated from “apt” so you are completely on your own. And it’s not really suitable for working on a daily basis because you hardly get access to anything outside of that jail.

So who’s fault is all this?

The actual problem is that the creator of the setuptools didn’t care for the needs of Linux distributions that come with some kind of package management. Redhat has just recently discovered they have the same problem. So it’s not just something a mad Debian developer has come up to confuse the world. Perhaps sometimes in the future we will have apt-aware setuptools or a way to automatically create Debian packages from ?PyPi modules.

What should you do then?

My personal recommendation:

  • Use “unstable” for development
  • Deploy your application in a virtual environment

Whatever you do – NEVER install a Python module system-wide with setuptools/ez_setup on your Debian system. And – no – /usr/local is not a safe place for Python modules either.

I told you so. 😉

How to create a favicon for your web site

If you like to create a tiny favicon for your web site (the little icon displayed left of the URL in your browser) there is not much you need to know:

  • apt-get install netpbm

  • Create a bitmap of 16×16 pixels (Gimp is recommended)
  • Save the bitmap as .ppm format (select "raw mode")
  • Convert the file using ppmtowinicon favicon.ppm > favicon.ico

Co-maintaining a Debian package with Git and git-buildpackage

Although I’m still not entirely comfortable with Git for revision control I partly use it to maintain my Debian packages. Mainly because hg-buildpackage has been abandoned and git-buildpackage is actively maintained. This article deals with co-maintaining a Debian package with Git and git-buildpackage.

Install Git

On the development workstation (where you build the packages – running Sid):

  • aptitude install git-core git-buildpackage pristine-tar

Make sure you set your name and email address in your ~/.gitconfig file via:

  • git config –global user.name ‘<name>’
  • git config –global user.email ‘<email>’

On the publicly available web server (where you publish your repository – Lenny or newer):

  • aptitude install git-core git-daemon-run gitweb 

Move an existing Debian source package under Git control (only if there is not yet a repository containing the package!)

Either: Import the source package into a fresh Git repository

  • git-import-dsc –pristine-tar /path/to/package*.dsc

This creates a package/ directory containing a Git repository with two branches. The default branch “master” contains the debianized source package (like what you would get after running ‘dpkg-source -x *dsc’). The other “upstream” branch contains what was in the orig.tar.gz upstream tarball. You can see the branches with ‘git branch -a’. Each upstream version gets a tag like “upstream/1.3.4” and each Debian package gets a tag like “debian/1.3.4-3”. You can see the tags with “git tag”.

If you want to import further source package versions/revisions you can either run git-import-dsc in the order of their revision numbers or use ‘git-import-dscs’.

Building the package can then be done through:

  • git-buildpackage –git-pristine-tar

The ‘pristine-tar’ is important if a previous revision of this upstream version has already been uploaded to Debian. If you don’t use pristine tars then the created orig.tar.gz may be slightly different (not in content – but as an archive) from what’s already in Debian and the ftp-master will refuse your upload. If you forgot the ‘–pristine-tar’ option then at least make sure you have the orig.tar.gz in ../ that is also in Debian.

Or: Import just the orig.tar.gz into a fresh Git repository and start debianizing

  • mkdir package-name
  • cd package-name
  • git init
  • git-import-orig /path/to/orig.tar.gz
  • …work on Debian package…
  • git add debian… (or “git add .” to add everything that changed)
  • git commit
  • git-buildpackage
  • …upload to Debian… (dput/dupload)
  • git-buildpackage –git-tag

Or: package a new upstream revision

  • git-import-orig /path/to/new.orig.tar.gz

Or: pull the existing repository from your co-maintainer

  • git clone URL (for example: git clone git://git.workaround.org/cream.git)

What your co-maintainer did is in origin/upstream (the orig.tar.gz’s content) and origin/master (the debianized directory). Incidentally Git merged the origin/master into your own ‘master’ branch that you are now on. To make sure you are following the co-maintainers upstream branch, too, you’ll have to run:

  • git checkout -b upstream –track origin/upstream

The ‘–track’ option altered the .git/config file and added a [branch “upstream”] section telling Git where you fetched it from. That means you can later just say “git pull” and you will get both the ‘master’ and the ‘upstream’ repository merged into your repository automatically.

Now you will have these branches:

  • upstream (a copy of the origin/upstream – must not change)
  • master (where you do the Debian package work in just like you had no repository underneath)
  • origin/upstream (where you co-maintainer unpackaged the orig.tar.gz using git-import-orig)
  • origin/master (the debianization that your co-maintainer did so far)
  • pristine-tar (optional – information on how to build the binary-identical orig.tar.gz that is in Debian)

Work on the Debian package

  • …work…work…work…
  • git add (what you added/changed)
  • git commit
  • git-buildpackage

If you like the package:

  • …upload to Debian… (dput/dupload)
  • git-buildpackage –git-tag

And then…

Publish your own works to your public repository so that your co-maintainer can fetch what you did

The most efficient protocol to transport Git repositories and their changes is the git:// protocol. (You could also dump your repository into a web server’s htdocs directory and let others fetch it with HTTP.) 

Using git.debian.org

The Debian project is offering Git hosting at git.debian.org. You can request a new repository there through a process defined in the Debian wiki.

Using Github

Github is a nifty web service that allows hosting of Git repository. While it’s debatable that Git is actually decentralized and a central repository doesn’t make sense it’s still nice to have a place to pull and push changes. (Of course this applies to git.debian.org as well.) Repositories up to 100 MB that are public are free (beer). They also offer paid plans.

Setting up git-daemon

git-daemon is a software to serve repositories through the git:// protocol. It looks for repositories in /var/cache/git on your public (web-) server. Put up your Git repository as a “bare” repository there. (“Bare” Git repositories just consists of what’s usually in the project’s .git directory but the directory is called project.git)

This will create a copy of your repository on the ‘myserver’. “–mirror” makes it push all the branches (so you don’t miss to push the “upstream” branch) and tags (without upstream/* and debian/* tags git-buildpackage would be very unhappy).

Log into ‘myserver’, go into /var/cache/git/myproject.git and touch the file “git-daemon-export-ok” there. It tells git-daemon that this repository is allowed to be displayed publicly. (I personally prefer to change /etc/sv/git-daemon/run and add the “–export-all” option there. Because if I clone my repository into /var/cache/git I know what I’m doing and don’t accidentally put up something private there.)

To simplify pushing I recommend:

Then you can just:

  • git push

…in the future and get all your commited changes online to the public server.

Now tell your co-maintainers about it so they can “git pull git://myserver/myproject.git” from your server. Make sure you always “git pull” before you start working on the package.

Setting up gitweb (optional)

If you want to show what repositories you are offering you can run ‘gitweb’ on your web server. Just “aptitude install gitweb” on the server and follow the instructions in /usr/share/doc/gitweb/README*

See also

© 2021 workaround.org - Proudly powered by theme Octo