(Hint: This article talks about the Pylons web framework which is essentially dead. But the warning about easy_install and pip is still valid.)
This article tries to make clear why blindly running setuptools/ez_setup on Debian/Ubuntu is dangerous and will happily break your operating system.
I loved the Pylons web framework. A significatnt number of developers was using it. It could be installed easily through a simple – though not sophisticated – tool called easy_install. Easy_install is a wrapper/bootstrapper around the Python setuptools.
Setuptools isn’t terribly bad per se
Pylons depends on a number of Python modules. It would be a real pain to download and install all the right versions of these modules from different web sites just to run Pylons. Fortunately there are the setuptools that download the dependencies and install them on your system. It uses the PyPi Python package index that lists information on available packages and where setuptools can get them from. If your operating system has no sane way of tracking the installed software (like Windows or UNIX flavors that do not have a package management) then this approach is fine.
Setuptools is not a decent software package management
Unfortunately most modern operating systems like Linux distributions have come with a sophisticated package management. Its job is to track what software is installed and what files belong to what package. You can install and uninstall packages (without leaving debris on your system), upgrade them all easily, install security patches, keep your configuration upon upgrades and track dependencies and conflicts. So it’s similar to setuptools but does a whole lot more than that. Setuptools does not even allow you to uninstall a module. After all we are not in the dark ages of operating systems any more where you just install software and the regular way of tidying up your operating system is by reinstalling it because it breaks after a few months anyway. I didn’t have to reinstall Debian for years on my system but installed tons of software for playing around with it and removed it later. Actually the only explanation I have why setuptools is so widespread is perhaps that many Python developers use operating systems without software management. And don’t get me started about its version numbers. Anyway setuptools is a quick’n’dirty way to install Python packages.
Apt versus Setuptools
Debian has a large number of Python modules available as Debian packages through the advanced package tool (apt). Installing Pylons is just a matter of running aptitude install python-pylons. Do you smell the trouble already? Apt does not care for Python modules installed via Setuptools and vice versa. So it’s possible to install a certain module twice in different locations. Depending on the order of your PYTHONPATH you may use different versions of the same module. Some people argue that they know which Python modules they install through “apt” and which are installed through “setuptools”. That’s a dangerous game to play. Since both know how to install dependent packages/modules you will quickly install software you are not aware of. Welcome to the mess.
Debian’s modules are always outdated
Yeah, right. That is the old rumor. People who say that have still not understood the difference of “stable” and “unstable” branches of Debian. Debian consists of pretty up-to-date packages. Whatever is available via setuptools is likely available as a Debian package within days, too. So you will not miss anything. Debian releases a “stable” version roughly every two years though. Of course there is a lot of change happening within two years. But many people favor stability over new features. They are happy that their desktops will work smoothly for two years. If you prefer the newest software then just use the “unstable” branch instead of the “stable” one. The name “unstable” is misleading. It doesn’t mean that your system will constantly be broken if you use it. It contains the newest software though that couldn’t be tested thoroughly. So my personal recommendation to Debian users is: use “unstable” if you like to stay up-to-date.
Jailed by virtualenv
If you do not like to bring your system to “unstable” and still want to run Pylons applications you can create a virtual environment (virtualenv). It will debootstrap a Python environment into a directory that does not conflict with the rest of your operating system. I wouldn’t recommend that for daily work. But it allows to use setuptools to install the newest software. And if you get fed up or do not need it any more then just remove that special directory and no harm is done. An additional advantage is that you can run different projects that each require different versions of software. Perhaps one project does only work with a package of version <= 0.3 while another project needs at least >= 0.4. In this case you just give each project what it needs.
The drawback is that you do not get security updated from “apt” so you are completely on your own. And it’s not really suitable for working on a daily basis because you hardly get access to anything outside of that jail.
So who’s fault is all this?
The actual problem is that the creator of the setuptools didn’t care for the needs of Linux distributions that come with some kind of package management. Redhat has just recently discovered they have the same problem. So it’s not just something a mad Debian developer has come up to confuse the world. Perhaps sometimes in the future we will have apt-aware setuptools or a way to automatically create Debian packages from ?PyPi modules.
What should you do then?
My personal recommendation:
- Use “unstable” for development
- Deploy your application in a virtual environment
Whatever you do – NEVER install a Python module system-wide with setuptools/ez_setup on your Debian system. And – no – /usr/local is not a safe place for Python modules either.
I told you so. 😉
What I don’t like of easy_install is that it get’s it’s packages from urls from who-knows-were websites.
If one site for one package get’s compromised; it’s hello worms, virus and troyans for you. No thanks.
I use Debian Testing, it’s the best of both worlds, it’s not so up to date as unstable, but it’s pretty stable for home use and web development.
Cheers!
Yes, I agree with you. Debian “testing” (currently “squeeze”) is very up-to-date so will likely always work on the newest versions. And time and again I’m stuck trying to install dependencies via easy_install while some URLs were given incorrectly or the respective web server is down. After all the Python packages and modules are not installed in a central place but PyPi just points to the actual web servers where the software is hosted. It’s common to be on-site at a client trying to deploy a software and getting stuck here.
But there is a problem: you may run into when relying on “testing”: imagine you wrote a Pylons application using SQLAlchemy 0.4. And you are trying to run that on Debian “testing” which installs SQLAlchemy 0.5. Your application just won’t run.
So the problem is in both directions. Most of the time you may want to have the newest software. But sometimes you need special versions for a certain application. And that’s where a chroot-style environment like easy_install is inevitable.
By the way Debian “testing” contains the versions from “unstable” that have been there for 10 days without getting a serious bug reported against. So if you don’t mind the 10-day delay you will get the newest software with the smallest risk. 🙂
As I understand things right, there are no security updates for unstable, right?
But as it seems testing has security updates: http://www.debian.org/security/faq#testing
So is it a good idea for an exposed server, where you do web-development, to run testing?
And what does all this mean for kernel updates? Look at an example:
http://www.debian.org/security/2010/dsa-2012
is this also available for unstable? I do not see this updates in testing or unstable???
To answer myself:
Debian/stable package linux-2.6 is fixed in stable-security.
Debian/testing not known to be vulnerable
Debian/unstable not known to be vulnerable.
http://security-tracker.debian.org/tracker/DSA-2012-1
Another piece of the answer can be found here:
http://security-tracker.debian.org/tracker/status/release/testing
http://security-tracker.debian.org/tracker/status/release/unstable
Looks like it is not a good idea to run unstable on an exposed host.
What do you think?
One thing is for sure: never use "unstable" on an exposed system unless you know what you are doing (e.g. care for security critical updates yourself even if the package maintainer does not).
"testing" is a tempting choice for web development. But even then you have to be aware that different web applications have different requirements. For example I have multiple Pylons applications running on a server. One of them needs Pylons 1.0rc1 because I use the newer features. One is still 0.9.6 and would not even run on 0.9.7. In the setuptools' "setup.py" file you can declare the exact set of dependencies you want to have. This will barely work on "testing". For a while I tried to keep all my web applications matching the "testing" package versions. That's impossible.
So my personal choice and recommendation is: for Pylons applications always use "virtualenv". It's easy to use and makes you completely independent from what is currently in Debian. You would just need the "python-dev" package and perhaps "libpq-dev" if you need to compile the psycopg2 package.
Both Debian Testing and Unstable are not supposed to be used on exposed and mission critical system simply because they provide you newer software.
Unfortunately many developers and even commercial distributions seem not to understand how come maturity is important in production environments and focus only on bleeding edge code.
The stdeb project produces Debian source packages from Python packages. In regards to the post here, you may be particularly interested in pypi-install, which will query the Python Package Index (PyPI) for a package, download it, create a .deb from it, and then install the .deb.
I've just spent an evening playing around with stdeb, it's really very nice and fits in with python development very easily. Fundamentally, you're looking at something like this:
$ python setup.py –command-packages=stdeb.command sdist_dsc
$ cd deb_dist/
There's a specific configuration file (stdeb.cfg) where you can over-ride specific configuration options. In an evening I've managed to get blogofile building and installing cleanly!
hello,
how does the setup.py file look like?
i got error in the command:
$python setup.py –command-packages=stdeb.command sdist_dsc
Well, perhaps you need to run ‘apt-file update’ first.
maybe http://packages.python.org/distribute/ ?