bor.borygmus

A programming weblog by Hao Lian. • A long walk through an angry forest. • A series of memory leaks brought on by senility.

matplotlib takes the goodness of the Matlab plotting API (and more!) and puts it in Python. It’s an astonishingly identical API. And now you want to compile it on Cygwin. You’ve scoured the Usenets. You’ve scoured your pots and pans. And it’s impossible. Or is it?

(This has been tested on matplotlib-0.98.5.2, February 21, 2009, on an absolutely up-to-date Cygwin. Expect to improvise more and more for later and later versions. Feel free to add any additional tips or questions to the comments section. This is the magic of webbogs.)

Well, it probably isn’t, since the matplotlib people can do it or otherwise it wouldn’t be in their setup.py. But things break, and manly administrators like us need to fix them.

Grabbing the source.

First, run easy_install, which will automatically find the right .tar.gz file from SourceForge. At that point, abort with Ctrl-C and copy and paste the URL. We won’t be using easyinstall because we’ll be fidgeting around in the building files.

Now choose a directory. wget and untar the file into this directory.

Normally, we’d run python setup.py install and go grab a cheesecake. Not so fast. You’ll get errors aplenty.

As proof, run python setup.py install. It will raise an IndexError in setupext.py in trying to read from your /usr/lib/tkConfig.sh in the function parse_tcl_config. It seems to expect a different format than what’s actually present. Fortunately, the guess_tcl_config function is Good Enough. All we have to do is force the function to enter the guess_tcl_config code path:

To swallow this exception, add “IndexError” to the comma-delimited list beneath that try-block. In other words, turn except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): into except (ConfigParser.NoSectionError, ConfigParser.NoOptionError, IndexError):. Exception swallowed!

Now run python setup.py install again like you did before. We’ll need to make sure we have all the dependencies, so let’s see what the library sniffer picks up. Install anything, via Cygwin, that it can’t find.

Two notable dependencies it won’t pick up: libX11-devel and pkg-config. Make sure you have these before running the library sniffer because you’ll need to save up your insanity for this build. Keep running setup.py and keep installing until you resolve all the dependencies.

An initial failure in building C extensions.

Here’s the crux of the problem with setup.py: You’ll get weird fork resource errors when setup.py tries to compile the C/C++ extensions. Happily enough, Pete Harris noticed that this doesn’t happen if you manually run gcc/g++. A mystery, one that can’t be solved with Cygwin’s rebase, and one that we don’t have time for.

A simple solution is to simply mimic setup.py’s calls to gcc from our own prompt. setup.py (distutils, to be exact) is smart enough to not recompile when it’s not necessary, which makes this fast and easy to implement. The only tricky part is getting setup.py to spit out all its gcc forks, not just the one it dies on.

To review: How do we get setup.py to tell us all the gcc commands it’s going to run, without actually running them?

Squeezing a build script out of distutils.

For that, we turn to python setup.py --dry-run build > a.txt or, shorter, python setup.py -n build > a.txt. We’re piping this to a.txt because there’s a lot of junk to remove. It should look like this. Once setup.py finishes, open a.txt in your favorite text editor, which we’ll call “Emacs”, and delete everything around the gcc/g++ chunks.

Within this chunk, there are messages to the user such as “building such-and-such” or “creating such-and-such”. Delete those. And slap a sh shebang at the top to get a file like this, which you can save to matplotlib-clean-build.sh if you so choose. Now you have a build script.

At this point, you’re eager to run the script and see all that glorious gcc output, a crimson-violet rainbow of warnings and C++ templates and library paths. But we have more to do.

Invalid file paths.

For some reason, there are paths to old files in this output, files that have since then been renamed. In particular: src/backend_agg.cpp, src/image.cpp, and src/path.cpp. These do not exist. Find the occurrence of each and add an underscore the file name. For example, it should be src/_path.cpp, not src/path.cpp. Warning: stick to .cpp files only. The .o files, which are intermediate output filenames that nobody cares about, are fine.

On the very last line, when g++ tries to build _tkagg.dll, change -ltk8.4 to -ltk84. Similarly, change -ltcl8.4 to -ltcl84. Cygwin maintainers and matplotlib developers have different ideas about how these two libraries should be named in your /usr/lib.

To recap: distutils thinks there should be files where there are none because they were renamed. We are here to fix it.

End game.

Now run the build script with ./my-build.sh and then run python setup.py install, which will copy the Python files and your built extension—that you lovingly built by hand! it’s your children!—to all the right places.

Most likely, though, you’ll encounter other errors that I never ran into. Google and smart guessing are your friends.

The worst enemies of build errors are: reading, Googling, logical reasoning, acking or grepping, a good text editor, lathering, rinsing, and repeating.

[(March 19, 2009) .]

Great. Thanks a lot. It worked fine for me on matplotlib-1.0.0 I had to create a few directories manually though at the compilation step.

—Dan, 2010.09.02 (4 am)

Thanks, Dan! Glad to hear that it still semi-works on recent matplotlib versions.

Hao Lian, 2010.09.04 (9 pm)

Thanks very much! I have successfully installed matplotlib-1.0.1 on Cygwin without any further changes.

—Alfred Smoothbottom, 2011.02.25 (3 pm)

Thanks a lot. It was much easier to install and use cygwin matplotlib in cygwin xterm.

—sj, 2011.03.23 (8 pm)

Thought I’d post my experiences for those who come after.

These instructions still work with matplotlib-1.0.1 on Cygwin with python2.6, but for whatever reason I was encountering a fork error at the point where setup.py is performing the optional USETEX dependency checks. This prevented the setup script from reaching the —dry-run option. I managed to work around this by:

  1. Force (uncomment) tkagg = False etc. in the setup.cfg file
  2. Run python setup.py —dry-run build > a1.txt
  3. Run python setup.py build so that the majority of the script can compile, create directories etc.
  4. Comment out the tkagg = False etc. in the setup.cfg file
  5. Comment out check_for_dvipng(), check_for_ghostscript(), check_for_latex(), check_for_pdftops() in setup.py
  6. Run python setup.py —dry-run build > a2.txt
  7. diff a1.txt a2.txt to get a list of the additional gcc/g++ lines that still need to be run and run these either in a script or just copy pasting to the command line

To install, I had to jump through similar hoops:

  1. Undo the changes of step 4 and redo step 1.
  2. Run python setup.py install
  3. Redo the changes for step 4
  4. Run python setup.py install

I hope this helps someone, and if not I have a nice set of notes if I ever need to repeat this traumatic exercise. :)

—Rhoda, 2011.06.13 (12 pm)

I was able to use most of the same steps to compile matplotlib-1.1.0:

1) Unpack your source and change into source directory:

$ tar xvzf matplotlib-1.1.0.tar.gz
$ cd matplotlib-1.1.0

2) parse_tcl_config in setupext.py will fail when running `python setup.py build’, but making it return None will make other mechanisms take over, so insert as the first line with your editor or do:

$ sed "/def parse_tcl_config/a\    return None" -i setupext.py

3) calls to gcc/g++ from `python setup.py build’ will fail, but extreacting into a script and running that will work:

$ python setup.py -n build | grep -E '(g\+\+|gcc)' > compile.sh

4) move some source files into place, run compile.sh, run the build script, and repeat this until the build script doesn’t call gcc anymore. I believe you have to call it 11 times, to build all the dll’s. Maybe it should be put into a loop to call it 11 times automatically…

$ for file in src/_*; do cp ${file} ${file/\/_/\/}; done ; sh compile.sh ; python setup.py build

5) switch to an administrator-started cygwin-shell and finish the installation:

# python setup.py install

Step 4) is an approximation of what I did. Basically compile.sh seems to contains 11 sections of compilation commands, where each section ends with a line ending in “.dll”. If you split up compile.sh into those 11 section, and substitute them one by one into step 4 instead of the full compile.sh, it should work and this is closer to how I managed it. Maybe the next person can put together a magic incantation for that ;P

—hkBst, 2011.10.13 (5 am)

For me the suggestions here didn’t quite work. Anyway: Catching the IndexError exception cloaks the actual problem!

First of all, pkg-config is an (undocumented?) dependency of the building process. On my cygwin at least, it wasn’t installed.

The IndexError results from empty values in /usr/lib/tkConfig.sh, which setupext.py doesn’t handle. However, I recently found that tcl/tk on cygwin has many issues anyway (e.g. it seems to expect Windows paths, despite being meant as a cygwin port), so “fixing” this probably is the reason for the other problems.

I just created a setup.cfg from setup.cfg.template where I did nothing but uncomment

tkagg = False

After this, everything compiled like a charm. No manual compilations necessary.

Well, not truely. I still had to install missing dependencies, in particular the *-devel packages for each of the dependencies (e.g. libpng14-devel).

The beginning of my build output currently looks like this:

basedirlist is: ['/usr/local', '/usr']
============================================================================
BUILDING MATPLOTLIB
            matplotlib: 1.1.0
                python: 2.6.5 (r265:79063, Jun 12 2010, 17:07:01)  [GCC
                        4.3.4 20090804 (release) 1]
              platform: cygwin

REQUIRED DEPENDENCIES
                 numpy: 1.5.1
             freetype2: 13.2.7

OPTIONAL BACKEND DEPENDENCIES
                libpng: 1.4.8
            pkg-config: looking for pygtk-2.0 gtk+-2.0
                        * Package libffi was not found in the pkg-config
                        * search path. Perhaps you should add the directory
                        * containing `libffi.pc' to the PKG_CONFIG_PATH
                        * environment variable Package 'libffi', required by
                        * 'PyGObject', not found
                        * You may need to install 'dev' package(s) to
                        * provide header files.
                  Gtk+: no
                        * Could not find Gtk+ headers in any of
                        * '/usr/local/include', '/usr/include', '.'
       Mac OS X native: no
                    Qt: no
                   Qt4: no
                 Cairo: 1.10.0

OPTIONAL DATE/TIMEZONE DEPENDENCIES
              datetime: present, version unknown
              dateutil: 1.5
                  pytz: 2011c

OPTIONAL USETEX DEPENDENCIES
                dvipng: 1.5
           ghostscript: 9.04
                 latex: 3.141592
               pdftops: 3.02

Still the compilation runs through like a charm. It remains to be seen though, if the missing dependencies cause problems.

Note that running the tests

user:~> python
>>> import matplotlib
>>> matplotlib.test()

will still fail for the greater part, because the tests have a couple of additional dependencies, building which didn’t seem worth the effort unless I run into further problems at some point.

About libffi

Sadly, for libffi there is no *-devel package in the cygwin repository. cygffi-4.dll seems to be shipped as part of gcc (though encapsuled in a single-file-package libffi4), but a libffi.pc file is not shipped.

I tried building libffi-3.0.9 from source, which worked. In order for pkg-config to find it, I had to set

PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

but it didn’t help anyway: When trying to build with this custom libffi, I would get compilation failures because a function “__sync_synchronize” wasn’t defined:

  [...]
gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
    -DPY_ARRAY_UNIQUE_SYMBOL=MPL_ARRAY_API -DPYCXX_ISO_CPP_LIB=1
    -D_REENTRANT -I/usr/local/include -I/usr/include -I.
    -I/usr/lib/python2.6/site-packages/numpy/core/include -Isrc
    -Iagg24/include -I.
    -I/usr/lib/python2.6/site-packages/numpy/core/include
    -I/usr/include/freetype2 -I/usr/local/include -I/usr/include -I.
    -I/usr/local/include -I/usr/include -I.
    -I/usr/local/lib/libffi-3.0.9/include -I/usr/include/pygtk-2.0
    -I/usr/include/gtk-2.0 -I/usr/include/glib-2.0
    -I/usr/lib/glib-2.0/include -I/usr/lib/gtk-2.0/include
    -I/usr/include/atk-1.0 -I/usr/include/cairo
    -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/pango-1.0
    -I/usr/include/gio-unix-2.0/ -I/usr/include/pixman-1
    -I/usr/include/libpng14
    -I/usr/lib/python2.6/site-packages/numpy/core/include
    -I/usr/include/python2.6 -c src/_gtkagg.cpp -o
    build/temp.cygwin-1.7.9-i686-2.6/src/_gtkagg.o
In file included from /usr/include/glib-2.0/glib/gasyncqueue.h:34,
                 from /usr/include/glib-2.0/glib.h:34,
                 from /usr/include/pygtk-2.0/pygobject.h:7,
                 from src/_gtkagg.cpp:3:
/usr/include/glib-2.0/glib/gthread.h: In function `gboolean g_once_init_enter(volatile gsize*)':
/usr/include/glib-2.0/glib/gthread.h:347: error: `__sync_synchronize' undeclared (first use this function)
/usr/include/glib-2.0/glib/gthread.h:347: error: (Each undeclared identifier is reported only once for each function it appears in.)
error: command 'gcc' failed with exit status 1
—Klaus-Dieter Bauer, 2011.11.29 (8 am)

Even though this is an old thread, I thought it might be useful to note that as of today (2014/09/21), matplotlib builds fine on cygwin with easy_install.

I used the usual cygwin setup tool to install setuptools, py-qt4, and libffi-devel (it exists now), then

easy_install matplotlib

worked fine for me.

—Glenn H-S, 2014.09.21 (5 pm)

Abandon your ideas.

Use Markdown+, but not HTML. In code blocks, beware angle brackets.