Calvin's

The new postgis 2.0 library (easily installed via `sudo port install postgis2 +postgresql92` if you are using macports as mentioned in my previous post) helps us convert a standard postgresql database into a spatially-aware database via the simple `CREATE EXTENSION postgis;` command:

$  psql92 -p 5433 -U postgres -d your_db

psql92 (9.2.1)
Type "help" for help.
your_db=#  CREATE EXTENSION postgis;

Unfortunately, when we run our django tests, `test_your_db`, which is newly created by django’s test runner whenever the test begins, will not be postgis-enabled. This leads immediately to test failure when our code fails to recognize the Geometry fields in our project’s GeoDjango models.

To get around this problem, we will need to specifically set up a `template_postgis` template database, much like what we did before postgis 2.0 (reference django docs https://docs.djangoproject.com/en/dev/ref/contrib/gis/install/postgis/#creating-a-spatial-database-template-for-earlier-versions).  But because I am running an instance of postgresql 9.1 server and an instance of 9.2 server concurrently, I will need to modify my commands a little.

Like this:-

calvin$ psql92 -p 5433 -U postgres
psql92 (9.2.1)
Type "help" for help.

postgres=# CREATE DATABASE template_postgis ENCODING='utf-8';
CREATE DATABASE
postgres=# UPDATE pg_database SET datistemplate='true' WHERE datname='template_postgis';
UPDATE 1
postgres=# \q

calvin$ POSTGIS_SQL_PATH=/opt/local/share/postgresql92/contrib/postgis-2.0

calvin$ psql92 -p 5433 -U postgres -d template_postgis -f $POSTGIS_SQL_PATH/postgis.sql

calvin$ psql92 -p 5433 -U postgres -d template_postgis -f $POSTGIS_SQL_PATH/spatial_ref_sys.sql

calvin$ psql92 -p 5433 -U postgres -d template_postgis -c "GRANT ALL ON geometry_columns TO PUBLIC;"
GRANT

calvin$ psql92 -p 5433 -U postgres -d template_postgis -c "GRANT ALL ON geography_columns TO PUBLIC;"
GRANT

calvin$ psql92 -p 5433 -U postgres -d template_postgis -c "GRANT ALL ON spatial_ref_sys TO PUBLIC;"
GRANT

With this, running `./manage.py test` with your geodjango-based application will work perfectly fine because django’s default test runner will look for the template_postgis template database to instantiate our now spatially-aware test database.

One of the trickier problems when managing multiple projects involve supporting different versions of open source library or software that your varying projects depend on.

In my current scenario, I need to support web projects which are still running on postgresql 9.1, while building up new projects which uses postgresql 9.2.

Getting postgresql92, postgresql92-server packages installed via macports is the easy part.

$ sudo port install postgresql92 postgresql92-server

A simple `port install` command does the  job.  The problem is getting making sure that they run alongside each other so that I can switch out from working on a project that uses postgresql 9.1 server easily to another project that uses postgresql 9.2.

Once our macports install completes running, we will create our initial database. Like so:

$ sudo mkdir -p /opt/local/var/db/postgresql92/defaultdb
$ sudo chown postgres:postgres /opt/local/var/db/postgresql92/defaultdb
$ sudo su postgres -c '/opt/local/lib/postgresql92/bin/initdb -D /opt/local/var/db/postgresql92/defaultdb'

Nothing surprising here. Except that our postgresql 9.1 server has occupied the default 5432, so we will need to make sure that our postgresql 9.2 server uses a different port.  Opening up our postgresql 9.2 server’s postgresql.conf file with `sudo vim /opt/local/var/db/postgresql92/defaultdb/postgresql.conf`,we set our postgresql 9.2′s port to a different one.

port = 5433 # (change requires restart)

Once our postgresql 9.2 server is properly initialized, we can manually start it using

$ sudo su postgres -c '/opt/local/lib/postgresql92/bin/postgres -D /opt/local/var/db/postgresql92/defaultdb'

which may not release the process to the background.

To release the process to the background, we can instead use Mac OSX’s daemondo. Like this:

$ sudo daemondo --label=postgresql92-server --start-cmd /opt/local/etc/LaunchDaemons/org.macports.postgresql92-server/postgresql92-server.wrapper start

Finally, to ensure that our postgresql 9.2 server starts up everytime we boot up our Mac, alongside our postgresql 9.1 server, we can set

$ sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql92-server.plist

No different from our we make `launchctl` start up postgresql 9.1 server when our Mac boots up.

To check that our postgresql 9.2 server is indeed running, we can execute:

$ psql92 -p 5433 -U postgres
psql92 (9.2.1)
Type "help" for help.
postgres=#

From now on, we will have to be sure that settings and configuration for projects that try to connect to our postgresql 9.2 server uses port 5433.

One more thing – Postgis 2.0.

$ sudo port -v install postgis2 +postgresql92 +raster +topology

Since we use postgis regularly in our projects (for applications requiring GIS functionalities), we need to install the version compatible with postgresql 9.2. This is trivial with Postgis 2.0(.1). We no longer need to go through multiple steps to create a template spatial database (“template_postgis”). In Postgis 1.5, we need to initialize our database with this “template_postgis” template spatial database that we create.

From Postgis 2.0 onwards, it’s straightforward.

$ psql92 -p 5433 -U postgres -d your_db

psql92 (9.2.1)
Type "help" for help.

your_db=#  CREATE EXTENSION postgis;

your_db=# CREATE EXTENSION postgis_topology;

The 2nd `CREATE EXTENSION postgis_topology` is optional at this time of writing as geodjango does not have the necessary functionality to support Postgis’ Topological features yet.

Getting iPython installed in a virtualenv is straightforward.

The gotcha for new pythonistas not familiar with iPython is that it requires gnu readline to display colors, provide autocomplete functionality and a host of other usability features. Unfortunately, on Mac OSX, getting readline via `pip install readline` will not quite work.  This is due to a Mac OSX-specific PYTHONPATH problem.  With `pip installs readline`, it will never be imported, because readline.so goes in site-packages, which ends up behind the libedit system one, located in lib-dynload (yes, OSX Python path order is very odd).

To solve this problem, we will, instead, run this after we have run `pip install readline`:-

easy_install -a readline

The `-a` option means `–always-copy` which copies all needed packages into our installation directory.  Executing this command while in our virtualenv ensures that the readline package gets placed correctly in our virtualenv directory.  Checking:-

(python-for-scientists)Calvins-MacBook-Pro.local ttys009 Sat Nov 03 12:36:36 |~/work/python-for-scientists|
calvin$ ls -la ~/.virtualenvs/python-for-scientists/lib/python2.7/site-packages/
total 16
drwxr-xr-x 14 calvin staff 476 Nov 3 12:27 .
drwxr-xr-x 51 calvin staff 1734 Nov 2 15:21 ..
drwxr-xr-x 19 calvin staff 646 Nov 3 12:27 IPython
drwxr-xr-x 10 calvin staff 340 Nov 2 15:03 distribute-0.6.28-py2.7.egg
-rw-r--r-- 1 calvin staff 285 Nov 2 15:36 easy-install.pth
drwxr-xr-x 10 calvin staff 340 Nov 3 12:27 ipython-0.13.1-py2.7.egg-info
drwxr-xr-x 38 calvin staff 1292 Nov 2 15:22 numpy
drwxr-xr-x 7 calvin staff 238 Nov 2 15:22 numpy-1.6.2-py2.7.egg-info
drwxr-xr-x 4 calvin staff 136 Nov 2 15:03 pip-1.2.1-py2.7.egg
drwxr-xr-x 3 calvin staff 102 Nov 2 15:03 readline
drwxr-xr-x 6 calvin staff 204 Nov 2 15:36 readline-6.2.4.1-py2.7-macosx-10.7-x86_64.egg
drwxr-xr-x 38 calvin staff 1292 Nov 2 16:00 scipy
drwxr-xr-x 7 calvin staff 238 Nov 2 16:00 scipy-0.11.0-py2.7.egg-info
-rw-r--r-- 1 calvin staff 30 Nov 2 15:03 setuptools.pth

Now, in our virtual env, our iPython shell should work perfectly, with all the features afforded by the gnu readline library.

Moving on, we now want the power of iPython on in a Qt4-powered GUI console.  This will prove to be a little tricky on Mac OSX with Macports.

1.  Qt4 on Mac OSX

The first thing we need to do is to install the qt4-mac package.  Straightforward via MacPorts:-

calvin$ sudo port -v install qt4-mac

2. Python bindings to Qt4

The problem begins when we try to install a python binding library for Qt4. We have to possible options from PyPi

• PyQt - http://pypi.python.org/pypi/PyQt
• PySide - http://pypi.python.org/pypi/PySide

Attempting `pip install PyQt` in your virtual env fails out right. Because PyQt package on pypi does not include a setup.py.

Going the PySide route requires us to have cmake, which can easily be solved by installing cmake system-wide with `sudo port install cmake`.  However, that still fails to help us get `pip install pyside` working right.  We will encounter an error like this:-

error: Failed to locate the Python library /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.so.1
 Complete output from command /Users/calvin/.virtualenvs/python-for-scientists/bin/python -c "import setuptools;__file__='/Users/calvin/.virtualenvs/python-for-scientists/build/pyside/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/kl/_52jng9s6sl2knv_0jds9w140000gn/T/pip-7zqGiu-record/install-record.txt --single-version-externally-managed --install-headers /Users/calvin/.virtualenvs/python-for-scientists/bin/../include/site/python2.7:
 Removing /Users/calvin/.virtualenvs/python-for-scientists/build/pyside/PySide

To solve this problem, we will instead rely on the Macports version of PyQt.

calvin$ port search pyqt4
py-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py24-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py25-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py26-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py27-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py31-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py32-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

3. System-wide iPython qtconsole

So,

calvin$ sudo port -v install py27-pyqt4

---> Computing dependencies for py27-pyqt4.
---> Fetching archive for py27-pyqt4
---> py27-pyqt4-4.9.4_0.darwin_11.x86_64.tbz2 doesn't seem to exist in /opt/local/var/macports/incoming/verified
---> Attempting to fetch py27-pyqt4-4.9.4_0.darwin_11.x86_64.tbz2 from http://packages.macports.org/py27-pyqt4
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
---> Attempting to fetch py27-pyqt4-4.9.4_0.darwin_11.x86_64.tbz2 from http://mse.uk.packages.macports.org/sites/packages.macports.org/py27-pyqt4
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
---> Attempting to fetch py27-pyqt4-4.9.4_0.darwin_11.x86_64.tbz2 from http://lil.fr.packages.macports.org/py27-pyqt4
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
---> Fetching distfiles for py27-pyqt4
---> Verifying checksum(s) for py27-pyqt4
---> Checksumming PyQt-mac-gpl-4.9.4.tar.gz
---> Extracting py27-pyqt4
---> Extracting PyQt-mac-gpl-4.9.4.tar.gz
---> Applying patches to py27-pyqt4
---> Applying patch-configure.py
patching file configure.py
---> Applying patch-fix-qt_apps_dir.diff
patching file examples/demos/qtdemo/menumanager.py
patching file examples/designer/plugins/plugins.py
---> Configuring py27-pyqt4
Determining the layout of your Qt installation...

...

Qt v4.8.3 free edition is being used.
Qt is built as a framework.
SIP 4.13.3 is being used.
The Qt header files are in /opt/local/include.
The shared Qt frameworks are in /opt/local/Library/Frameworks.
The Qt binaries are in /opt/local/bin.
The Qt mkspecs directory is in /opt/local/share/qt4.
These PyQt modules will be built: QtCore, QtGui, QtHelp, QtMultimedia,
QtNetwork, QtDeclarative, QtOpenGL, QtScript, QtScriptTools, QtSql, QtSvg,
QtTest, QtWebKit, QtXml, QtXmlPatterns, QtDesigner.
The PyQt Python package will be installed in
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages.
PyQt is being built with generated docstrings.
PyQt is being built with 'protected' redefined as 'public'.
The Designer plugin will be installed in /opt/local/share/qt4/plugins/designer.
The PyQt .sip files will be installed in /opt/local/share/py27-sip/PyQt4.
pyuic4, pyrcc4 and pylupdate4 will be installed in
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin.
Generating the C++ source for the QtCore module...

...

x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Loader/qobjectcreator.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/__init__.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/compiler.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/indenter.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/misc.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/proxy_metaclass.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/qobjectcreator.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/qtproxies.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pylupdate4
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pyrcc4
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pyuic4
x ./opt/local/bin/pylupdate4-2.7
x ./opt/local/bin/pyrcc4-2.7
x ./opt/local/bin/pyuic4-2.7
---> Cleaning py27-pyqt4
---> Removing work directory for py27-pyqt4
---> Updating database of binaries: 100.0%
---> Scanning binaries for linking errors: 100.0%
---> No broken files found.

With this done, and with a system-wide zmq/py27-zmq installed (via `sudo port -v install zmq py27-zmq` command), our system-wide ipython will work beautifully with `ipython qtconsole` outside a python virtual env.

4. iPython qtconsole in virtualenv

However, if we attempt to do the same in an isolated python virtual env, we will encounter this error triggered when qtconsoleapp.py in our virtualenv iPython gets called:-

File "/Users/calvin/.virtualenvs/python-for-scientists/lib/python2.7/site-packages/IPython/frontend/qt/console/qtconsoleapp.py", line 56, in <module>
 from IPython.external.qt import QtCore, QtGui
 File "/Users/calvin/.virtualenvs/python-for-scientists/lib/python2.7/site-packages/IPython/external/qt.py", line 43, in <module>
 raise ImportError('Cannot import PySide >= 1.0.3 or PyQt4 >= 4.7')
ImportError: Cannot import PySide >= 1.0.3 or PyQt4 >= 4.7

which of course, is to be expected.

So, we aren’t satisfied yet because we cannot run our ipython qtconsole within an isolated python virtual env.  Our isolated python virtualenv will not be able to locate the system-wide py27-qt4 that we have installed above; and we also cannot depend on `pip install pyqt` or `pip install pyside` commands inside the virtual env, we will need to resort to a little bit of bash scripting trickery – i.e. symlink to our installed system-wide libraries (PyQt4 and its dependency SIP) to solve this problem from our virtualenv location.

And here’s our little bash script to achieve this:-

#!/bin/bash

LIBS=( PyQt4 sip.so )
PYTHON_VERSION=python$(python -c "import sys; print (str(sys.version_info[0])+'.'+str(sys.version_info[1]))")
VAR=( $(which -a $PYTHON_VERSION) )
GET_PYTHON_LIB_CMD="from distutils.sysconfig import get_python_lib; print (get_python_lib())"
LIB_VIRTUALENV_PATH=$(python -c "$GET_PYTHON_LIB_CMD")
LIB_SYSTEM_PATH=$(${VAR[1]} -c "$GET_PYTHON_LIB_CMD")

for LIB in ${LIBS[@]}
do
# Make each library specified in LIBS available in our current virtual env
ln -s $LIB_SYSTEM_PATH/$LIB $LIB_VIRTUALENV_PATH/$LIB
done

Once we run `./symlink_pyqt4_and_sip.sh` while in our virtual env, PyQt4 and SIP will now be available to us!

5. Finishing up: ipython qtconsole in virtualenv!

Quick check:-

calvin$ ls -la $VIRTUAL_ENV/lib/python2.7/site-packages
total 32
drwxr-xr-x 12 calvin staff 408 Nov 3 13:59 .
drwxr-xr-x 51 calvin staff 1734 Oct 31 14:07 ..
drwxr-xr-x 19 calvin staff 646 Oct 31 10:54 IPython
lrwxr-xr-x 1 calvin staff 93 Nov 3 13:59 PyQt4 -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4
drwxr-xr-x 10 calvin staff 340 Oct 31 10:13 distribute-0.6.28-py2.7.egg
-rw-r--r-- 1 calvin staff 285 Oct 31 10:15 easy-install.pth
drwxr-xr-x 10 calvin staff 340 Oct 31 10:15 ipython-0.13.1-py2.7.egg-info
drwxr-xr-x 4 calvin staff 136 Oct 31 10:13 pip-1.2.1-py2.7.egg
drwxr-xr-x 3 calvin staff 102 Oct 31 10:13 readline
drwxr-xr-x 6 calvin staff 204 Oct 31 10:15 readline-6.2.4.1-py2.7-macosx-10.7-x86_64.egg
-rw-r--r-- 1 calvin staff 30 Oct 31 10:13 setuptools.pth
lrwxr-xr-x 1 calvin staff 94 Nov 3 13:59 sip.so -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sip.so

Yup.  As you can see, PyQt4 and sip.so are correctly symlinked from our virtualenv directory to the system-wide one.

Recall that iPython qt console requires a couple more dependencies; but now we can get them via pip (and not macports) – `pip install pygments pyzmq` (Note that zmq library itself should already be installed system-wide but the python bindings pyzmq – different name compared to our macports’ py27-zmq – needs to be installed within our virtualenv).

Once done, running `ipython qtconsole` will work beautifully, as expected.