pgindent Jenkins job

I have set up a Jenkins job that runs pgindent. Besides checking that the procedure of running pgindent works, it also provides a preview of what pgindent would do with the current source (pgindent.diff), which can be educational or terribly confusing.

Introducing the Pex package manager for PostgreSQL

I have written a new light-weight package manager for PostgreSQL, called pex. It's targeted at developers, allows easy customization, and supports multiple PostgreSQL installations.

Here is how it works:

Installation:

git clone git://github.com/petere/pex.git
cd pex
sudo make install

Install some packages:

pex init
pex install plproxy
pex search hash
pex install pghashlib

Multiple PostgreSQL installations:

pex -g /usr/local/pgsql2 install plproxy
pex -p 5433 install pghashlib

Upgrade:

pex update
pex upgrade

It works a bit like Homebrew, except that it doesn't use Ruby or a lot of metaphors. ;-)

Check it out at https://github.com/petere/pex.

PostgreSQL and Jenkins

A lot of places use Jenkins nowadays, including where I now work and have previously worked. I enjoy working with Jenkins, and so I always wanted try out how this would work with PostgreSQL. Obviously, there would be some overlap with the build farm, but that's OK. The point of the build farm, after all, is to build things in many different ways to find potential problems, and this would just support that overall effort.

So I have set this up now: http://pgci.eisentraut.org/jenkins/

It's already been very helpful during the last couple of weeks that I've run this. The main point behind the effort is to automate things. These are things I do just about every day and won't have to anymore:

• build PostgreSQL
• check for compiler warnings
• run various test suites
• do this for all supported branches

These are things I do every couple of weeks and have now automated:

• check distribution building (make distcheck)
• test build of additional documentation formats
• cpluspluscheck
• check external web links in the documentation (The job for that currently appears to be reporting false positives. Use with caution.)
• test coverage reporting

Moreover, I have set up to build some extensions and external modules, which weren't regularly tested. (The build farm is making some efforts in this area, though.)

Actually, many of the checks I had set up immediately found problems: newly introduced compiler warnings, secondary documentation format builds broken, cpluspluscheck failing, broken links in the HTML documentation, various extensions no longer build with Postg reSQL 9.3devel.

But there is more cool stuff:

• There are various RSS feeds for all builds or failed buids.
• You can interact with the system on mobile devices. I use JenkinsMobi for iOS.
• You can get up to date documentation builds on a more predictable schedule.

The one thing (just about) it doesn't do is test operating system and CPU architecture portability. Jenkins comes from a Java background, where this isn't much of an issue, and so there isn't good built-in support for that sort of thing. But anyway, we have the build farm for that.

You can get the code at http://bitbucket.org/petere/pgci. The entire setup is automated with Puppet. You can fork it and set up your own (or send me your changes), or you can run it locally using Vagrant (which is what I do to test changes).

If you have any ideas, let me know (file an issue on Bitbucket). I have plans for a number of enhancements already, foremost pg_upgrade testing. Also, let me know if there are additional extensions you want tested. I have just put in a few I use myself at the moment, but other can easily be added.

Happy New Year!

psqlrc files

In PostgreSQL 9.2, you can use major-version-specific .psqlrc files, such as .psqlrc-9.2. PostgreSQL 9.2 also added the "include relative" command \ir to psql. Combining these two, you can set up psql initialization to take advantage of any new features you want without breaking the use of old psql releases.

For example, I'd like to set up psql to automatically use \x auto. But if I just put that into .psqlrc, older psql releases will complain about an unknown command. (I usually have multiple PostgreSQL versions installed, and I share dotfiles across hosts.) On the other hand, I don't want to have to duplicate the entire .psqlrc file to add one command, which is where \ir comes in.

Here is what I use, for example:

.psqlrc-9.2

\ir .psqlrc
\set QUIET yes
\set COMP_KEYWORD_CASE preserve-lower
\x auto
\unset QUIET

.psqlrc-9.3

\ir .psqlrc-9.2

pgxnclient supports tarballs and HTTP

Need to install a PostgreSQL server add-on module? The devel branch of pgxnclient now supports this type of thing:

pgxnclient install http://pgfoundry.org/frs/download.php/3274/plproxy-2.4.tar.gz

This downloads, unpacks, builds, and installs. And the module doesn't need to be on PGXN. And of course you don't have to use HTTP; a file system location will work as well. I think this can be very useful, especially during development, when not everything is available in packaged form, or even for deployment, if you don't want to bother packaging everything and have been installing from source anyway.

Funny version numbers

Often, I install a new Debian package using apt-get install, and as the progress output flies by, I wonder, Whoa!, should I really be using a piece of software with that version number?

It says a lot, after all. If I see

tool 2.4.1-2

then I (subconsciously) think, yeah, the upstream maintainers are obviously sane, the tool has been around for a while, they have made several major and minor releases, and what I'm using has seen about one round of bug fixing, and a minimal amount of tweaking by the Debian maintainer.

On the other hand, when I see

tool 7.0.50~6.5~rc2+0.20120405-1

I don't know what went on there. The original release version 7.0.50 was somehow wrong and had to be renamed 6.5? And then the 2nd release candidate of that? And then even that wasn't good enough, and some dated snapshot had to be taken?

Now, of course, there are often reasons for things like this, but it doesn't look good, and I felt it was getting out of hand a little bit.

I tried to look into this some more and find a regular expression for a reasonably sane version number. It's difficult. This is how far I've gotten: https://gist.github.com/3345974. But this still lists more than 1500 packages with funny version numbers. Which could be cause for slight concern.

Take a look at what this prints. You can't make some of that stuff up.

Tracing shell scripts with time stamps

A random tip for shell script hackers. You know that with set -x you can turn on tracing, so that every command is printed before being executed. In bash, you can also customize the output prefix by setting the PS4 variable. The default is PS4='+ '. Here is an example. I wanted to "profile" a deployment script, to see why it took so long. Ordinarily, I might have sprinkled it with date calls. Instead, I merely added

set -x
PS4='+\t '

near the top. \t stands for time stamp. (The script was already using bash explicitly, as opposed to /bin/sh.) That way, every line is prefixed by a time stamp, and the logs could easily be analyzed to find a possible performance bottleneck.