Tricky shell local variables

I have a word of warning against improper use of local in shell functions.

If you are using shell functions, you might want to declare some variables local to the shell function. That is good. The basic syntax for that is

local a b c

In some shells, you can also combine the local declaration and assignment, like this:

local foo=$1
local bar=$2

(The Debian policy even explicitly allows it.)

This is somewhat dangerous.

Bare shell assignment like

foo=$bar

does not perform word splitting, so the above is safe even if there are spaces in $bar. But the local command does perform word splitting (because it can take multiple arguments, as in the first example), so the seemingly similar

local foo=$bar

is not safe.

This can be really confusing when you add local to existing code and it starts breaking.

You can avoid this, of course, by always quoting everything to like

local foo="$bar"

but overquoting isn't always desirable, because it can make code less readable when commands are nested, like

local foo="$(otherfunc "other arg")"

(Nesting is legal and works fine in this case, however.)

I suggest using local only for declaring variables, and using separate assignment statements. That way, all assignments are parsed in the same way.

Time to retrain the fingers

For years, no decades, I've typed tar tzf something, tar xzf something. Except when someone annoying sent an uncompressed tar file and I had to then go and take out the z in the middle.

Then came bzip2, and we learned tar tjf, tar xjf. OK, I could live with that. One emerging problem was that the tab completion now worked the wrong way around conceptually, because you had to pick and type the right letter first in order to see the appropriate set of files to unpack offered for completion.

Then came lzma, which was (quick, guess?), tar tJf, tar xJf. And then there was lzop, which was too boring to get its own letter, so you had to type out tar -x --lzop -f.

But lzma was short-lived, because then came xz, which was also J, because lzma was now too boring as well to get its own letter.

Oh, and there is also the old compress, which is Z, and lzip, which I'd never heard of.

But stop that. Now there is

 -a, --auto-compress
            use archive suffix to determine the compression program

This handles all the above compression programs, and no compression. So from now on, I always use tar taf and tar xaf. Awesome.

The finger movements will be almost the same on QWERTY and AZERTY, and easier than before on QWERTZ.

Actually, this option is already four years old in GNU tar. Funny I'd never heard of it until recently.

Setting the time zone on remote SSH hosts

The tasks: I have one or more desktop/laptop machines with varying local time zones (because the persons using them are actually in different time zones, or because the one person using them travels). I also have a number of servers configured in some random time zones. (It could be the time zone where they are physically located, or the time zone of the person who installed it, or UTC for neutrality.)

Now what I would like to have happen is that if I log in using SSH from a desktop to a server, I see time on that server in my local time zone. For things like ls -l, for example. Obviously, this illusion will never be perfect. Nothing (except something very complicated) will adjust the time stamps in the syslog output, for example. But the ls -l case in particular seems to come up a lot, to check how long ago was this file modified.

This should be completely doable in principle, because you can set the TZ environment variable to any time zone you like, and it will be used for things like ls -l. But how do you get the TZ setting from here to there?

First, you have to make the remote SSH server accept the TZ environment variable. At least on Debian, this is not done by default. So make a setting like this in /etc/ssh/sshd_config:

# Allow client to pass locale environment variables
AcceptEnv LANG LC_* TZ

You also need to make the equivalent setting on the client side, either in /etc/ssh/ssh_config or in ~/.ssh/config:

SendEnv LANG LC_* TZ

Which leaves the question, how do you get your local time zone into the TZ variable to pass to the remote server? The actual time zone configuration is the file /etc/localtime, which belongs to glibc. In current Debian, this is (normally) a copy of some file under /usr/share/zoneinfo/. In the distant past, it was a symlink, which would have made things easier, but now it's a copy, so you don't know where it came from. But the name of the time zone is also written to /etc/timezone, so you can use that.

The format of the TZ environment variable can be found in the glibc documentation. If you skip past most of the text, you will see the relevant parts:

The third format looks like this:

:CHARACTERS

Each operating system interprets this format differently; in the GNU C library, CHARACTERS is the name of a file which describes the time zone.

So what you could do is set

TZ=":$(cat /etc/timezone)"

Better yet, for hopping through multiple SSH hosts in particular, make sure to preserve an already set TZ:

TZ=${TZ:-":$(cat /etc/timezone)"}

And finally, how does one hook this into ssh? The best I could think of is a shell alias:

alias ssh='TZ=${TZ:-":$(cat /etc/timezone)"} ssh'

Now this set up has a number of flaws, including:

• Possibly only works between Linux (Debian?) hosts.

• Only works if available time zone names match.

• Only works when calling ssh from the shell.

But it practice it has turned out to be quite useful.

Comments? Improvements? Better ideas?

Related thoughts:

• With this system in hand, I have resorted to setting the time zone on most servers to UTC, since I will see my local time zone automatically.

• Important for the complete server localization illusion: some ideas on dealing with locales on remote hosts.

How to set xterm titles in psql

As most of you might know, you can customize the prompts in psql, the PostgreSQL interactive terminal. If you didn't know, you can read up on it in the documentation. You can also add color to your prompt using VT100 control sequences; an example of that can be found in said documentation. I have my shell and psql prompts in different colors so I can tell quickly when I'm logged in to what. It spares me the embarrassment of typing "ls" into psql.

As it turns out, you can use that same mechanism to customize the xterm titles from within psql. It might not be completely clear why that is necessary. After all, if you have customized the xterm titles in your shell, say to show the name of the command currently running, then it would already show something like "psql somedb" when you are using psql. The trick, or perhaps danger, with that is that if you use commands like \c or SET SESSION AUTHORIZATION in psql, then your connection parameters change, but the shell doesn't update the xterm title. This had me confused more than once now.

The control sequence to change the xterm title is ESC]0;stringBEL, where ESC is \033 and BEL is \007. In psql prompt language, this becomes [%033]0;string%007%]. Add that somewhere to your prompt (doesn't really matter where, as it won't be printed). Say you are using the default prompt (boring!) and want to show "psql dbname" in the xterm window title when logged in to PostgreSQL, then put this into your ~/.psqlrc:

\set PROMPT1 '%[%033]0;psql %/%007%]%/%R%# '

Note: If you don't have your shell to set up to update the xterm title, then the title set by psql will remain after you quit psql. That might be even more confusing, so don't try this if you don't have your shell set up for this.

If you have other fun and/or useful ideas for the psql prompt, please share them in the comments.

Adding Color to the Console: Round 2: Highlight vs. Pygments vs. Source-highlight

A month ago I compared the Pygments and Source-highlight packages for use as a syntax highlighting tool. A commenter pointed me toward another package called Highlight. So let's compare those three, based on the same criteria as last time:

	Pygments	Source-highlight	Highlight
Debian package	python-pygments	source-highlight	highlight
Dependencies	Python	C++, Boost	C++
License	BSD	GPL 2+	GPL 3+
Input Languages	136	30	143
Output Languages	~11	~6	~9
Styles	16	5	40
Performance test	6 min	64 sec	67 sec

So for interactive use, it looks like a pretty close call between Source-highlight and Highlight, unless you need one of the additional input formats supplied by Highlight. It appears that Highlight supports SGML files out of the box, unlike the other two, which could make it interesting for me.

Adding Color to the Console: Pygments vs. Source-highlight

The other day I write about code syntax highlighting with less and source-highlight, when someone pointed out the Pygments package, that can do the same thing. This called for

the great source-highlight vs. pygments face-off

Installation

Source-highlight is in C++ and needs the Boost regex library. About 3.5 MB together. Pygments is in Python and has no dependencies beyond that. About 2 MB installed. No problem on either side.

In Debian, the packages are source-highlight and python-pygments. Note that the Pygments command-line tool is called pygmentize.

Source-highlight is licensed under the GPL, Pygments under a BSD license.

Getting Started

pygmentize file.c

writes a colored version of file.c to the standard output. Nice.

source-highlight file.c

writes a colored version of file.c to file.c.html. As I had written before, the correct invocation for this purpose is

source-highlight -fesc -oSTDOUT file.c

That makes pygmentize slightly easier to use, I suppose.

Supported Languages

Source-highlight supports 30 languages, Pygments supports 136.

Source-highlight can produce output for DocBook, ANSI console, (X)HTML, Javadoc, LaTeX, Texinfo. Pygments can produce output for HTML, LaTeX, RTF, SVG, and several image formats.

Source-highlight supports styles, but only ships a few. Pygments supports styles and filters, and ships over a dozen styles.

So more options with Pygments here.

Also note that Pygments is a Python library that can be used, say, in web applications for syntax highlighting. This is used in Review Board, for example. Source-highlight is just a command-line tool, but it could of course also be invoked by other packages. Horde uses this, for instance.

Speed

To process all the C files in the PostgreSQL source tree (709271 lines), writing the ANSI console colored version of each file.c to file.c.txt:

source-highlight

25 seconds

pygmentize

5 minutes

So for interactive use with less, source-highlight is probably still the better option.

Robustness

pygmentize gave me a few errors of this sort during the processing of the PostgreSQL source tree:

*** Error while highlighting:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb7' in position 83: ordinal not in range(128)
  (file "/usr/lib/python2.5/site-packages/Pygments-1.0-py2.5.egg/pygments/formatters/terminal.py", line 93, in format)

*** Error while highlighting:
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 196-198: invalid data
  (file "/usr/lib/python2.5/encodings/utf_8.py", line 16, in decode)

That probably shouldn't happen.

source-highlight gave no spurious errors in my limited testing.

Miscellaneous

Source-highlight can highlight its own configuration files, which are in a custom language, and Pygments' configuration files, which are in Python. Conversely, Pygments can of course highlight its own configuration files, but doesn't know what to do with those of Source-highlight.

Summary

I will stick with Source-highlight for interactive use, but Pygments is a great alternative when you need more formatting options or want to integrate the package as a library.

Adding Color to the Console: Code Syntax Highlighting with less and source-highlight

Syntax highlighting is a standard feature of every nontrivial editor, but if you just want to look at a file or the output of a command, then you will often only stumble upon niche solutions such as colordiff, or you end up using vim as your file viewer. But one very flexible program that apparently doesn't get much attention is source-highlight, available from the GNU project or as a Debian package. Besides being able to syntax-highlight all the usual programming and markup languages, it can also produce various output formats such as HTML, LaTeX, and of course plain text.

The standalone usage is a bit bizarre, because it tries to produce HTML by default and write the output to a file named infile.html. To produce colored plain text on standard output, use something like

source-highlight -fesc -oSTDOUT FILENAME

To get the most out of this, integrate it with the less pager program, so all the files you look at are automatically colored. Create a file .lessfilter in your home directory with the following contents:

#!/bin/sh

source-highlight -fesc -oSTDOUT "$1" 2>/dev/null

and make this file executable. Be sure to redirect the error output away as shown, so that you don't get annoying messages when source-highlight complains in case it doesn't know the format of the file. Then put the following in your shell startup file (.bashrc or .zshrc):

export LESS="-R"
eval "$(lesspipe)"

The first line makes sure that less will show colors, and the second line sets things up so that the .lessfilter file is invoked.

That's it. It recognizes the file type by the file name extension, so try anything like

less somefile.c
less somefile.py
less somefile.patch

to see the colors. It won't work when less is at the end of a pipe; I have some ideas on how to work around that, which I will post here later.