I have been collecting monthly PostgreSQL regression test code coverage reports at <http://developer.postgresql.org/~petere/coverage/>. So if you are wondering what this thing is but haven't had the courage to try it out yourself, there is your chance. (Hmm, buildfarm integration could be nice, someday.)
We have had a line coverage rate of about 66% steadily for a few months now (well, it's feature freeze). The lcov tool labels that as "green" (=good). The new version of lcov, which I have in use as of the April report, also reports function coverage, where we have about 73%, which lcov labels as "red" (=bad).
For the next release cycle, I have two goals in this area: First, expand the test coverage reporting to the entire source tree, not only the backend. And second, improve the test coverage of various neglected areas. There is reduced coverage, for example, in the areas of non-btree indexes, vacuuming, recovery, GEQO; and once we analyze other parts of the source tree, we will probably find gaping holes there.
Monday, May 18, 2009
Wednesday, May 13, 2009
The Big Shots
As the occasional thinker about open-source development practices, communities, and issues, I have been wondering for a while: What are the largest open-source projects? What projects have the most code, the most users, and the most issues to deal with, and how do they cope?
The Debian archive should provide some insights into the first one or two questions, as it contains a very large portion of all available and relevant open-source software and exposes them in a fairly standard form. In the old days one might have gotten out grep-dctrl to create some puzzling statistics, but nowadays this information is actually available in an SQL database: the Ultimate Debian Database (UDD). (And it's in PostgreSQL. And it comes with a postgresql_autodoc-generated schema documentation. Excellent.)
So here is a first question. Well, the zeroth question would have been, which source packages have the largest unpacked orig tarball, but that information doesn't seem to be available, either via UDD or via apt. So the first question anyway is, which source packages produce the largest installation size across all their binary packages:
Here is another idea. Take a package's installation footprint and multiply it by its popularity contest installation count. So you get some kind of maintenance effort score, either because the package is large or because you have a lot of users or both.
I noticed linux-2.6 is suspiciously absent because of a low popcon score (?!?).
I don't want to dump the entire database into this blog post, but if you try this yourself you can look at about the first 200 to 300 places to find reasonably large and complex projects before it gets a bit more obscure. A few highlights:
The UDD is also an interesting use case that shows how you can deploy a PostgreSQL database as a semi-public service with direct access. A great tool, and a great tool to build other great tools on top of.
The Debian archive should provide some insights into the first one or two questions, as it contains a very large portion of all available and relevant open-source software and exposes them in a fairly standard form. In the old days one might have gotten out grep-dctrl to create some puzzling statistics, but nowadays this information is actually available in an SQL database: the Ultimate Debian Database (UDD). (And it's in PostgreSQL. And it comes with a postgresql_autodoc-generated schema documentation. Excellent.)
So here is a first question. Well, the zeroth question would have been, which source packages have the largest unpacked orig tarball, but that information doesn't seem to be available, either via UDD or via apt. So the first question anyway is, which source packages produce the largest installation size across all their binary packages:
udd=> SELECT source, sum(installed_size)/1024 AS mib FROM packages WHERE distribution = 'debian' AND release = 'sid' AND component = 'main' AND architecture IN ('all', 'i386') AND section <> 'debug' GROUP BY source, version ORDER BY mib DESC LIMIT 30;
source | mib
------------------+------
openoffice.org | 1797
kde-l10n | 648
gcj-4.4 | 544
vtk | 465
linux-2.6 | 404
openclipart | 353
vegastrike-data | 311
ghc6 | 308
gclcvs | 303
wesnoth | 300
fpc | 269
axiom | 256
webkit | 255
gcc-snapshot | 255
lazarus | 241
kdebase-workspace | 226
plt-scheme | 221
torcs-data-tracks | 219
scilab | 213
openscenegraph | 211
eclipse | 210
sagemath | 201
insighttoolkit | 198
acl2 | 195
kdebindings | 181
atlas | 165
gcl | 163
trilinos | 153
paraview | 153
asterisk | 144
(30 rows)This produces a few well-known packages, but also a number of obscure ones. If you look closer, many of them appear to be themed around scientific, numerical, visualization, Scheme, Lisp, that sort of thing. Hmm.Here is another idea. Take a package's installation footprint and multiply it by its popularity contest installation count. So you get some kind of maintenance effort score, either because the package is large or because you have a lot of users or both.
SELECT rank() OVER (ORDER BY score DESC), source, sum(installed_size::numeric * insts) AS score FROM packages JOIN popcon USING (package) WHERE distribution = 'debian' AND release = 'sid' AND component = 'main' AND architecture IN ('all', 'i386') GROUP BY source, version ORDER BY score DESC LIMIT 30;
rank | source | score
-----+-----------------------------+-------------
1 | openoffice.org | 12638492332
2 | mysql-dfsg-5.0 | 3411344560
3 | eglibc | 3371485240
4 | perl | 3019183024
5 | evolution | 2669948000
6 | samba | 2308923872
7 | mesa | 1853902860
8 | texlive-base | 1684245516
9 | gcj-4.3 | 1610495484
10 | foomatic-db-engine | 1608178104
11 | foomatic-db | 1423947704
12 | inkscape | 1413910080
13 | qt4-x11 | 1258220636
14 | gcc-4.3 | 1248741312
15 | kdelibs | 1021058256
16 | gnome-applets | 998434136
17 | xulrunner | 958232688
18 | coreutils | 954766896
19 | openssl | 877067672
20 | ncurses | 827679424
21 | python2.5 | 815826384
22 | aptitude | 808161380
23 | gimp | 786015124
24 | gnome-utils | 781756328
25 | nautilus | 774319690
26 | openoffice.org-dictionaries | 761075576
27 | eclipse | 756072380
28 | dpkg | 736626200
29 | openclipart | 731244240
30 | wine | 707967500
(30 rows)
(Yeah, they run this thing on PostgreSQL 8.4 beta 1.)I noticed linux-2.6 is suspiciously absent because of a low popcon score (?!?).
I don't want to dump the entire database into this blog post, but if you try this yourself you can look at about the first 200 to 300 places to find reasonably large and complex projects before it gets a bit more obscure. A few highlights:
51 | gnupg | 455660464This is obviously still biased in a lot of ways, but it does show the major projects.
59 | php5 | 386417572
60 | mutt | 381148176
83 | icu | 258602756
84 | xorg-server | 255186332
101 | exim4 | 224857700
107 | openssh | 215792828
113 | tar | 201520400
114 | postgresql-8.3 | 196844584
115 | libx11 | 195856564
116 | ruby1.8 | 194681656
272 | emacs22 | 62047476
The UDD is also an interesting use case that shows how you can deploy a PostgreSQL database as a semi-public service with direct access. A great tool, and a great tool to build other great tools on top of.
Labels:
Computing,
Debian,
English,
PostgreSQL
Europawahl
Ich habe letzte Woche die Briefwahlunterlagen zur Europawahl bekommen. Als Briefwähler hat man ja etwas Zeit, sich den Stimmzettel etwas genauer durchzulesen. So trifft man gelegentlich alte Bekannte wieder. Zum Beispiel sehe ich auf Platz 6 der Liste der CDU
Sabine Verheyen, Hausfrau, Aachen
Das ist die selbe Frau Verheyen, vermute ich ganz stark, die vor ein paar Jahren, also ich noch in Aachen wohnte, die Oberbürgermeisterwahl gegen
Jürgen Linden, Oberbürgermeister, Aachen
verloren hatte. Tipp vom einfachen Wahlvolk: Bei allem Respekt für Hausfrauen, schreiben Sie doch was anderes auf den Wahlzettel. Das sieht besser aus. Im Notfall so wie der Platz 2 auf der Liste DIE LINKE: "Angestellte". Geht doch.
Auf dem Stimmzettel befinden sich hier bei mir insgesamt 31 Wahlvorschläge. Vereinigungsfreiheit ist ja eine tolle Sache, aber man kann sich ja auch mal absprechen. Vielleicht könnten ja
10. Volksabstimmung, und
26. Für Volksentscheide
das nächste Mal gemeinsame Sache machen?
Und vielleicht finden sich auch unter
18. 50plus
22. Die Grauen
30. Rentnerinnen und Rentner Partei
31. Rentner-Partei-Deutschland
das nächste Mal Schnittpunkte?
Der eigentliche Knaller sind ja aber die Berufe der Kandidaten auf der Liste der Piratenpartei Deutschlands: Dipl.-Wirtschaftsmathematiker, Dipl.-Physiker, selbst. IT-Unternehmer, Informatiker, Geschäftsführer, Student, Student, Programmierer, Web-Entwickler, Consultant. Alles klar. ;-)
Dieses Mal gibt es übrigens wieder den Wahl-O-Mat. Den hatten Sie ja letztes Mal wohl gerichtlich verboten, weil er nicht alle Parteien berücksichtigt hatte. Dieses mal sind fast alle 31 dabei. Toll ...
Sabine Verheyen, Hausfrau, Aachen
Das ist die selbe Frau Verheyen, vermute ich ganz stark, die vor ein paar Jahren, also ich noch in Aachen wohnte, die Oberbürgermeisterwahl gegen
Jürgen Linden, Oberbürgermeister, Aachen
verloren hatte. Tipp vom einfachen Wahlvolk: Bei allem Respekt für Hausfrauen, schreiben Sie doch was anderes auf den Wahlzettel. Das sieht besser aus. Im Notfall so wie der Platz 2 auf der Liste DIE LINKE: "Angestellte". Geht doch.
Auf dem Stimmzettel befinden sich hier bei mir insgesamt 31 Wahlvorschläge. Vereinigungsfreiheit ist ja eine tolle Sache, aber man kann sich ja auch mal absprechen. Vielleicht könnten ja
10. Volksabstimmung, und
26. Für Volksentscheide
das nächste Mal gemeinsame Sache machen?
Und vielleicht finden sich auch unter
18. 50plus
22. Die Grauen
30. Rentnerinnen und Rentner Partei
31. Rentner-Partei-Deutschland
das nächste Mal Schnittpunkte?
Der eigentliche Knaller sind ja aber die Berufe der Kandidaten auf der Liste der Piratenpartei Deutschlands: Dipl.-Wirtschaftsmathematiker, Dipl.-Physiker, selbst. IT-Unternehmer, Informatiker, Geschäftsführer, Student, Student, Programmierer, Web-Entwickler, Consultant. Alles klar. ;-)
Dieses Mal gibt es übrigens wieder den Wahl-O-Mat. Den hatten Sie ja letztes Mal wohl gerichtlich verboten, weil er nicht alle Parteien berücksichtigt hatte. Dieses mal sind fast alle 31 dabei. Toll ...
Tuesday, April 28, 2009
Choosing wisely
The upshot of last week's database news is: You have to make prudent choices when selecting the software you use. No one knows what MySQL, Sybase, Informix, Greenplum, Kickfire, or EnterpriseDB, to name a few, are going to do next week. If you have bet your business on selling a proprietary software application that links against MySQL, you may be in trouble soon. Just like MySQL has been more or less in at least serious threat of trouble for some years now by selling a proprietary version of their software that links against someone else's proprietary version of their software (InnoDB). Or, to leave the database realm for one sentence, by building their development process on a proprietary version-control system.
The question isn't so much whether Oracle will kill MySQL or MySQL will kill itself. Database deployments are very slow to go away, and MySQL's adoption is massive. The question is what will happen to the investment you have made when you chose MySQL. Switching to a different SQL DBMS is extremely painful. Drizzle may now have a BSD-licensed client library, but it isn't going to be a replacement for MySQL, unless you targeted your application for MySQL 3.23. (And Drizzle development, for what it's worth, is hosted on a proprietary hosting platform.) The other forks of MySQL aren't going to help you because they can't sell you license and they are not even allowed to provide you with an updated MySQL documentation.
Of course I will argue that PostgreSQL is somehow positioned more appropriately here, with the BSD license and the way the community works. But this episode shows that open-source software isn't just about free downloads or a fun community experience; it is about protecting your investments, your IT strategy, and ultimately your business.
The question isn't so much whether Oracle will kill MySQL or MySQL will kill itself. Database deployments are very slow to go away, and MySQL's adoption is massive. The question is what will happen to the investment you have made when you chose MySQL. Switching to a different SQL DBMS is extremely painful. Drizzle may now have a BSD-licensed client library, but it isn't going to be a replacement for MySQL, unless you targeted your application for MySQL 3.23. (And Drizzle development, for what it's worth, is hosted on a proprietary hosting platform.) The other forks of MySQL aren't going to help you because they can't sell you license and they are not even allowed to provide you with an updated MySQL documentation.
Of course I will argue that PostgreSQL is somehow positioned more appropriately here, with the BSD license and the way the community works. But this episode shows that open-source software isn't just about free downloads or a fun community experience; it is about protecting your investments, your IT strategy, and ultimately your business.
Labels:
Computing,
English,
MySQL,
PostgreSQL
Tuesday, April 21, 2009
Oracle, Sun, MySQL
So Oracle is to buy Sun now. Well, that makes sense. I guess I jumped ship just in time then. People started congratulating me yesterday that I avoided being an Oracle employee. :-) Although I was de facto a MySQL employee for a while; not sure which is worse.
John Dvorak, possibly of of the most tracked-back men these days, apparently knew all along that the acquisition of MySQL by Sun was just a proxy deal for Oracle. Now with IBM nearly snatching up Sun, I guess Oracle had to react.
The question inevitably arose, what will this mean for PostgreSQL. Probably not much. For one thing, you'd be overestimating the impact that Sun has on anything these days. ;-) The acquisition of MySQL by Sun eventually led to the (very small) PostgreSQL business at Sun being phased out, and it is possible that whatever is left will now fade away even more. But it really hasn't impacted the PostgreSQL community much, and won't do so now.
On the contrary, open-source databases, both MySQL and PostgreSQL, have always (at least since they became credible) been considered the anti-Oracle proposition. Now with MySQL actually owned by Oracle, this makes PostgreSQL the primary alternative. Considering the sometimes irrational but not insignificant ill-will that the Sun acquisition has spread among open-source database enthusiasts, it is easy to imagine how even worse this will impact MySQL's reputation on perception alone.
One might of course also think that Oracle now has a devilish plan to corner the database market from both sides. But it would be too presumptuous to assume that Oracle spends $7 billion on an anti-PostgreSQL plan. In some way, this basically only continues the path that Oracle started on when it bought Sleepycat and InnoDB a few years ago, and the impact of that on the open-source database field was pretty minimal.
I don't expect that MySQL will be "killed" either. It is too valuable as a "foot in the door" in any case. Moreover, much of the MySQL momentum already lies outside of Sun anyway, in the hands of Percona, Open Query, the Monty Program, Drizzle, and others, so killing MySQL is already impossible for a single company. Which is probably a good situation for the extended open-source database community.
What about the rest of Sun? I assume Oracle will be quite interested in Solaris and Java, of course, also Open Storage, Glassfish, and some of the other pieces of middleware that no one outside Sun really knows (yay, Sun marketing). Some of these pieces nicely fill the gaps in Oracle's offerings, reduce the interdependencies of Oracle with IBM and/or solidify the already existing relationships with Sun. The cloud computing initiatives might also be of interest, as Oracle has on occasion tried to become an Internet-enabled company no matter how silly the approach (remember the Network Computer or Oracle 8i/9i?). And as many have commented, keeping OpenOffice.org around just to annoy Microsoft is probably going to be worthwhile even if it doesn't make any money. I won't be surprised, however, if the hardware business, meaning SPARC on the one side, and StorageTek on the other, will be sold to someone else.
What I will be interested in seeing is whether Oracle can do a better job exploiting the vertical integration story that Sun has been been pushing for basically two decades but has repeatedly failed to deliver on. (Like MySQL not building on SPARC (at some point), and JavaFX not being available for OpenSolaris.)
I do feel sorry for the people at MySQL, who will have to readjust to a new employer and probably a new strategy and a few more sets of reorgs for the second time in less than two years. And all this a day before the MySQL Conference, which will now probably be completely distracted by this news. Add the conspicuous absence of any mention of MySQL in the official announcements, and the uncertainty couldn't be greater.
John Dvorak, possibly of of the most tracked-back men these days, apparently knew all along that the acquisition of MySQL by Sun was just a proxy deal for Oracle. Now with IBM nearly snatching up Sun, I guess Oracle had to react.
The question inevitably arose, what will this mean for PostgreSQL. Probably not much. For one thing, you'd be overestimating the impact that Sun has on anything these days. ;-) The acquisition of MySQL by Sun eventually led to the (very small) PostgreSQL business at Sun being phased out, and it is possible that whatever is left will now fade away even more. But it really hasn't impacted the PostgreSQL community much, and won't do so now.
On the contrary, open-source databases, both MySQL and PostgreSQL, have always (at least since they became credible) been considered the anti-Oracle proposition. Now with MySQL actually owned by Oracle, this makes PostgreSQL the primary alternative. Considering the sometimes irrational but not insignificant ill-will that the Sun acquisition has spread among open-source database enthusiasts, it is easy to imagine how even worse this will impact MySQL's reputation on perception alone.
One might of course also think that Oracle now has a devilish plan to corner the database market from both sides. But it would be too presumptuous to assume that Oracle spends $7 billion on an anti-PostgreSQL plan. In some way, this basically only continues the path that Oracle started on when it bought Sleepycat and InnoDB a few years ago, and the impact of that on the open-source database field was pretty minimal.
I don't expect that MySQL will be "killed" either. It is too valuable as a "foot in the door" in any case. Moreover, much of the MySQL momentum already lies outside of Sun anyway, in the hands of Percona, Open Query, the Monty Program, Drizzle, and others, so killing MySQL is already impossible for a single company. Which is probably a good situation for the extended open-source database community.
What about the rest of Sun? I assume Oracle will be quite interested in Solaris and Java, of course, also Open Storage, Glassfish, and some of the other pieces of middleware that no one outside Sun really knows (yay, Sun marketing). Some of these pieces nicely fill the gaps in Oracle's offerings, reduce the interdependencies of Oracle with IBM and/or solidify the already existing relationships with Sun. The cloud computing initiatives might also be of interest, as Oracle has on occasion tried to become an Internet-enabled company no matter how silly the approach (remember the Network Computer or Oracle 8i/9i?). And as many have commented, keeping OpenOffice.org around just to annoy Microsoft is probably going to be worthwhile even if it doesn't make any money. I won't be surprised, however, if the hardware business, meaning SPARC on the one side, and StorageTek on the other, will be sold to someone else.
What I will be interested in seeing is whether Oracle can do a better job exploiting the vertical integration story that Sun has been been pushing for basically two decades but has repeatedly failed to deliver on. (Like MySQL not building on SPARC (at some point), and JavaFX not being available for OpenSolaris.)
I do feel sorry for the people at MySQL, who will have to readjust to a new employer and probably a new strategy and a few more sets of reorgs for the second time in less than two years. And all this a day before the MySQL Conference, which will now probably be completely distracted by this news. Add the conspicuous absence of any mention of MySQL in the official announcements, and the uncertainty couldn't be greater.
Labels:
Computing,
English,
PostgreSQL
Friday, April 17, 2009
PostgreSQL videos on Vimeo
There has been the occasional talk of PostgreSQL video content lately, and many will probably have found this already, but for those who haven't, check out the PostgreSQL channel on Vimeo, or subscribe to the RSS feed for a regular delivery of educational PostgreSQL presentations on video. Great work, keep it up.
Labels:
Computing,
English,
PostgreSQL
Thursday, April 16, 2009
Web browsers vs. debtags
So I wanted to see what web browsers are available in Debian. The first stop was http://packages.debian.org/. Going to the page of one browser package and clicking on "Browser" in the tags area only gives you the explanation of the tag, but not the list of other packages with that tag. Is that available somewhere?
So next try maybe grep-dctrl ... oh, grep-debtags appears to be the ticket.
$ grep-debtags -n -sPackage web::browser
arora
browser-history
caudium-dev
caudium-modules
caudium-perl
caudium-pixsl
caudium-ultralog
chimera2
conkeror
cookietool
dhelp
edbrowse
elinks
elinks-lite
elvis
elvis-console
epiphany-browser
epiphany-browser-dev
epiphany-extensions
epiphany-gecko
ezmlm-browse
galeon
galeon-common
gtkcookie
iceweasel
jsmath
junior-internet
kazehakase
konq-plugins
links
links2
lynx
lynx-cur
lynx-cur-wrapper
mozilla-firefox-adblock
mozilla-noscript
netrik
netsurf
saods9
stripclub
sugar-web-activity
surfraw
w3m
w3m-el
w3m-img
wapua
claws-mail-dillo-viewer
dillo
konqueror
midori
How many of those are actually web browsers? Probably about half of them. (Example: caudium (not listed) is a web server, caudium-dev is its development package, not very close to a web browser.)
This would actually be quite a useful interface if the tags had any relationship to reality. I was in fact looking for a lightweight graphical browser, so this is a plausible command:
$ grep-debtags -n -d -sPackage web::browser -a interface::x11 -a -! suite::gnome -a -! suite::kde
which gives me 15 hits, of which 8 or 9 are actual web browsers.
Well, my search for a lightweight browser stopped here:
iceweasel
lightweight web browser based on Mozilla
Yeah! ;-)
So next try maybe grep-dctrl ... oh, grep-debtags appears to be the ticket.
$ grep-debtags -n -sPackage web::browser
arora
browser-history
caudium-dev
caudium-modules
caudium-perl
caudium-pixsl
caudium-ultralog
chimera2
conkeror
cookietool
dhelp
edbrowse
elinks
elinks-lite
elvis
elvis-console
epiphany-browser
epiphany-browser-dev
epiphany-extensions
epiphany-gecko
ezmlm-browse
galeon
galeon-common
gtkcookie
iceweasel
jsmath
junior-internet
kazehakase
konq-plugins
links
links2
lynx
lynx-cur
lynx-cur-wrapper
mozilla-firefox-adblock
mozilla-noscript
netrik
netsurf
saods9
stripclub
sugar-web-activity
surfraw
w3m
w3m-el
w3m-img
wapua
claws-mail-dillo-viewer
dillo
konqueror
midori
How many of those are actually web browsers? Probably about half of them. (Example: caudium (not listed) is a web server, caudium-dev is its development package, not very close to a web browser.)
This would actually be quite a useful interface if the tags had any relationship to reality. I was in fact looking for a lightweight graphical browser, so this is a plausible command:
$ grep-debtags -n -d -sPackage web::browser -a interface::x11 -a -! suite::gnome -a -! suite::kde
which gives me 15 hits, of which 8 or 9 are actual web browsers.
Well, my search for a lightweight browser stopped here:
iceweasel
lightweight web browser based on Mozilla
Yeah! ;-)
Subscribe to:
Posts (Atom)
Posts
