Wilmer's stuff

Bank statement scraper for Bank of Ireland

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2011-08-01T15:14:35Z

Like many people, I was also losing track of my finance. Having bank accounts in use in both NL and IE probably didn't help. :-) As any proper FOSS geek, I learned to like the monster called GnuCash. (Psst! Guys! It's pretty amazing that a product more than ten years old still doesn't let you do operations (like delete) on multiple entries at once, dont you think?)

And there's this thing about Irish Banks. They have bigger issues to worry about than how well their Internet banking service works. What keeps you away from looking at my bank account? You (hopefully) not knowing my six-digit user ID, date of birth (top secret information! Have I mentioned that my birthday is next Saturday? ;-P) and another six-digit number, this time my PIN number. No one-time passwords, no challenge-response system, nothing else.

My only hope is that this lets you transfer money only to accounts to which I've transferred money before. IOW all you can do is give my landlady a little present. Pfew!

Also, going back to the original topic, there's no way to export info from their web interface. So I wrote one myself. One advantage of a pretty simple website is that I could easily write a scraper for it. Run it with the right arguments, and it'll spit out a CSV bank statement, ready to be fed to your favourite accounting software.

What else have I been doing? Been working on Giggity. Android development's fun. I spent the weekend scraping the Dance Valley timetable page, Google, Last.FM, Wikipedia and more to automatically generate a Giggity schedule file for it. Love it! :-)

On Pandaboard SD card performance

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2011-06-13T21:03:53Z

I have the Pandaboard running as my home server for a while now. Until last weekend, I was using a Microdrive as its root filesystem. Sadly, the drive seems to be broken. :-( That means I finally had a chance to try bootstrapping a server very quickly using Puppet. This worked fairly well, which means the time investment is paying off already.

Since all the storage I had at home was the 32GB SD card I bought for this thing anyway, I decided to give it another chance. At some point I was reminded already that alignment really matters with these things. Some Bonnie++ runs do seem to confirm this. I removed the second partition on the SD, and recreated it on a 4MB barrier. (The trick to do this is to use the "u" command in fdisk to switch units to sector instead of cylinders, and make sure the start sector is a multiple of 8192.)

To be honest, I did run most of these benchmarks with the SD card reader/writer in my desktop machine. Only the last test was done on my Pandaboard, but as you can see the results are very similar.

Version 1.96		Sequential Output						Sequential Input				Random Seeks			Sequential Create						Random Create
	Size	Per Char		Block		Rewrite		Per Char		Block				Num Files	Create		Read		Delete		Create		Read		Delete
		K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU		/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU
ext3-noalign	4G	189	41	3514	1	4878	1	2646	98	21147	2	20.9	0	16	463	0	+++++	+++	2832	3	3214	4	+++++	+++	3221	4
	Latency	6643ms		28833ms		19841ms		6668us		483ms		47888ms		Latency	1335us		655us		933us		531us		85us		60814us
ext3	4G	437	93	3727	1	5631	1	2725	99	21484	2	21.5	0	16	938	1	+++++	+++	4303	5	3743	6	+++++	+++	4304	5
	Latency	412ms		32024ms		5622ms		6422us		469ms		52114ms		Latency	1574us		760us		713us		1644us		13us		585us
logfs	4G	1149	75	13670	5	3530	4	1067	81	5920	7	6.5	3	16	2033	9	+++++	+++	4731	22	1755	8	+++++	+++	1516	11
	Latency	10944us		1810ms		4562ms		29505us		54279us		2401ms		Latency	568ms		9300us		7881us		2783ms		1483us		570ms
nilfs2	4G	797	95	7118	2	5186	1	2711	98	24112	2	617.3	29	16	2102	36	+++++	+++	+++++	+++	4592	71	+++++	+++	10886	56
	Latency	20454us		2767ms		2788ms		25974us		21810us		3432ms		Latency	4928ms		1288us		1020us		1339us		358us		294us
ext4	4G	443	91	18315	2	9299	2	2602	99	25041	2	22.4	0	16	3567	8	+++++	+++	4369	7	3792	7	+++++	+++	4393	6
	Latency	32264us		920ms		926ms		12851us		14137us		5042ms		Latency	530us		1451us		1316us		393us		401us		771us
ext4-panda	1496M	106	96	16838	8	8865	7	663	99	22243	12	30.1	1	16	3479	26	+++++	+++	4035	23	8760	65	+++++	+++	10388	63
	Latency	78219us		3925ms		961ms		13732us		104ms		1251ms		Latency	1190us		1801us		1892us		762us		61us		671us

Click here for a table not f*cked up by my blog software.

Although the throughput numbers for ext3 are pretty similar for non-aligned and aligned access, look at the latency numbers. Unfortunately I haven't got a clue how Bonnie++ calculates these and can't find very good documentation on it. Throughput may be average and latency worst-case? Either way, as you can see a misaligned partition can cause some slowdowns.

What surprised me more is that a switch to ext4fs sped up things a lot more, up to the point that the performance is perfectly reasonable! I'm running with this SD as my root filesystem now and everything just works. (While before a simple apt-get install run could take several minutes.)

While I was at it, I also tried out logfs and nilfs2, which are officially optimised for flash media. However, AFAIK they're more meant for raw NAND storage, not for block devices with all the NAND logic abstracted away (like anything you buy in stores these days). Not worth it for these SDs.

Obviously this test is far from scientific. Only in the case of ext4-panda have I run the test five times to then pick a decent result (there were some outliers in all areas). All other tests were done on a freshly formatted filesystem, which I'm sure also doesn't make the result that reliable.

Just my 2 cents! But my Pandaboard's definitely happier now. Here's hoping that wear leveling works well..

If you're interested, here is a more thorough overview of SD card performance. The LWN article about flash storage it links to is interesting too. The Flash card I used here is a 32GB class 10 Transcend card.

Splitting PDFs with pyPdf

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2011-06-03T22:04:02Z

A simple task, yet I couldn't find a quick cmdline to do it with, apart from pdftk, 15MB of Java rubbish.

Instead, here only 10 or so lines of Python. It was so fast I wasn't sure if it worked until I saw the results were there. Usage: split [prefix] [infiles...]. Multiple infiles possible. First argument is the filename prefix to use for all created files.

CODE:

import pyPdf
import sys

n = 0
for f in sys.argv[2:]:
f = pyPdf.PdfFileReader(open(f))
for p in f.pages:
of = pyPdf.PdfFileWriter()
of.addPage(p)
of.write(open("%s-%03d.pdf" % (sys.argv[1], n), "w"))
n += 1

Don't pay attention to Serendipity screwing up the code layout. We all know it's rubbish, I just can't be arsed to migrate to something better. :-/

dnsrev

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2011-05-24T00:09:14Z

As a bit of a cloud "sceptic" I still like to waste too much time maintaining my own network/IT infrastructure. :> I'm definitely trying to avoid the more tedious stuff though. I started using Puppet a while ago which definitely helps.

Last week I was looking for a way to automatically populate DNS reverse lookup zones. The only thing I could find was mkrdns which is unmaintained for almost ten years and doesn't seem to support IPv6. So I decided to write my own thing, dnsrev.

It's pretty simple, written in Python with help from some modules. It can read any number of zonefiles and update any number of reverse zonefiles. There's no need for any kind of 1:1 mapping between them, so it can deal with multiple netblocks in one zonefile, etc. I hope it'll be useful to someone. Comments, suggestions and patches are welcome.

Shiny happy hardware

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2011-04-23T13:05:00Z

For years I'm using Winterms as simple home "servers". It was a fun project to work on and some people were even nice enough to send me some examples of more powerful (relatively, we're talking about ~300MHz here at most) hardware. Two of them are still working as nameservers/printservers and one of them even hosted the Winterm hacking website for a while.

But they're getting old, slow, and pretty painful to upgrade. Time to move on I'm afraid. So before my last trip to the US, I ordered two shiny pieces of hardware: A Pandaboard and an Nvidia Tegra developer board. Due to circumstances, I didn't really expect both (or even either) of them to arrive - Nvidia seemed to send the board only to people who have projects they find interesting/important (stuff like the Motorola Xoom probably), and the Pandaboards never seem to be in stock.

Yet, here I am with both of them, wondering which one to actually use. :-)

Left: Pandaboard, right: Tegra2 250 Harmony board

I guess I'll just write down my findings here so far. I'll probably end up using both, one as a server and the other one to run stuff like xbmc on my TV.

Both boards seem quite similar, spec-wise. Two 1GHz ARM cores, 1G of RAM, USB, sound, networking (including WiFi and Bluetooth), HDMI output, and an SD card slot. The Pandaboard has an internal antenna, no clue about the range.

Although both boards' USB ports apparently aren't really meant for powering 2.5" USB HDDs, it seems to work quite well anyway. Which is good, because SD cards as root filesystems seems like a bad idea. Did you know that (according to bonnie++) a desktop hard disk from 2007 outperforms SD cards (at least in the Pandaboard and Tegra) not just on sequential reads, but also on seeks? So yeah, I may be using USB HDDs instead, which sadly means more power usage. :-( Especially in the Pandaboard SD performance is too bad to be usable.

One big advantage of the Pandaboard seems to be the community. A pretty busy (and generally helpful) IRC channel, lots of info online on Wikis. The Pandaboard iss "just another OMAP architecture" so lots of stuff that worked for BeagleBoard should work on the Panda with some customizations. Canonical/Ubuntu also support the thing officially.

Here comes the biggest contrast with the Tegra. Nvidia seems to be too busy with Android, the result is that there's little support for doing other stuff with the board. The only thing you get for now is L4T (Linux 4 Tegra), which is an Ubuntu Jaunty (yes, 9.04, that's two years ago by now..) image you can run on it. There are efforts on getting Lucid to run, don't know where those are ATM. But one complication there is some binary-only drivers/helpers (like nvrm_daemon, which I guess manages the memory shared between OS and video/etc), which means troubles getting X to work after an upgrade. Ouch.

The Panda also certainly wins in the bootloader department, as it just loads uboot stuff from a FAT partition on the SD card (tricky part here is that if you do anything wrong with the partitioning and formatting of this SD card, the boot process will just fail silently). For flashing the Tegra you need a proprietary fastboot flasher binary. Possibly, once booted, I can just write my kernels to NAND myself from inside the OS, but I haven't yet tried this.

So yes, with this all in mind, it's a delight to run a normal (and not outdated) Debian/Ubuntu install on a Pandaboard. Video is also supposed to work flawlessly almost out of the box on Ubuntu. However, I seem to be unlucky/doing it wrong since the framerate is not impressive, and playback seems buggy. (While the little video playback I've done on the Tegra was pretty good, super smooth, and with only 10% of CPU usage!)

For my original goal, running a simple home server, I feel that both boards are suitable - I'd just run Debian inside a chroot on the Tegra so the helper daemons (and maybe some video stuff) can run outside it. But before I get this video stuff to work, I have some work to do. And hopefully, if I wait for long enough, some other patient souls out there will also fix some of these problems...

Xen horror, just upgraded my box to Squeeze

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2011-02-13T23:49:48Z

Of course I could just not upgrade, but I'd have to do it sooner or later anyway..

It looks like Xen, now that it's owned by Citrix, also suffers from the XML manager syndrome. At least the extremely annoying clock bug was fixed, which means I get >4ms precision in timing again.

Just dumping this here in my blog since more people seem to have this problem and aren't getting very helpful answers so far. Or maybe I just didn't try the right Google queries..

This:

hypnotoad:/tmp# xm suspend bijtje
Error: Domain is not managed by Xend lifecycle support.
Usage: xm suspend <DomainName>

Suspend a Xend managed domain

means you're not using the shiny new (whatever the purpose is) lifecycle tool that keeps track of domain configs and other stuff in /var/lib/xend/domains/$UUID/. You can use "xm new" to set this up, except this is broken on Debian:

hypnotoad:/tmp# xm new
Unexpected error: <type 'exceptions.ImportError'>
...
ImportError: No module named xmlproc

This, my dear reader, means that Debian's shipping Xen utils that depend on a Python XML module that was actually removed from Debian since it's not maintained by upstream anymore.

So I was just trying to figure out how to make this all go, and then I realised:

hypnotoad:/tmp# xm save bijtje bijtje.sav
hypnotoad:/tmp# xm restore bijtje.sav

Oh look! I can still do it. I just have to tell Xen where to save the statefile.

So in short:

Xen is also suffering from XML-itus
Debian drops packages that other packages still depend on
Use "xml save", not "xml suspend"

Burn all spammers!

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2010-08-08T14:31:06Z

I have a habit of always having a tail -f /var/log/mail.log running on my mailserver somewhere. It's noisy, but has been useful in the past. Over the last weeks/months, I noticed open relay probes are getting incredibly popular (again), but also extremely aggressive. They're frequent, done by hundreds of botnet drones all the time.

Obviously my Postfix is configured properly, so this is mostly a waste of (fairly scarce, on a DSL box several km away from the exchange) bandwidth and annoying noise in the logs. But getting rid of it is harder than I hoped. :-(

This is what I have now: iptables -I FORWARD -p tcp --sport 25 -s 192.168.0.0/16 -m string --algo kmp --string '554 5.7.1 <' -j REJECT --reject-with tcp-reset

This works as-in it kills the connection as soon as my mailserver sends a "554 5.7.1 Relaying denied" response. The REJECT goes to the mailserver, but together with the tcp-reset this also kills the TCP connection on both sides fairly quickly. However, the little fuckers are also using pipelining, so I still get a screen full of logspam for pretty much every attempt. Although this is mostly cosmetic, I'd love to get rid of that crap..

What I really wonder is, WTF are they even doing this? Are open relays really still that common? Don't they have their botnets already? I guess the open relays are nice multipliers and are also more willing to deal with stuff like graylisting...

[edit]Looks like "554 5.7.1" is not just about "relaying denied", so possibly not such a great idea. Don't try this at home!

BitlBee, alive and kicking

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2010-06-10T23:43:23Z

As quin in #bitlbee said a little while ago, I stole someone's mojo and found an amazing amount of productivity when it comes to writing code, and it feels great. I'm quit relieved that I can still find plenty of time and motivation to work on BitlBee even though during the week I already spend a lot of time at the keyboard. This after not working much on it for probably at least a year.

I managed to finally do the IRC core rewrite + abstraction that I intended to do for so long already. It'll allow adding non-IRC frontends to BitlBee if someone ever wants to, and also the IRC core has the flexibility it needs to add many more features that I wanted for years already, and were impossible to implement without adding even more horrible hacks.

There's also a libpurple-based backend for a few months already, plus file transfer support (written by Uli Meis and Marijn Kruisselbrink actually, it just took me a long time to merge the >3000-line diff, fortunately Review Board did make it a lot less painful), all thrown into a bleeding edge branch called killerbee. It's code that needs a little bit more work before I really like it.

Also, BitlBee has Twitter support for about two months already (thanks to hard work done by Geert Mulders), and according to the application registration page on Twitter it has almost 500 users already. It's quite likely that many of those used it for five minutes and went back to a client with more features, but it's still nice to see.

Last of all, to help with the current lack/fragmentation of online documentation there's now a BitlBee Wiki. Its supposed to have easy-to-find docs about common FAQs, but the easy-to-find part isn't really working out yet since it hardly shows up in any search results. Hopefully this hyperlink from a high-profile weblog will improve that a tiny bit. ;o) Possibly the content is not that good yet either, so if anyone has something to add to it, by all means, please do!

With a 1.2.8 release coming up, BitlBee is totally alive - and is for almost eight years already. It's been a fun project to work on so far, and hopefully will be for a long time.

Sending files using netcat

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2010-05-22T20:34:34Z

There are plenty of articles about this already, but I couldn't find anyone who wrote a script to automate this simple task of transferring files using just netcat.

CODE:

wilmer@ruby:~/src/bitlbee/devel$ ncsend.sh /audio/03\ Tree\ of\ Life.mp3
nc 87.198.255.202 6886 | pv > 03\ Tree\ of\ Life.mp3

Run it and it will give you a command to run/copy-paste on the remote side/pass to the person who wants the file. It uses pv as a nice progress indicator, and the script assumes pv is available on both sending and receiving ends. But that's good, everyone should have pv installed on his/her machine.

I'd just include the code in this little article, but Serendipity would screw up the layout completely, so instead you can download it here.

Gotta love Unix hacks...

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2010-05-05T23:35:07Z

Forgetting your LUKS cryptopart password sucks. But writing a shell oneliner like this:

CODE:

for i in P{a,4}ssw{0,o}rd{-,}Permut{4,a}tions; do
echo -n $i > /tmp/pwd
cryptsetup luksOpen /dev/loop0 test --keyfile /tmp/pwd && echo $i
done

is awesome. :-) Took me only ten minutes to get it back once I had a copy of the superblock onto my workstation.

spamass-milter and IPv6

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2010-01-31T11:51:51Z

While migrating my mailserver from Ubuntu Dapper to Debian Lenny, I noticed spamass-milter didn't want to start:

Could not parse "2001:770:017b::" as a network

After scratching my head on that for a while (it worked on the old box!) I remembered two years ago I spent some time adding IPv6 support to spamass-milter myself. Support as-in allowing IPv6 subnets to be whitelisted/auto-accepted. Very useful if you want mails from your local IPv6 machines to be accepted automatically without waiting for 10s while spamassasin is checking if you're not a spammer...

I never published the thing and now I could hardly find back the damn thing at all. :-)

http://wilmer.gaast.net/downloads/spamass-milter-ipv6.diff

Just to make sure I won't lose it again ... and maybe it'll be useful for someone else.

Also just de-Ubuntufied my laptop. Debian's doing great so far: Suspend and Resume actually work better out of the box, but fonts look a bit ugly (and sometimes really less readable I'm afraid) compared to Ubuntu...

edns-client-ip

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2010-01-28T10:16:09Z

Finally, the Internet-Draft I was working on with my team over the last months made it to the public.

http://googlecode.blogspot.com/2010/01/proposal-to-extend-dns-protocol.html
http://www.ietf.org/id/draft-vandergaast-edns-client-ip-00.txt

This draft was the main purpose of my visit to IETF76 last year. I'll have to go to 77/78 this year then to hopefully get this idea accepted as a working group item.

It's interesting to see all the comments coming up claiming that this is evil and meant for tracking purposes just because it has the name Google on it... :-/

Running 32-bit 3D apps on 64-bit Debian NVIDIA systems

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2009-12-30T12:46:22Z

Because Intel on-board video isn't quite good enough at 3D and my somewhat old laptop with built-in NVIDIA chip wasn't doing all that well at X-Plane either, I bought an NVIDIA GT240 card. Mostly because it seemed to be a good performer without doubling the power consumption of my PC. Stuff wasn't working all that well though and I got pretty frustrated:

QUOTE:

X Error of failed request: BadLength (poly request too large or internal Xlib length error)
Major opcode of failed request: 135 (GLX)
Minor opcode of failed request: 2 (X_GLXRenderLarge)
Serial number of failed request: 1468
Current serial number in output stream: 1483

was all X-Plane could tell me. Google Earth also didn't work and seemed more like Google Black hole to me. Both are 32-bit apps. A 64-bit binary of Flightgear did work. Sigh.

After some poking I noticed the nvidia-glx package replaces /usr/lib{64,}/libGL* with its own versions, but didn't touch /usr/lib32. A-ha!

The fix: Get a 32-bit version of nvidia-glx, extract it somewhere (dpkg -X) and copy all the files in its usr/lib to /usr/lib32, overwriting the libGL symlink that is currently there.

Or, hmm, as I just found out: apt-get install nvidia-glx-ia32. I'm glad someone thought of it already.

Now X-Plane works perfectly (with maybe even ten times the frame rate I had with Intel on-board) and Google Earth can show me my house again.

Just blogging this since so far a Google search for any part of the error above didn't give useful results.

Now, I just have to find out if I can get a TV signal out of this thing somehow, since my TV was made long before HDMI was invented. :-/

Google Public DNS

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2009-12-04T00:13:03Z

Today at work my first user facing product was released, Google Public DNS. I've worked a lot on it over the last months and it was actually the reason for my visit to NY this Summer. :-)

But now, finally after a long time, I can actually start calling the thing by its name instead of saying "my secret project at work", which makes me very happy.

Praise, complaints and comments about the service are welcome! :-)

Back from Japan

blog@wilmer.gaast.net (Wilmer van der Gaast) — 2009-11-28T16:06:55Z

I'm back ... for almost two weeks already. Getting used again to bad weather (today's a very foggy and cold day in Dublin), working and all the other things of daily life. Japan was fantastic. I'll be careful to not sound like your average "everything's better in Japan" idiot, but admittedly, there were some things there I really liked.

One of the first things I noticed was that a rule about escalators I really like is actually enforced over there. Wherever I've been so far escalators are always full of people who don't understand how stairs work; they think the fact that the escalator goes up means they can just put their lazy feet on it and wait (forgetting that for lazy people we already invented elevators :-P). Many civilizations have tried introducing a "stand still on the left, walk on the right" rule, but Japan's the only place where I've seen people all obeying it.

Getting around there indeed turned out to be challenging at times. Even though people (at least from "my generation") learn English for about ten years at school, many people still don't or hardly speak it. This was never really a problem though. Menus in restaurants have pictures and pointing at things is the most successful international language ever invented. :-) One of the first days I had sushi in a place just a few steps away from my hotel in Tokyo. While eating my meal, the man behind the bar (probably also the owner of the place) came to me with a map, asking me to point at where I come from. Also, the little origami bird he folded for me from my chopsticks wrapper is still sitting on my desk here. :-D

The ramen place I went to for breakfast one of the first days was also amazing. Not so much because of the food (which was also good), but because of the size. Literally, it was a room of maybe just over 2m wide, about 10m deep, with a bar separating it into two small 10m × something areas. Two people working on one side, and ten seats on the other (all occupied, most of the time).

My camera claims I made over 700 pictures there. Obviously there are many duplicates and worthless pictures, and after soring there will probably only be around 100 left. I hope to put them online soon, and probably with some more stories. I wrote too much now already.

All I know is that I'm definitely going there again.