Friday 20 July 2007

Vacation, yes!

Starting tomorrow I will be on vacation until the 28th of July. Excellent! So if you expect some thing from me, just keep it on hold until the next month :-P .

One week of sunbathing on the Bulgarian seaside. You might ask why Bulgaria and not Romania since both have access to the same sea?

It appears that my predictions about the price of a ticket for a vacation on the Romanian sea side exceeding the price of one in Bulgaria or Greece (including transport) came true (Some might say this was the case last year, too, but this year is more obvious). I'll see if the services will be better, afaik, they should be.

Wednesday 18 July 2007

How to build rrdtool with soft-float for a debian arm machine (which uses hard float)

Being a good Debian user

Notes: the bulk of this article was written on the 17th; updates were added in later; this article is really long, you might want to read it in more sessions

Debian's arm port uses hardfloat. This means that apps like rrdtool (graph function) take ages to run because there is no floating point unit. In order to make this faster you'll want to use soft-float emulation. But this support needs to be present in all depended upon libs, including libc, which is an ABI incompatible change which would break all binaries. I can't use the yet-in-development-and-unofficial armel port since the machine has to be up and running almost 24/7 and I can't afford downtime.

So what to do?

Well, obviously use another soft-float enable libc, like uclibc, in parallel withe the hrad-float enabled libc. Debian Sid contains uclibc development libs, but building the whole toolchain is a pain. There is no real support in the distro itself and building on the arm machine is kind of out of the question. I tried a little in a debootstrapped chroot on the native machine, but this was a pain and I was stuck tring to figure out how to disable whitles and bells for rrdtool since I didn't needed any of the perl, tcl or python support. So I stopped, went to bed and decided I'll continue the next day (the day I wrote the biggest part of this article). The future looked dark, trying to build a softfloat uclibc rrdtool on a system that wasn't prepared for that and from which softfloat was pulled away somewhere during etch's development stage....

Deciding to go with the competition - native building

From link to link I found out that gentoo has arm-softfloat-linux-uclibc port, among others, so I decided to use that on the target machine (a NSLU2) in a chroot. Well I stumbled on the low computing power, when I tryed to emerge sync the tree. I gave up un that and tried to use emerge-websync. It looked better, but I didn't got too far with this either since the machine ended up being overloaded, with figures varying from 3 to 8 for the load average (all of them).

Deciding to go with the competition - cross building

It was time to bring in the big guns and I decided I was going to crossbuild those binaries, probably statically since it meant lower risk for things to break on the target machine.


Since I observed that gentoo has a nice tool called crossdev which actually uses emerge (the equivalent of dpkg+apt in Debian world) to make the cross chain tool and saw that they have some really good documentation and a development version of an arm-softfloat-linux-uclibc, I decided I could .

So I went on and downloaded the latest stage3 tarball for x86 (so I can have a gentoo system to start with), made all the necessary things and created a gentoo chroot. I emerge sync-ed, updated portage, installed crossdev (following the nice instructions on the gentoo embedded site about the subject). This took a while, but it went like a breeze. I made the necessary arangements to be able to cross build packages (setting SYSROOT, making that nice xmerge script).

As my goal was to make an arm-softfloat-linux-uclibc binary for rrdtool I tried directly to xmerge rrdtool (of course, first just pretended -p).

Finding bugs in the packages

First bug

All went fine until I found freetype which for some weird reason detected the i486-pc-linux-gnu-gcc compiler as i486-pc-linux-gnu (observe the missing trailing -gcc). I looked over the logs and determined that the CC_BUILD environment variable was not set. So exported CC_BUILD=i486-pc-linux-gnu-gcc and xmerged the library.

(While trying to fix this issue I entered #gentoo-embedded on freenode and actually met Yuri Vasilev, whom I had met in Edinburgh, at DebConf. Look for the video about Ligusk if you are interested about his project to make a Debian like distribution based on Gentoo. It was nice to meet a friendly "face" - as much you can say that online.)


I reported the bug (already there is a patch, I haven't tested, but looks ok) and the workaround in the gentoo bugzilla... after creating an account, since I never used that BTS before.

Second bug

Next pain was libart_lgpl whose upstream, for some comfortability reason chose to detect the target type sizes (and basically ignoring inttypes.h) with a small program which was built with the cross compiler which later was supposed to be ran on the host. I don't think is necessary to say that i failed, of course, since the binary was an ARM binary, while the host was an x86 machine. Yupee! not.

Ok, two bugs, no softfloat rrdtool yet. I reported this second bug and tried to work around the issue. Since I didn't really used too much a gentoo system, I wanted to know details and what possibilities I had. I got some really nice tips from "solar" (aka Ned Ludd) on #gentoo-embedded. He pointed out that while building a package one can press CTRL+z, do some stuff, like altering the Makefile.in file in the build directory and then fg back. Ok, cool.

All I need now is somebody with direct access to an arm machine that can compile and run for me the program in question. seanius on #debian-devel was kind enough to do this for me and, to my surprize, I got the same output header file as I got in the host x86 chroot. But that detail is not important, let's just compile that rrdtool. Well, first libart_lgpl...

Ok, I CTRL+Z-ed the build process just after the configure stage ended, cd-ed into the build directory, I tampered with the Makefile.in file and changed that line from

./gen_art_config > gen_art_config.h

into

cp /gen_art_config.h.arm gen_art_config.h

Haha! In your face! Ok, nasty did was done so I fg-ed and went on confident that I will get my hands on that rrdtool very soon. The package build ended succesfully, without any other suprizes, so I went for the graal!


xmerging rrdtool, the bugs don't stop

xmerge -v rrdtool

Bla, bla, bla.... bla, bla, bla, what do you know, there are a few IEEE math function tests in the configuration script of rrdtool. They failed. I don't know why, but I have a feeling that this should be happening on softfloat or even maybe the rrdtool source is not that cross compile aware. I didn't care that much, so I did what I knew best, tampering with an ongoing build.

This time is was a little bit harder to catch since I had to wait for the source to unpack (well, there is a larger frame, but that is the rough timing) and at that moment CTRL+Z the build.

I didn't had much time to waste, so after looking shortly over the tests I realized I should just ignore the error and look for the "failing command" that was interrupting my beloved build. Muhahah!

Thank God for verbose error messages, I could find the place fast and chaged the two "exit 1" statements into two harmless 'echo "Ignoring IEEE math errors"', sic. I had to do this a few times since I got the timping wrong, at some point I got the syntax wrong (at which point I remembered the cp trick I did earlier and saved the modified file in /configure.rrdtool )...

xmerging rrdtool and the 4th bug

After passing past this stage, I got a compilation error which, judging from the command, meant that the linker tried to link the target (arm) binary against the host (x86) libraries.

Oooook, this is not going to stop me! I look at the Makefile.in to try understand why this happened and then realized it wasn't worth it for me to try to fix it the right way. So I cheated once again! Muhaha, I just ran some useful commands which I'll let you feast your eyes to:

ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libart_lgpl_2.so /usr/lib/
ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libpng12.a /usr/lib/
ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libfreetype.so /usr/lib/

Ok, good to go. Of course, I had to stop again the build to copy the "fixed" configure script that ignored IEEE math errors, but that is already history.

And, believe it or not, actually the build succeded. I had a cross built rrdtool using shared libraries built against softfloat ucllibc. No guarantees about its correctness, but it built!

This is when I decided to start writing this article.

Still there are things to do

I will have to do a few things before being happy about the victory:
  • check that all the resulted binaries are indeed ARM ELF files, and not Intel x86. It appears the shared libs are already.
    • done, they are (checked the next day)
  • figure out qpkg (is a utility from portage-utils which seems to be a way to make binary packages) to package somehow the result of this work or maybe just ignore it and use plain cp
  • see how I can fit together both the softfloat uclibc binaries and the native debian binaries on the same filesystem, including ld.so configuration...
    • but I fear this will be a nightmare, so I might decide to go with static compilation of rrdtool, which I don't think it will be quite straight forward given the experiences describe above; there could be also the size issue to take into account, and since the NSLU2 has only 32MB of RAM, this could be problematic
      • I think I'll have to use a statically linked binary after all (note added late, the second day)
  • see if the rrdtool binary really works and if all the work payed off, performance wise - it should, taking into account what I have heard from wookey during the debconf7 talk about the armel (warning, 73MB ogg file) port
  • be happy with the solution I got or wait for the Debian armel port to catch up and switch to that asap
    • if I'll have to do the switch I'll probably think of ways of migrating the whole system from arm to armel, and not to reinstall everything; this might prove useful for the whole debian armel port, when people are working on migration plans, since arm will be deprecated in favour of armel

Friday 13 July 2007

glest: a free 3D real time strategy game

After almost a year of working on and off on the package, glest is now in the new queue.

Glest is really a free 3D real time strategy game with amazing graphics and really interesting game play.

So maybe you're wondering why did it took me so long to package it. Well, glest is mainly developed on a Windows platform and GNU/Linux is the platform it was later ported to. So this came with some problems:
  • the game uses jam as a build system; don't get me started on the intelligence of this build system; I had to go through real pain to make it build
  • the game is entirely written with "little endian" assumtion in mind
  • apparently the editor is not functional on Linux yet, but there is a patch (I will enable this later, after some testing)
  • the build system was not bootstrapped before building the tarball
  • the linux build creates some symlinks and I had to do some really nasty tricks so that there are no unrepresentable changes in the source
  • the upstream is currently not that active, but it used to be a lot less last year; it seesm that there is a motivation/time issue with the leader of the project
  • upstream moved the project to sourceforge and they keep the code now in subversion, but I wasn't aware about this until recently
  • there are a few people talking about forks and a real fork which proclaims itself ast the real project continuation; I suspect that all this mess appeared due to inactivity of upstream last year
  • the game used a non-free font
Since when I started the packaging work I was using a PowerBook G4 as my main machine I was upset that the game was little endian only, so I started working on a patch. I made an incomplete patch but I got stuck at some point. Time passed by and I replaced my laptop and I wasn't able to continue working on the patch. So I kind of forgot about it. Before debconf I decided is time to do the upload for little endian machines, add the endianess patch disabled.

During debconf I asked Tolimar to make an upload, and it turns out that for some reason I still don't know, the build resulted in a statically linked binary. I was amazed since I didn't recall doing any changes that might have triggered this. At some point I suspected that the build system is broken on other arches than i386 (my main machine is amd64) and halted. This until a few days ago Joey Hess added a comment in the ITP bug pointing out that the game is ok. After a couple of emails I realized that the static issue was just some strange temporary issue on my machine and now the package is in NEW.

So, that's my excuse. I hope that upstream glest issue get fixed. If not, I'll probably package glevolution, too.

Tuesday 10 July 2007

R U a programmer or a hax0r?

I have a few questions:
  • when you write your new piece of code, be it in bash, C, python, haskell, ruby, perl or whever langauge, do you think about possible future uses? do you care about those when designing it?
  • do you take good care to make sure you can expand your data containers at a later point in ways which you didn't envisioned initially?
  • are you able to implement a feature which is cool 2 year after your application is in production? how about another feature? how many until you realize you can't anymore or you need to redesign and rewrite the whole thing?
  • do you care if your interface remains compatible?
  • how easy is it for you to jump into somebody else's code and fix it to do what you want? ...without breaking the original functionality?
If you are programmer you will know what you should have answered...

Update: this post was tagged as "rant" and the rant was about a badly written and too rigid code; this was not about trying to tell the future and implementing now what you might need later

More "RPM argh" or how to use %includes properly

It turns out that if you want to use includes in your .spec file and want the files to be included in the .src.rpm you have to declare those scripts as supplemental sources.

So instead of:
Name: foo
Version: x.y
Release: z
Summary: foo package
Group: Applications/System
Source: foo.tar.gz
Packager: Eddy Petrisor
BuildArch: noarch
BuildRoot: $RPM_BUILD_ROOT
....


You have to have:

Name: foo
Version: x.y
Release: z
Summary: foo package
Group: Applications/System
Source: foo.tar.gz
Source101: foo.pre.sh
Source102: foo.post.sh
Source103: foo.preun.sh
Source104: foo.postun.sh
Source105: foo.changelog
Packager: Eddy Petrisor
BuildArch: noarch
BuildRoot: $RPM_BUILD_ROOT
....

In case you want to have the scripts separated in different files and you use them all.

Note: the number in the "SourceXXX" is arbitrary but must be unique.

But mentioning the files as being Source... means you have to write:


%post
%include ../SOURCES/foo.post.sh

%preun
%include ../SOURCES/foo.preun.sh

%postun
%include ../SOURCES/foo.postun.sh


Instead of:


%post
%include foo.post.sh

%preun
%include foo.preun.sh

%postun
%include foo.postun.sh


Which should have been more reasonable...

RPM, please die... or grow some 21st century features... or some sense!!!


Update: Thanks to a few people, I was pointed out I am still not thinking in terms or RPM.
The proper way of writing the include without them looking odd is something like:

%post
%include %{SOURCE102}

Friday 6 July 2007

RPMs, gaaaaaah!

Always when I have to deal with RPM systems I feel so handicapped and it feels so wrong from multiple PoV.

This is a rant and a dear lazyweb (not that lazy, actually) type of post.

I would like to be able to have a few things solved when dealing with RPMs:
  1. I would like a pbuilder like builder for specs
    • I know about mock, but
      • it accepts only src.rpm files (I know about rpmbuild -bs, but is broken, read further for the groovy details)
      • I have some specs that have %include directives for the maintainer scripts and when rpmbuild -bs is ran, guess what, the extra files are left outside the src.rpm;oh yes, the directives are processed correctly for the binary rpm
      • mock relies on bootstrapping a system using fedoralegacy.org, which was recently discontinued
      • I have a redhat-9 chroot tarball which I would like to use and tell mock to use that without caring about upgrades (since my packages are not at all that dependent on the build system)
      • mock fails to bootstrap even for fedora-devel-i386-core with the message "Cannot find a valid baseurl for repo: extras"
      • the older versions fail to bootstrap because, apparently, it "Could not find useradd in chroot, maybe the install failed?"
    • plague (how's that for irony?) relies on mock, which is broken
    • mach seems to be deprecated
  2. I would like a simple repo creation and maintainance tool, something in the line of mini-dinstall, which should have apt4rpm and yum support, but it seems that repos are still the "new shiny thing" which has no proper tools
  3. I would like to be able to know from which version I am upgrading from the current package, not only that I am upgrading (in case you didn't knew, the only way to know that you are upgrading in a post/pre install/remove script, is the wonderful $1 parameter which indicates the number of packages installed after the script finishes or something along that line..
So my question is, is there any way to fix any of the issues above in a sane way, without me reimplementing apt-like functionality in rpm land?

Thursday 5 July 2007

required file `./config.rpath' not found

If you ever encounter this error:

eddy@twix:/tmp/wormux/wormux$ ./autogen.sh
[+] Clean files generated by autoconf, aclocal, automake and configure
[+] Run aclocal
[+] Run autoheader
[+] Run automake
configure.ac:25: required file `./config.rpath' not found

Try this simple fix:

eddy@twix:/tmp/wormux/wormux$ touch config.rpath
eddy@twix:/tmp/wormux/wormux$ ./autogen.sh
[+] Clean files generated by autoconf, aclocal, automake and configure
[+] Run aclocal
[+] Run autoheader
[+] Run automake
[+] Run autoconf

Now run ./configure


Now you can go on with your work. Credits go to Daniel P. Berrange: http://www.mail-archive.com/fedora-xen@redhat.com/msg01017.html


P.S.: Apparently this is an automake 1.10 issue and can be avoided by using automake 1.9

I've seen it from a Debian PoV

I have seen my life in the company of my GF, and is not bright, looking from my Debian work PoV.

Some early conclusions:
  1. do whatever you can to have two computers in the house, otherwise yours will be arrested
  2. using any spare moment will not gain your GF's simpathy
  3. if there is slacking, it should happen with your GF, you can't affort slacking on your own
  4. if you are, by nature, less needy for sleep, you should use those extra minutes you can get for free
  5. it would be sooooo cool to get her hooked into "FLOSS" ;-)