uClibc based toolchain using Gentoo for NSLU2

I was asked in the previous post why I didn't used the Debian armel port for my NSLU2.

My intention was to create a uclibc based system (and a uclibc based toolchain) for my NSLU2. This was not obvious in the post because building the uclibc based toolchain resulted in this error:

crossdev armv5-softfloat-linux-uclibceabi

[..]/var/tmp/portage/cross-armv5-softfloat-linux-uclibceabi/gcc-4.9.2/work/gcc-4.9.2/libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cc:67:21: fatal error: wordexp.h: No such file or directory
 #include
                     ^
compilation terminated.
Makefile:416: recipe for target 'sanitizer_platform_limits_posix.lo' failed
make[4]: *** [sanitizer_platform_limits_posix.lo] Error 1
make[4]: *** Waiting for unfinished jobs....

So, yesterday, to not forget what I did, I tried building a glibc based toolchain and posted that.

Today, after looking at the offending error, it seems gcc 4.9.2 assumes wordexp.h is always available on non-Android platforms and Gentoo does not make that file availble when installing uclibc.


I think the problem was introduced in gcc in 2013 with this commit, but I haven't cheked in detail. What I know for sure is that gcc 4.8.3 works at this moment with uclibc and the buildroot guys are still using gcc 4.8.4 in their default uClibc based toolchain. So here it is the command that generated the uClibc based toolchain:

crossdev --g 4.8.3 armv5-softfloat-linux-uclibceabi

I hope this helps other people. Yes, I know I should report the issue to GCC/Gentoo after further investigation.

Using Gentoo to create a cross toolchain for the old NSLU2 systems (armv5te)

This is mostly written so I don't forget how to create a custom (Arm) toolchain the Gentoo way (in a Gentoo chroot).

I have been a Debian user since 2001, and I like it a lot. Yet I have had my share of problems with it, mostly because due to lack of time I have very little disposition to try to track unstable or testing, so I am forced to use stable.

This led me to be a fan of Russ Albery's backport script and to create a lot of local backports of packages that are already in unstable or testing.

But this does not help when packages are simply missing from Debian or when something like creating an arm uclibc based system that should be kept up to date, from a security PoV.

I have experience with Buildroot and I must say I like it a lot for creating custom root filesystems and even toolchains. It allows a lot of flexibility that binary distros like Debian don't offer, it does its designated work, creating root filesystems. But buildroot is not appropriate for a system that should be kept up to date, because it lacks a mechanism by which to be able to update to new versions of packages without recompiling the entire rootfs.

So I was hearing from the guys from the Linux Action Show (and Linux Unplugged - by the way, Jupiter Broadcast, why do I need scripts enabled from several sites just to see the links for the shows?) how Arch is great and all, that is a binary rolling release, and that you can customize packages by building your own packages from source using makepkg. I tried it, but Arm support is provided for some specific (modern) devices, my venerable Linksys NSLU2's (I have 2 of them) not being among them.

So I tried Arch in a chroot, then dropped it in favour of a Gentoo chroot since I was under the feeling running Arch from a chroot wasn't such a great idea and I don't want to install Arch on my SSD.

I used succesfully Gentoo in the past to create an arm-unknown-linux-gnueabi chroot back in 2008 and I always liked the idea of USE flags from Gentoo, so I knew I could do this.


So here it goes:


# create a local portage overlay - necessary for cross tools
export LP=/usr/local/portage
mkdir -p $LP/{metadata,profiles}
echo 'mycross' > $LP/profiles/repo_name
echo 'masters = gentoo' > $LP/metadata/layout.conf
chown -R portage:portage $LP
echo 'PORTDIR_OVERLAY="'$LP' ${PORTDIR_OVERLAY}"' >> /etc/portage/make.conf
unset LP

# install crossdev, setup for the desired target, build toolchain
emerge crossdev
crossdev --init-target -t arm-softfloat-linux-gnueabi -oO /usr/local/portage/mycross
crossdev -t arm-softfloat-linux-gnueabi

 

more news from the nslu2 front

The main reason for the lack of activity during this week was that I had some problems with my router, a NSLU2 running the debian arm port. This was the machine I wanted the softfloat rrdtool I was talking about in these 2 posts.

It turns out that for some weird reason the USB controller of the slug resets when there is too much traffic going on. I got lots and lots of messages like:


$ grep reset -A 1 /var/log/syslog | grep -E '(reset|repeated)'
Aug 3 07:02:27 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
...
Aug 3 11:50:29 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 11:51:03 ritter last message repeated 4 times
Aug 3 11:53:15 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 11:53:46 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 11:55:48 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 11:56:07 ritter last message repeated 2 times
Aug 3 11:56:52 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 11:56:57 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2


That wouldn't have been a problem as long as the damn thing worked, but in some cases after the reset the root filesystem was lost and all the services running on the machine died (except the networking itself since all of that is part of the kernel and is loaded in memory).

It took me about three evenings to get to this conclusion (imagine being locked out of a machine which advertised running ssh, http, and dns services, being able to telnet and getting a respons, but the full protocol failed).

It all became more clear when, while being connected via ssh I got his result:

eddy@ritter ~ $ ls
-bash: ls: command not found

"WHAT?" was the first reaction, then I figured that life without ls is unbearable and came up with this bit:

alias ls='for I in * ; do echo $I ; done'

Quite cool to understand a little bit about the system. Still, cat was unavailable so having /proc and /sys available did not help that much. At some point I was thinking about a busybox shell, tried that, it didn't work since it wanted to remove some important bits like the initramfs tools.


I googled again and found a good starting point for a discussion of the same problem someone else was having. I ended up trying all the proposed solutions: rmmod-ing ehci-hcd, blacklisting the module, regenerating a initrd image with the blacklisted module... None of these worked (individually) and I ended up having an even more unstable system when ehci was not present (I was loosing the root FS after aprox. 1 min after logging in remotely immediately after start). Restoring the image with ehci was a pain (not to mention the time spent trying to figure a way to revert the change safely, since bricking the machine was not out of the question - imagine loosing the FS while flashing the ram drive image).

I finally managed to revert the ehci enabled image and tried the workaround proposed in this gentoo BR. I tried 128 and now I am at 64 and I am still getting those resets; Still something good came out of this, I followed Michael Prokop's Use root=UUID on NSLU2 article and I think I have now a stable root file system just thanks to him.

Now I am stuck:


grep -E '(max_sect|reset)' /var/log/syslog
Aug 3 07:02:27 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 08:27:38 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 08:38:43 ritter manual message: max_sectors was set to 128
Aug 3 08:55:25 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 09:31:47 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 09:54:11 ritter set_max_sectors: max_sectors was set to 128
Aug 3 10:09:32 ritter set_max_sectors: max_sectors was set to 128
Aug 3 10:15:27 ritter set_max_sectors: sda's max_sectors was set to 128
Aug 3 10:24:24 ritter set_max_sectors: sda's max_sectors was set to 128
Aug 3 10:38:12 ritter set_max_sectors: sda's max_sectors was set to 128
Aug 3 10:47:04 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 10:48:27 ritter set_max_sectors: sda's max_sectors was set to 64
Aug 3 11:04:36 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
Aug 3 11:06:16 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2
....
Aug 3 15:05:58 ritter kernel: usb 3-1: reset high speed USB device using ehci_hcd and address 2

Should I try setting to 32?

In case you're wondering, I am doing the setting with this script:

#!/bin/sh

MAX_SECTORS=64
DISK=$(ls -l `grep 'uuid.* / ' /etc/fstab | awk '{print $1;}'` | sed 's#.*/\(.*\)$#\1#' | tr -d '[0-9]')

[ -z "$DISK" ] && logger -t 'set_max_sectors' "error: could not detect root disk" && exit

echo "$MAX_SECTORS" > /sys/block/$DISK/device/max_sectors
logger -t 'set_max_sectors' "${DISK}'s max_sectors was set to ${MAX_SECTORS}"

exit 0


Dear (not at all) lazy web, what can I do to fix/workaround this issue? Any help would be gratly appreciated (personal mail or comments).

Note: I am reluctant to try the fix proposed in the last comment that claims to fix the issue since compiling an arm kernel natively is a pain and I would like to have an official kernel from Debian Etch, as I do now.

softfloat rrdtool sequel

Hey, I am back from vacation. I was back on Saturday but I didn't had the time to blog or do Debian work, so hello :-D.

After coming back from my vacation I went back to the softfloat rrdtool thingie I was working on before and I finaly managed to compile a statically linked softfloat uclibc rrdtool. Needless to say that I found that a pain and I ended up:

• avoiding the Makefile and made the linking manually because:
• for some unknown reason, the -static option didn't ended up in the final link call
• libfreetype and libart_lgpl always were added as .so files... - libtool is probably the usual susupect, but I suspect the upstream libraries themselves
• avoided (completely) using xmerge and used directly the source since:
• CFLAGS="-Wl,-static" did not do the right thing (it didn't do in raw source, either)
• I was about to make a static rrdtool, so I didn't needed gentoo's ebuilds at that point
  

If you are curious about the hideous details, just add a comment or email me and I'll send them or publish them.

So, in the end, I managed to produce a staticlly linked arm-softfloat-uclibc rrdtool which:

• did not work directly on the .rrd files generated by the debian packaged collectd since they were "generated on another achitecture"; well with debian's rrdtool, 'rrdtool dump' was not that slow, so it was acceptable to dump with debian's rrdtool and restore with the statically linked one
• generated graphs in about 1 minute (way slower than my amd64 machine, which does in 3s, but at least 100+ times faster than debian's arm rrdtool - remember, I tested the generation and it didn't finish the first graph not even after 50 minutes)

So that was fun. The total round trip for a round of graph generation was about 15 min which was more than cool and fast compared to the slow native variant, whohoo!

Now the downside, I am not through with it. If you look at the picture you'll notice that there is a "small-ish" problem: there are no scales, legends and no letters/glyphs of any kind, so the graphs are still quite useless, but better than nothing:


They should look something like this:




Notice any difference? :-/

Of course, I won't stop here!

I suspect the problems come from the cross compiled libfreetype and libart_lgpl, and I'll probably end up doing some native compiling after all, sigh!

How to build rrdtool with soft-float for a debian arm machine (which uses hard float)

Being a good Debian user

Notes: the bulk of this article was written on the 17th; updates were added in later; this article is really long, you might want to read it in more sessions

Debian's arm port uses hardfloat. This means that apps like rrdtool (graph function) take ages to run because there is no floating point unit. In order to make this faster you'll want to use soft-float emulation. But this support needs to be present in all depended upon libs, including libc, which is an ABI incompatible change which would break all binaries. I can't use the yet-in-development-and-unofficial armel port since the machine has to be up and running almost 24/7 and I can't afford downtime.

So what to do?

Well, obviously use another soft-float enable libc, like uclibc, in parallel withe the hrad-float enabled libc. Debian Sid contains uclibc development libs, but building the whole toolchain is a pain. There is no real support in the distro itself and building on the arm machine is kind of out of the question. I tried a little in a debootstrapped chroot on the native machine, but this was a pain and I was stuck tring to figure out how to disable whitles and bells for rrdtool since I didn't needed any of the perl, tcl or python support. So I stopped, went to bed and decided I'll continue the next day (the day I wrote the biggest part of this article). The future looked dark, trying to build a softfloat uclibc rrdtool on a system that wasn't prepared for that and from which softfloat was pulled away somewhere during etch's development stage....

Deciding to go with the competition - native building

From link to link I found out that gentoo has arm-softfloat-linux-uclibc port, among others, so I decided to use that on the target machine (a NSLU2) in a chroot. Well I stumbled on the low computing power, when I tryed to emerge sync the tree. I gave up un that and tried to use emerge-websync. It looked better, but I didn't got too far with this either since the machine ended up being overloaded, with figures varying from 3 to 8 for the load average (all of them).

Deciding to go with the competition - cross building

It was time to bring in the big guns and I decided I was going to crossbuild those binaries, probably statically since it meant lower risk for things to break on the target machine.


Since I observed that gentoo has a nice tool called crossdev which actually uses emerge (the equivalent of dpkg+apt in Debian world) to make the cross chain tool and saw that they have some really good documentation and a development version of an arm-softfloat-linux-uclibc, I decided I could .

So I went on and downloaded the latest stage3 tarball for x86 (so I can have a gentoo system to start with), made all the necessary things and created a gentoo chroot. I emerge sync-ed, updated portage, installed crossdev (following the nice instructions on the gentoo embedded site about the subject). This took a while, but it went like a breeze. I made the necessary arangements to be able to cross build packages (setting SYSROOT, making that nice xmerge script).

As my goal was to make an arm-softfloat-linux-uclibc binary for rrdtool I tried directly to xmerge rrdtool (of course, first just pretended -p).

Finding bugs in the packages

First bug

All went fine until I found freetype which for some weird reason detected the i486-pc-linux-gnu-gcc compiler as i486-pc-linux-gnu (observe the missing trailing -gcc). I looked over the logs and determined that the CC_BUILD environment variable was not set. So exported CC_BUILD=i486-pc-linux-gnu-gcc and xmerged the library.

(While trying to fix this issue I entered #gentoo-embedded on freenode and actually met Yuri Vasilev, whom I had met in Edinburgh, at DebConf. Look for the video about Ligusk if you are interested about his project to make a Debian like distribution based on Gentoo. It was nice to meet a friendly "face" - as much you can say that online.)


I reported the bug (already there is a patch, I haven't tested, but looks ok) and the workaround in the gentoo bugzilla... after creating an account, since I never used that BTS before.

Second bug

Next pain was libart_lgpl whose upstream, for some comfortability reason chose to detect the target type sizes (and basically ignoring inttypes.h) with a small program which was built with the cross compiler which later was supposed to be ran on the host. I don't think is necessary to say that i failed, of course, since the binary was an ARM binary, while the host was an x86 machine. Yupee! not.

Ok, two bugs, no softfloat rrdtool yet. I reported this second bug and tried to work around the issue. Since I didn't really used too much a gentoo system, I wanted to know details and what possibilities I had. I got some really nice tips from "solar" (aka Ned Ludd) on #gentoo-embedded. He pointed out that while building a package one can press CTRL+z, do some stuff, like altering the Makefile.in file in the build directory and then fg back. Ok, cool.

All I need now is somebody with direct access to an arm machine that can compile and run for me the program in question. seanius on #debian-devel was kind enough to do this for me and, to my surprize, I got the same output header file as I got in the host x86 chroot. But that detail is not important, let's just compile that rrdtool. Well, first libart_lgpl...

Ok, I CTRL+Z-ed the build process just after the configure stage ended, cd-ed into the build directory, I tampered with the Makefile.in file and changed that line from

./gen_art_config > gen_art_config.h

into

cp /gen_art_config.h.arm gen_art_config.h

Haha! In your face! Ok, nasty did was done so I fg-ed and went on confident that I will get my hands on that rrdtool very soon. The package build ended succesfully, without any other suprizes, so I went for the graal!


xmerging rrdtool, the bugs don't stop

xmerge -v rrdtool

Bla, bla, bla.... bla, bla, bla, what do you know, there are a few IEEE math function tests in the configuration script of rrdtool. They failed. I don't know why, but I have a feeling that this should be happening on softfloat or even maybe the rrdtool source is not that cross compile aware. I didn't care that much, so I did what I knew best, tampering with an ongoing build.

This time is was a little bit harder to catch since I had to wait for the source to unpack (well, there is a larger frame, but that is the rough timing) and at that moment CTRL+Z the build.

I didn't had much time to waste, so after looking shortly over the tests I realized I should just ignore the error and look for the "failing command" that was interrupting my beloved build. Muhahah!

Thank God for verbose error messages, I could find the place fast and chaged the two "exit 1" statements into two harmless 'echo "Ignoring IEEE math errors"', sic. I had to do this a few times since I got the timping wrong, at some point I got the syntax wrong (at which point I remembered the cp trick I did earlier and saved the modified file in /configure.rrdtool )...

xmerging rrdtool and the 4th bug

After passing past this stage, I got a compilation error which, judging from the command, meant that the linker tried to link the target (arm) binary against the host (x86) libraries.

Oooook, this is not going to stop me! I look at the Makefile.in to try understand why this happened and then realized it wasn't worth it for me to try to fix it the right way. So I cheated once again! Muhaha, I just ran some useful commands which I'll let you feast your eyes to:

ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libart_lgpl_2.so /usr/lib/
ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libpng12.a /usr/lib/
ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libfreetype.so /usr/lib/

Ok, good to go. Of course, I had to stop again the build to copy the "fixed" configure script that ignored IEEE math errors, but that is already history.

And, believe it or not, actually the build succeded. I had a cross built rrdtool using shared libraries built against softfloat ucllibc. No guarantees about its correctness, but it built!

This is when I decided to start writing this article.

Still there are things to do

I will have to do a few things before being happy about the victory:

• check that all the resulted binaries are indeed ARM ELF files, and not Intel x86. It appears the shared libs are already.
• done, they are (checked the next day)
• figure out qpkg (is a utility from portage-utils which seems to be a way to make binary packages) to package somehow the result of this work or maybe just ignore it and use plain cp
• see how I can fit together both the softfloat uclibc binaries and the native debian binaries on the same filesystem, including ld.so configuration...
• but I fear this will be a nightmare, so I might decide to go with static compilation of rrdtool, which I don't think it will be quite straight forward given the experiences describe above; there could be also the size issue to take into account, and since the NSLU2 has only 32MB of RAM, this could be problematic
• I think I'll have to use a statically linked binary after all (note added late, the second day)
  
• see if the rrdtool binary really works and if all the work payed off, performance wise - it should, taking into account what I have heard from wookey during the debconf7 talk about the armel (warning, 73MB ogg file) port
• be happy with the solution I got or wait for the Debian armel port to catch up and switch to that asap
• if I'll have to do the switch I'll probably think of ways of migrating the whole system from arm to armel, and not to reinstall everything; this might prove useful for the whole debian armel port, when people are working on migration plans, since arm will be deprecated in favour of armel