Monday, 6 April 2015

HOWTO: Dnsmasq server for network booting using TFTP and DHCP

Dnsmasq is a very lightweight server that besides the expected DNS caching functionality, it also offers DHCP and TFTP functionality in a single binary.

This makes it very useful if one needs to network boot a system since you can have the TFTP and DHCP part of the setup done easily, and only add NFS for a complete network boot.

Add to that that

One extra nice thing dnsmasq has is that it can mark specific hosts, addresses or ranges with some internal markers, then use those markers as symbolic names to apply settings based for classes of devices.

In the configuration snippet below, there is a rule I set up to make sure I would apply the 'netbsd' label to any system connecting through specific ethernet interfaces (one is the interface of the system, the other is a USB NIC I use from time to time):
#You will need a range for static IPs in the main file

# give the name 'kinder' to any machine connecting through the given ethernet nics and apply 'netbsd' label
dhcp-host=00:1a:70:99:60:BB,00:06:4F:0D:B1:95, kinder,, set:netbsd

# Machines tagged 'netbsd' shall use the given NFS root path
dhcp-option=tag:netbsd, option:root-path,/export/netbsd-nslu2/root
# Enable dnsmasq's built-in TFTP server

# Set the root directory for files available via FTP.
Saving this configuration file in /etc/dnsmasq.d/kinder-netboot will enable this to be used by dnsmasq if this line is present in /etc/dnsmasq.conf
Commenting it will disable the netbsd part easily.

Sunday, 29 March 2015

HOWTO: Disassemble a big endian Arm raw memory dump with objdump

This is trivial and very useful for embedded code dumps, but in case somebody (including future me) needs this, here it goes:
arm-none-eabi-objdump -D -b binary -m arm -EB dump.bin | less
The options mean:
  • -D - disassemble
  • -b binary - input file is a raw file
  • -m arm - arm architecture
  • -EB - big endian
By default, endianness is assumed to be little endian, or at least that's happened with my toolchain.

Wednesday, 11 March 2015

Friday, 6 March 2015

Occasional Rsnapshot v1.3.1

I was writing in the previous post about Occasional Rsnapshot and  how I ended up writing it.

Just before releasing v1.2.1 I realized it would make sense Semantic Versioning which, in just a few words means:
Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards-compatible manner, and
PATCH version when you make backwards-compatible bug fixes.
I just released Occasional Rsnapshot v1.3.1 which mainly fixed issue 4:
When deciding to do a backup for interval INT, there should be a check that the oldest snapshot in INT-1 interval is older than the threshold for the INT interval. Otherwise the INT interval will be populated with backups more frequent than desired and it is possible older backups in INT interval to completely lost.
The condition should be:
ts(oldest(INT-1)) - ts(newest(INT)) >= threshold(INT)
For example:
  • if weekly.0 is from 15th of February and daily.6 is from 17th of February, weekly should not be triggered, but
  • if weekly.0 is from 15th of February and daily.6 is from 23rd of February, weekly shall be triggered
This extra check should probably added in can_backup_interval.
It was a small bug, but it might have lead to losing important older backups because newer and more frequent backups would be pushed from hourly, interval by interval up to the yearly interval, in spite of the fact that the distance between backups wouldn't have been respecting the upper interval minimum distance in time.

There was also a small syntax bugfix, but functionally nothing has changed because bash was doing the right thing even with that small error.

If you have started using Occasional Rsnapshot, you definitely want now Occasional  Rsnapshot v1.3.1. If you haven't started and you don't have backups, please start doing backups, and while you're at it, you might want to try Occasional Rsnapshot (with Rsnapshot).

Wednesday, 25 February 2015

Occasional Rsnapshot v1.3.0

It is almost exactly 1 year and a half since I came up with the idea of having a way of making backups using Rsnapshot automatically triggered by my laptop when I have the backup media connected to my laptop. This could mean connecting a USB drive directly to the laptop or mounting a NFS/sshfs share in my home network. Today I tagged Occasional Rsnapshot the v1.3.0 version, the first released version that makes sure even when you connect your backup media occasionally, your Rsnapshot backups are done if and when it makes sense to do it, according to the rsnapshot.conf file and the status of the existing backups on the backup media.

Quoting from the README, here is what Occasional Rsnapshot does:

This is a tool that allows automatic backups using rsnapshot when the external backup drive or remote backup media is connected.

Although the ideal setup would be to have periodic backups on a system that is always online, this is not always possible. But when the connection is done, the backup should start fairly quickly and should respect the daily/weekly/... schedules of rsnapshot so that it accurately represents history.

In other words, if you backup to an external drive or to some network/internet connected storage that you don't expect to have always connected (which is is case with laptops) you can use occasional_rsnapshot to make sure your data is backed up when the backup storage is connected.

occasional_rsnapshot is appropriate for:
  • laptops backing up on:
    • a NAS on the home LAN or
    • a remote or an internet hosted storage location
  • systems making backups online (storage mounted locally somehow)
  • systems doing backups on an external drive that is not always connected to the system
The only caveat is that all of these must be mounted in the local file system tree somehow by any arbitrary tool, occasional_rsnapshot or rsnapshot do not care, as long as the files are mounted.

So if you find yourself in a simillar situation, this script might help you to easily do backups in spite of the occasional availability of the backup media, instead of no backups at all. You can even trigger backups semi-automatically when you remember to or decide is time to backup, by simply pulging in your USB backup HDD.

But how did I end up here, you might ask?

In December 2012 I was asking about suggestions for backup solutions that would work for my very modest setup with Linux and Windows so I can backup my and my wife's system without worrying about loss of data.

One month later I was explaining my concept of a backup solution that would not trust the backup server, and leave to the clients as much as possible the decision to start the backup at their desired time. I was also pondering on the problems I might encounter.

From a security PoV, what I wanted was that:
  1. clients would be isolated from each other
  2. even in the case of a server compromise:
    • the data would not be accessible since they would be already encrypted before leaving the client
    • the clients could not be compromised

The general concept was sane and supplemental security measures such as port knocking and initiation of backups only during specific time frames could be added.

The problem I ran to was that when I set up this in my home network a sigle backup cycle would take more than a day, due to the fact that I wanted to do backup of all of my data and my server was a humble Linksys NSLU2 with a 3TB storage attached on USB.

Even when the initial copy was done by attaching the USB media directly to the laptop, so the backup would only copy changed data, the backup with the HDD attached to the NSLU2 was not finished even after more than 6 hours.

The bottleneck was the CPU speed and the USB speed. I tried even mounting the storage media over sshfs so the tiny xscale processor in the NSLU2 would not be bothered by any of the rsync computation. This proved to an exercise in futility, any attempt to put the NSLU2 anywhere in the loop resulted in an unacceptable and impractically long backup time.

All these attempts, of course, took time, but that meant that I was aware I still didn't have appropriate backups and I wasn't getting closer to the desired result.

So this brings us August 2013, when I realized I was trying to manually trigger Rsnapshot backups from time to time, but having to do all sorts of mental gymnastics and manual listing to evaluate if I needed to do monthly, weekly and daily backups or if weekly and daily was due.

This had to stop.
Triggering a backup should happen automatically as soon as the backup media is available, without any intervention from the user.
I said.

Then I came up with the basic concept for Occasional Rsnapshot: a very silent script that would be called from  cron every 5 minutes, would check if the backup media is mounted, if is not, exit silently to not generate all sorts of noise in cron emails, but if mounted, compute which backup intervals should be triggered, and trigger them, if the appropriate amount of time passed since the most recent backup in that backup interval.

Occasional Rsnapshot version v1.3.0 is the 8th and most recent release of the script. Even if I used Occasional Rsnapshot since the day 1, v1.3.0 is the first one I can recommend to others, without fearing they might lose data due to it.

The backup media can be anything, starting from your regular USB mounted HDD, your sshfs mounted backup partition on the home NAS server to even a remote storage such as Amazon S3 online storage, and there are even brief instructions on how to do encrypted backups for the cases where you don't trust the remote storage.

So if you think you might find anything that I described remotely interesting, I recommend downloading the latest release of Occasional Rsnapshot, go through the README and trying it out.

Feedback and bug reports are welcome.
Patches are welcomed with a 'thank you'.
Pull requests are eagerly waited for :) .

Monday, 9 February 2015

uClibc based toolchain using Gentoo for NSLU2

I was asked in the previous post why I didn't used the Debian armel port for my NSLU2.

My intention was to create a uclibc based system (and a uclibc based toolchain) for my NSLU2. This was not obvious in the post because building the uclibc based toolchain resulted in this error:
crossdev armv5-softfloat-linux-uclibceabi
[..]/var/tmp/portage/cross-armv5-softfloat-linux-uclibceabi/gcc-4.9.2/work/gcc-4.9.2/libsanitizer/sanitizer_common/ fatal error: wordexp.h: No such file or directory

compilation terminated.
Makefile:416: recipe for target 'sanitizer_platform_limits_posix.lo' failed
make[4]: *** [sanitizer_platform_limits_posix.lo] Error 1
make[4]: *** Waiting for unfinished jobs....

So, yesterday, to not forget what I did, I tried building a glibc based toolchain and posted that.

Today, after looking at the offending error, it seems gcc 4.9.2 assumes wordexp.h is always available on non-Android platforms and Gentoo does not make that file availble when installing uclibc.

I think the problem was introduced in gcc in 2013 with this commit, but I haven't cheked in detail. What I know for sure is that gcc 4.8.3 works at this moment with uclibc and the buildroot guys are still using gcc 4.8.4 in their default uClibc based toolchain. So here it is the command that generated the uClibc based toolchain:

crossdev --g 4.8.3 armv5-softfloat-linux-uclibceabi

I hope this helps other people. Yes, I know I should report the issue to GCC/Gentoo after further investigation.

Saturday, 7 February 2015

Using Gentoo to create a cross toolchain for the old NSLU2 systems (armv5te)

This is mostly written so I don't forget how to create a custom (Arm) toolchain the Gentoo way (in a Gentoo chroot).

I have been a Debian user since 2001, and I like it a lot. Yet I have had my share of problems with it, mostly because due to lack of time I have very little disposition to try to track unstable or testing, so I am forced to use stable.

This led me to be a fan of Russ Albery's backport script and to create a lot of local backports of packages that are already in unstable or testing.

But this does not help when packages are simply missing from Debian or when something like creating an arm uclibc based system that should be kept up to date, from a security PoV.

I have experience with Buildroot and I must say I like it a lot for creating custom root filesystems and even toolchains. It allows a lot of flexibility that binary distros like Debian don't offer, it does its designated work, creating root filesystems. But buildroot is not appropriate for a system that should be kept up to date, because it lacks a mechanism by which to be able to update to new versions of packages without recompiling the entire rootfs.

So I was hearing from the guys from the Linux Action Show (and Linux Unplugged - by the way, Jupiter Broadcast, why do I need scripts enabled from several sites just to see the links for the shows?) how Arch is great and all, that is a binary rolling release, and that you can customize packages by building your own packages from source using makepkg. I tried it, but Arm support is provided for some specific (modern) devices, my venerable Linksys NSLU2's (I have 2 of them) not being among them.

So I tried Arch in a chroot, then dropped it in favour of a Gentoo chroot since I was under the feeling running Arch from a chroot wasn't such a great idea and I don't want to install Arch on my SSD.

I used succesfully Gentoo in the past to create an arm-unknown-linux-gnueabi chroot back in 2008 and I always liked the idea of USE flags from Gentoo, so I knew I could do this.

So here it goes:

# create a local portage overlay - necessary for cross tools
export LP=/usr/local/portage
mkdir -p $LP/{metadata,profiles}
echo 'mycross' > $LP/profiles/repo_name
echo 'masters = gentoo' > $LP/metadata/layout.conf
chown -R portage:portage $LP
echo 'PORTDIR_OVERLAY="'$LP' ${PORTDIR_OVERLAY}"' >> /etc/portage/make.conf
unset LP

# install crossdev, setup for the desired target, build toolchain
emerge crossdev
crossdev --init-target -t arm-softfloat-linux-gnueabi -oO /usr/local/portage/mycross
crossdev -t arm-softfloat-linux-gnueabi


Monday, 28 October 2013

The stupidest trend in laptop design is...

... numpads on laptop keyboards.

Just because a very, very, very restricted segment of the population is into accounting or other jobs needing frequent numeric input, almost all laptop manufacturers feel the stupid urge to place a numpad on laptops with screens over 14''.

Reasons why a numpad on a laptop is a bad idea:
  • can lead to health issues: it forces the user to assume a bad position in front of the screen; this can affect the eyes and the backbone; It's bad enough many people have a bad posture in front of the computer as it is, no need to pump up Scoliosis' position in the laundry list of modern day health risks
The numpad forces eye focus and users' hands to be way off center.
The touchpad looks as if it was thrown to the side by accident
Note how the designer tried to offset the misalignment forced
by the numpad by "positioning" the touchpad closer to center,
but not in line with the space bar (and the keyboard)

  • ugly design: it simply looks ugly; why do people think Apple didn't jump into this stupid bandwagon? Because it's ugly design!
Without numpad the position is almost perfect!
The symmetry of the laptop improves on the design.
Notice how the touchpad is also centered.

  • the numpad is useless for the vast majority of people, and those who need numpads, already use them at desktop (keyboard) or, can buy numpads
  • it's more expensive to manufacture (more moving parts, more complex wiring and more expensive materials - plastic is cheaper than plastic+copper+rubber)
  • less resilient: more ways to fail, higher risk to get liquids inside
  • bad mechanical design: wider keyboard means less distance from the edge to the keyboard, which can mean a more fragile case
  • makes the laptop heavier: the plastic or material that covers the keys would be enough to cover the insides; the extra rubber, wiring, support etc, add extra weight
  • it kills the opportunity to use the real estate for other useful things (or things which improve the overall design): speakers, volume keys, finger print readers, other special function keys, LEDs or other output devices
What's worse is that most resellers do not have filters for this particular mis-feature, so you can't easily exclude laptops corrupted by this horrendous idea.

So, if you're a laptop designer, please stop putting numpads on laptops.

It will make for better looking laptops and you'll have the opportunity to be the one stating the obvious: numpads are generally useless! Even on desktop keyboards!

Monday, 2 September 2013

Integrating Beyond Compare with Semanticmerge

Note: This post will probably not be on the liking of those who think free software is always preferable to closed source software, so if you are such a person, please take this article as an invitation to implement better open source alternatives that can realistically compete with the closed source applications I am mentioning here. I am not going to mention here where the open source alternatives are not up to the same level as the commercial tools, I'll leave that for the readers or for another article.

Semanticmerge is a merge tool that attempts to do the right thing when it comes to merging source code. It is language aware and it currently supports Java and C#. Just today the creators of the software have started working on the support for C.

Recently they added Debian packages, so I installed it on my system. For open source development Codice Software, the creators of Semanticmerge, offers free licenses, so I decided to ask for one today, and, although is Sunday, I received an answer and I will get my license on Monday.

When a method is moved from one place to another and changed in a conflicting way in two parallel development lines, Semanticmerge can isolate the offending method and can pass all its incarnations (base, source and destination or, if you prefer, base, mine and theirs) to a text based merge tool to allow the developer to decide how to resolve the merge. On Linux, the Semanticmerge samples are using kdiff3 as the text-based merge tool, which is nice, but I don't use kdiff3, I use Meld, another open source visual tool for merges and comparisons.

OTOH, Beyond Compare is a merge and compare tool made by Scooter Software which provides a very good text based 3-way merge with a 3 sources + 1 result pane, and can compare both files and directories. Two of its killer features is having the ability split differences into important and non-important ones according to the syntax of the compared/merged files, and the ability to easily change or add to the syntax rules in a very user-friendly way. This allows to easily ignore changes in comments, but also basic refactoring such as variable renaming, or other trivial code-wide changes, which allows the developer to focus on the important changes/differences during merges or code reviews.

Syntax support for usual file formats like C, Java, shell, Perl etc. is built in (but can be modified, which is a good thing) and new file types with their syntaxes can be added via the GUI from scratch or based on existing rules.

I evaluated Beyond Compare at my workplace and we decided it would be a good investment to purchases licenses for it for the people in our department.

Having these two software separate is good, but having them integrated with each other would be even better. So I decided I would try to see how it can be done. I installed Beyond compare on my system, too and looked through the examples

The first thing I discovered is that the main assumption of Semanticmerge developers was that the application would be called via the SCM when merges are to be done, so passing lots of parameters would not be problem. I realised that when I saw how one of the samples' starting script invoked semantic merge:
semanticmergetool -s=$sm_dir/ -b=$sm_dir/ -d=$sm_dir/ -r=/tmp/ -edt="kdiff3 \"#sourcefile\" \"#destinationfile\"" -emt="kdiff3 \"#basefile\" \"#sourcefile\" \"#destinationfile\" --L1 \"#basesymbolic\" --L2 \"#sourcesymbolic\" --L3 \"#destinationsymbolic\" -o \"#output\"" -e2mt="kdiff3 \"#sourcefile\" \"#destinationfile\" -o \"#output\""
Can you see the problem? It seems Semanticmerge has no persistent knowledge of the user preferences with regards to the text-based merge tool and exports the issue to the SCM, at the price of overcomplicating the command line. I already mentioned this issue in my license request mail and added the issue and my fix suggestion in their voting system of features to be implemented.

The upside was that by comparing the command line for kdiff3 invocations, the kdiff3 documentation and, by comparison, the Beyond Compare SCM integration information, I could deduce what is the command line necessary for Semanticmerge to use Beyond Compare as an external merge and diff tool.

The -edt, -emt and -e2mt options are the ones which specify how the external diff tool, external 3-way merge tool and external 2-way merge tool is to be called. Once I understood that, I split the problem in its obvious parts, each invocation had to be mapped, from kdiff3 options to beyond compare options, adding the occasional bell and whistle, if possible.

The parts to figure out, ordered by compexity, were:

  1. -edt="kdiff3 \"#sourcefile\" \"#destinationfile\"

  2. -e2mt="kdiff3 \"#sourcefile\" \"#destinationfile\" -o \"#output\""

  3. -emt="kdiff3 \"#basefile\" \"#sourcefile\" \"#destinationfile\" --L1 \"#basesymbolic\" --L2 \"#sourcesymbolic\" --L3 \"#destinationsymbolic\" -o \"#output\""

Semantic merge integrates with kdiff3 in diff mode via the -edt option. This was easy to map to Beyond Compare, I just replaced kdiff3 with bcompare:
-edt="bcompare \"#sourcefile\" \"#destinationfile\""
Integration for 2-way merges was also quite easy, the mapping to Beyond Compare was:
-e2mt="bcompare \"#sourcefile\" \"#destinationfile\" -savetarget=\"#output\""
For the 3-way merge I was a little confused because the Beyond Compare documentation and options were inconsistent between Windows and Linux. On Windows, for some of the SCMs, the options that set the titles for the panes are '/title1', '/title2', '/title3' and '/title4' (way too descriptive for my taste /sarcasm), but for some others are '/lefttitle', '/centertitle', '/righttitle', '/outputtitle', while on Linux the options are the more explicit kind, but with a '-' instead of a '/'.

The basic things were easy, ordering the parameters in the 'source, destination, base, output' instead of kdiff3's 'base, source, destination, -o ouptut', so I wanted to add the bells and whistles, since it really makes more sense for the developer to see something like 'Destination: [method] readOptions' instead of '/tmp/tmp4327687242.tmp', and because that's exactly what is necessary for Semanticmerge when merging methods, since on conflicts the various versions of the functions are placed in temporary files which don't mean anything.

So, after some digging into the examples from Beyond Compare and kdiff3 documentation, I ended up with:
-emt="bcompare  \"#sourcefile\" \"#destinationfile\" \"#basefile\" \"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic' -centertitle='#basesymbolic' -outputtitle='merge result'"

Sadly, I wasn't able to identify the symbolic name for the output, so I added the hard coded 'merge result'. If Codice people would like to help with with this information (or if it exists), I would be more than willing to do update the information and do the necessary changes.

Then I added the bells and whistles for the -edt and -e2mt options, so I ended up with an even more complicated command line. The end result was this monstrosity:
semanticmergetool -s=$sm_dir/ -b=$sm_dir/ -d=$sm_dir/ -r=/tmp/ -edt="bcompare \"#sourcefile\" \"#destinationfile\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic'" -emt="bcompare  \"#sourcefile\" \"#destinationfile\" \"#basefile\" \"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic' -centertitle='#basesymbolic' -outputtitle='merge result'" -e2mt="bcompare \"#sourcefile\" \"#destinationfile\" -savetarget=\"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic'"
So when I 3-way merge a function I get something like this (sorry for high resolution, lower resolutions don't do justice to the tools):

I don't expect this post to remain relevant for too much time, because after sending my feedback to Codice, they were open to my suggestion to have persistent settings for the external tool integration, so, in the future, the command line could probably be as simple as:
semanticmergetool -s=$sm_dir/ -b=$sm_dir/ -d=$sm_dir/ -r=/tmp/
And the integration could be done via the GUI, while the command line can become a way to override the defaults.

Sunday, 1 September 2013

HOWTO add a shell script wrapper and preserve quoting for parameters

If you ever find yourself in the situation where you have to add a shell script wrapper over a command, but the parameters' quoting gets lost and you end up with the wrong parameters in the wrapped command/tool, you might want to read this post.

On my system I have some command line tools which are Windows only and, in order to easily use the same build system as on Windows on my Linux machine I added a wrapper script which invokes wine on the commands and made symlinks to the wrapper with the file names as the tools, but without the '.exe' suffix.

Of course, I wanted to properly pass the parameters through the wrapper to the tools so I wrote (note the bold text):
wine $0.exe "$@"
So the answer is: use $@ and quote like I did in the code above and the parameters will be passed correctly.

Update: stbuehler suggested to use exec to replace the shell process with wine with this construct:
exec wine $0.exe "$@"

Thanks for the suggestion.

Wednesday, 24 July 2013

HOWTO: git - change branch without touching working copy (at all)

Did you ever had the need in a git repository to change to another branch without altering AT ALL the working copy and ever wondered how that's done?

Usual use cases might be when you mde some changes to the working copy thinking you were on another branch, or you double-track in git a directory which is also tracked by another VCS (e.g. ClearCase).

What you need, in fact, is to update the index and not touch the working copy. The command that does that is

git read-tree otherbranch
If you also need to commit the state of your working tree to the otherbranch, you also need to tell git to actually associate the curent HEAD with the branch you just switched to:
git symbolic-ref HEAD refs/heads/otherbranch
I use this approach at my work place* to develop/experiment with possible code improvements on my machine before considering the merge into the official code.

* The preferred VCS is (Base) ClearCase, and I keep a git repository over the relevant part of the project in the ClerCase Dynamic View, so for synchronisations, the files in the working copy are updated by ClearCase and I have to resync my git branch (clearcaseint) following the latest official code from time to time, so I can pull in my local disk git repository the clearcaseint branch and merge it with my experimental changes in my git feature branches. 

If people are curious about how I work with ClearCase and git, I can expand on this in another post.

Saturday, 20 July 2013

(Not a) GNU Make quirk, or why logs should be provided

About two months ago I was writing about a quirk I found in GNU Make related to the $(patsubst ) function.

I have just tried this on my Debian Wheezy laptop which has make 3.81, but I wasn't able to reproduce the issue with the version from Debian (3.81-8.2).

The makefile looks like this:
PATH := ../some/prefixCPU12suf/include
CPUINC := $(patsubst ../some/prefix%,%,$(PATH))
CPU := $(patsubst %/include,%,$(CPUINC))

    @echo "PATH   = $(PATH)"
    @echo "CPUINC = $(CPUINC)"
    @echo "CPU    = $(CPU)"
And the result was correct:
0 eddy@heidi /tmp $ make
PATH   = ../some/prefixCPU12/include
CPUINC = CPU12/include
CPU    = CPU12
0 eddy@heidi /tmp $ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A

This program built for x86_64-pc-linux-gnu
The worst part is that I know I tested this issue on 3.82 on Cygwin and on Linux with the 3.82 version and it failed, but I wasn't able to remember how I did it. I started searching through the directory where I knew there could be the test makefile, I wasn't able to find it, until I remembered what I was trying to achieve.

From a path like ../some/prefixCPU12suf/include I wanted to use % to remove the parts 'some/prefix' and 'suf/include' because in the directory ../CPU12 there were some files that needed to be processed.

The actual issue is that GNU Make's '%' is not analogous to shell's '*', so that means code like this does not work as I assumed anf the 'pref' part is not an anchor:

PATH := ../some/prefCPU12suf/include
CPUINC := $(patsubst pref%,%,$(PATH))
CPU := $(patsubst %suf/include,%,$(CPUINC))

    @echo "PATH   = $(PATH)"
    @echo "CPUINC = $(CPUINC)"
    @echo "CPU    = $(CPU)"
Which leads to these results, no matter the version:

0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ ./make -f /tmp/makefile
PATH   = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU    = ../some/prefCPU12
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ ./make --version
GNU Make 3.82
Built for x86_64-unknown-linux-gnu
Copyright (C) 2010  Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make -f /tmp/makefile
PATH   = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU    = ../some/prefCPU12
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A

This program built for x86_64-pc-linux-gnu
Not sure if this could be qualified as a true bug, or a if the way I expected is a nice to have feature, but, in any case, the behaviour is consistent, unlike my brain which failed to initially identify the inconsistency in my code:

0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ grep patsubst /tmp/makefile
CPUINC := $(patsubst pref%,%,$(PATH))
CPU := $(patsubst %suf,%,$(CPUINC))
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make -f /tmp/makefile
PATH   = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU    = ../some/prefCPU12suf/include
Note that this behaviour of patsubst is asymtric to how subst works, as explained in the updated old post.

This took some extra effort to remember what was the actual issue, and shows why logs are important when reporting an issue, and why reporting issues as soon as they were encountered: because human brains are faulty. (Yes, yours, too!)

Thursday, 18 July 2013

HOWTO: git - remove those pesky remotes/origin/

Often people create branches to fix various bugs or implement various features, and publish those to a public repository to be pulled in the official repository (e.g. pull requests in the github model).

The problem is that after the fix is merged the local repo still has the branch that tracks the origin (your public repository).

To remove the local branch it is simple:

git branch -d feature_x

To remove the actual branch in the remote repository:

git push origin --delete feature_x

But, still the local tracking branches remain, those pesky 'remotes/origin/feature_x'. Here is an example from one of my repos:

$ git branch -a
* py.test
  remotes/origin/HEAD -> origin/master

All of those can be removed with this command:

git remote prune origin

 And the output for my appdirs repository is:

Pruning origin
 * [pruned] origin/appauthor-fallback
 * [pruned] origin/linux-fixes
 * [pruned] origin/next
 * [pruned] origin/root-directories
 * [pruned] origin/sysplatformtest
 * [pruned] origin/wrapper-defaults
Which, finally removes those damned refs:

$ git branch -a
* master
  remotes/origin/HEAD -> origin/master

Update 2013-07-24:  There have been also other solutions proposed in the comments section. I am quoting them here for better visibility.

Besides the solution I proposed, people commented about other alternatives for pruning remote branches, by using git fetch --prune or by using git-push's --prune option.

Brice provided a method to remove specific branches one by one with git branch -dr command:
git branch -dr origin/foo
What is ironic is that in the past I have used this format, and before writing the original article I was looking for this, but haven't found it that easily. I hope this article will help some other poor soul or myself in the future.