Sunday, 9 July 2017

Belgian eID Packages on Debian Testing: Updates Ignored

The Issue

As I documented in an earlier post, I installed the Belgian eID software under Debian Testing, using the candidate/sid section of the eID package repository.

My sources.list entries pertaining to this software archive look something like the following:

deb       http://files.eid.belgium.be/debian   candidate/sid   main
deb-src   http://files.eid.belgium.be/debian   candidate/sid   main

Initially, when I installed the Belgian eID software, the archive provided version 4.2.5v4.2.5.88.g4798c05-0sid1 of these packages—which were successfully installed:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                       Version            Architecture       Description
+++-==========================-==================-==================-=========================================================
ii  beid-mozilla-extension     4.2.5v4.2.5.88.g47 all                Belgian Electronic Identity Card - Mozilla plugin
ii  eid-mw                     4.2.5v4.2.5.88.g47 amd64              low-level support for Belgian Electronic Identity Card
ii  eid-viewer                 4.2.5v4.2.5.88.g47 amd64              Graphical interface to Belgian Electronic Identity Card
ii  libbeidpkcs11-0:amd64      4.2.5v4.2.5.88.g47 amd64              PKCS#11 library for Belgian Electronic Identity Card
ii  libbeidpkcs11-bin          4.2.5v4.2.5.88.g47 amd64              helper programs for libbeidpkcs11-0
ii  libeidviewer0:amd64        4.2.5v4.2.5.88.g47 amd64              eid-viewer library

Then, one day, upgrades to the packages became available from the candidate/sid section of the archive. The Synaptic Package Manager listed the following version information for any of these packages:

Available versions:
4.2.5v4.2.5.103.ge5c900e-0sid1 (candidate)
4.2.5v4.2.5.88.g4798c05-0sid1 (now)

I, therefore, expected the packages to get upgraded to version 4.2.5v4.2.5.103.ge5c900e-0sid1 when I installed system updates—but instead, nothing happened:

apt-get update
Hit:1 http://ftp.be.debian.org/debian testing InRelease
Hit:2 http://files.eid.belgium.be/debian candidate/sid InRelease
Hit:3 http://security.debian.org testing/updates InRelease
Hit:4 http://ftp.be.debian.org/debian testing-updates InRelease
Reading package lists... Done
apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

This behaviour made me wonder how the APT system actually decides which version to select when it is instructed to install or upgrade a given package from the configured software repositories.

The APT Policy and Package Priority Selections

To get an idea about the available versions of some software package, and about how the default APT policy will select the one to install, you can run the apt-cache policy command on said package—e.g.:

apt-cache policy beid-mozilla-extension
beid-mozilla-extension:
  Installed: 4.2.5v4.2.5.88.g4798c05-0sid1
  Candidate: 4.2.5v4.2.5.88.g4798c05-0sid1
  Version table:
     4.2.5v4.2.5.103.ge5c900e-0sid1 1
          1 http://files.eid.belgium.be/debian candidate/sid/main amd64 Packages
 *** 4.2.5v4.2.5.88.g4798c05-0sid1 100
        100 /var/lib/dpkg/status

The output from the command shows that the ‘.88.’ version of the package is currently installed, and that it will also be selected as the candidate for installation—in other words, the package will not be upgraded.

Even so, the version table indicates that version ‘.103.’ of the package is available from the candidate/sid section of the Belgian eID software repository. Even though this is a higher version than the one that is already installed, it will not be selected to upgrade the package.

Indeed, the ‘.103.’ version is assigned a ‘Priority’ of 1—corresponding to the default Priority of the repository that provides it. The installed version, on the other hand, by definition has a Priority of 100. Because the higher version has a lower Priority than the installed one, it will not get automatically selected to upgrade the package.

If you run the apt-cache policy command without further arguments, then it will list the package archives that are configured on the system, together with their Priority levels:

apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
   1 http://files.eid.belgium.be/debian candidate/sid/main amd64 Packages
     release o=FedICT,a=candidate,n=candidate,l=FedICT,c=main,b=amd64
     origin files.eid.belgium.be
 500 http://security.debian.org testing/updates/non-free amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian-Security,c=non-free,b=amd64
     origin security.debian.org
 500 http://security.debian.org testing/updates/main amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian-Security,c=main,b=amd64
     origin security.debian.org
 500 http://ftp.be.debian.org/debian testing/non-free amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=non-free,b=amd64
     origin ftp.be.debian.org
 500 http://ftp.be.debian.org/debian testing/contrib amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=contrib,b=amd64
     origin ftp.be.debian.org
 500 http://ftp.be.debian.org/debian testing/main amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=main,b=amd64
     origin ftp.be.debian.org
Pinned packages:

This output demonstrates that ‘regular’ software archives are assigned a priority level of 500, and confirms that the priority level of the candidate/sid section of the Belgian eID repository is 1. The currently installed package versions—which are recorded in the ‘/var/lib/dpkg/status’ file—get a priority level of 100.

The Priority levels assigned to the software archives do not come out of thin air, but they are determined according to well-defined rules. Additionally, an extensive mechanism is supported to override the defaults for any given archive—or even for individual packages. Even without such overrides, a specific version of a package can forcibly be selected for installation.

Selecting a Specific Version of a Package for Installation

By default, if you install a package, the APT system will not necessarily select the highest version of the package, but it will take the assigned Priority levels into account as well—as demonstrated above.

If you want to force selection of a specific version of a package, then you can run the apt install command and follow the package name with an equals sign (i.e., “=”) and the version string to select. For instance, the following command will install version 4.2.5v4.2.5.103.ge5c900e-0sid1 of each of the Belgian eID software packages:

apt install   \
   beid-mozilla-extension=4.2.5v4.2.5.103.ge5c900e-0sid1   \
   eid-mw=4.2.5v4.2.5.103.ge5c900e-0sid1                   \
   eid-viewer=4.2.5v4.2.5.103.ge5c900e-0sid1               \
   libbeidpkcs11-0=4.2.5v4.2.5.103.ge5c900e-0sid1          \
   libbeidpkcs11-bin=4.2.5v4.2.5.103.ge5c900e-0sid1        \
   libeidviewer0=4.2.5v4.2.5.103.ge5c900e-0sid1

Alternatively, instead of requesting a specific version string, you could follow the package name with a forward slash (i.e., “/”) and either the codename or suite name for the archive from which you want to select the package version—e.g.:

apt install   \
   beid-mozilla-extension/candidate   \
   eid-mw/candidate                   \
   eid-viewer/candidate               \
   libbeidpkcs11-0/candidate          \
   libbeidpkcs11-bin/candidate        \
   libeidviewer0/candidate

Also, given that all of these packages belong to the same archive, you may prefer to specify the target release only once, on the “-t” (or “--target-release” or “--default-release”) option:

apt --target-release candidate install        \
   beid-mozilla-extension eid-mw eid-viewer   \
   libbeidpkcs11-0 libbeidpkcs11-bin          \
   libeidviewer0

Priority Levels of Software Archives

When you use the APT system to install a software package, multiple versions of the package may be available from the various software sources that are known to the system—e.g.:

  • One version of the package may already be installed on your system—in which case installation of a newer version is commonly termed “upgrading” the package;
  • Another version may be provided by the official Debian repositories;
  • Yet another version may be present in a third-party repository that you set up for your system to use.

To select the package version that will get installed, the system will assign a Priority to each available instance of the package. By default, this priority value will depend entirely on the software source that contains the package, according to the following rules:

Priority=100
The package version that is already installed (if any).
Priority=500
Package versions that come from a ‘default’ software archive—i.e., one to which none of the other priority values are applied.
Priority=990
Package versions that belong to the target release, if specified.
Priority=1
Package versions from software archives that are marked “NotAutomatic: yes” in their ‘Release’ files.
Priority=100
Package versions from software archives that are marked “NotAutomatic: yes” as well as “ButAutomaticUpgrades: yes” in their ‘Release’ files.

If the same version of the package is found in more than one software source, then that version will be assigned the highest of the priorities that apply to it. For example, the installed version of a package will initially be assigned a Priority of 100. If, however, the same version of the package is present in a ‘default’ software archive, then its Priority will be raised to 500 instead.

Once each package version is assigned the appropriate priority, the APT system will apply the following rules, in order to determine which version of the package must be installed:

  • Ignore the package versions that are older than the installed version. In other words, by default, packages will never be ‘downgraded’.
  • Install the version with the highest priority.
  • If there are two or more instances with the highest priority, then install the one with the highest version number.
  • If there are two or more instances with the highest priority and the highest version number, then:
    • If the package is not yet installed, then install one of the selected instances;
    • Otherwise, if the “--reinstall” option is in effect, then replace the installed one with one of the selected uninstalled ones;
    • Otherwise, if all of the instances are equal in their metadata, then leave the installed one untouched;
    • Otherwise, replace the installed instance with one of the uninstalled ones that differ from the installed one in their metadata.
    Should there exist two or more possible candidates for installation, then the ‘first’ instance encountered in the software archives will be selected—where the archives are ordered according to the order in which the contents of the ‘sources.list’ fragment files are processed.

Reviewing the Belgian eID Software Priority Level

Taking into account the above explanation, you should now be able to better understand the output from the apt-cache policy command as it applies to one of the Belgian eID software packages—e.g.:

apt-cache policy beid-mozilla-extension
beid-mozilla-extension:
  Installed: 4.2.5v4.2.5.88.g4798c05-0sid1
  Candidate: 4.2.5v4.2.5.88.g4798c05-0sid1
  Version table:
     4.2.5v4.2.5.103.ge5c900e-0sid1 1
          1 http://files.eid.belgium.be/debian candidate/sid/main amd64 Packages
 *** 4.2.5v4.2.5.88.g4798c05-0sid1 100
        100 /var/lib/dpkg/status

This output demonstrates that the ‘.88.’ version of the package is already installed, and consequently is assigned a priority of 100. Version ‘.103.’, on the other hand, comes from a software repository with a priority of 1. The higher version will, therefore, not get installed unless you raise its priority—e.g., by specifying an appropriate target release (which will override its priority and set it to 990 instead).

Given that the Belgian eID software archive is assigned a priority of 1 by the APT system, it must be marked “NotAutomatic: yes” in its ‘Release’ file. Indeed, the header lines of the ‘Release’ file for the Belgian eID candidate/sid section show the following:

Origin: FedICT
Label: FedICT
Suite: candidate
Codename: candidate
Date: Ddd, dd Mmm YYYY hh:mm:ss UTC
Architectures: amd64 i386 armhf
Components: sid/main
Description: FedICT eID packages for Debian Sid - unsupported development builds
NotAutomatic: yes

Customising the Belgian eID Software Priority Level

To recapitulate, the developers of the Belgian eID software packages decided to assign a priority of 1 to the candidate/sid section of their Debian software repository, by marking it “NotAutomatic: yes” in its ‘Release’ file. As a consequence, updates provided by this archive will not be automatically installed.

Even so, this repository section is declared “relatively safe to use” by its maintainers, and you may decide that installing updates from it is no more risky than generally installing packages from Debian Testing in the first place.

To allow you to override the default priorities assigned to package versions, the APT system supports the use of Preferences control files, in which you can identify any packages and software sources for which you want to set up customised priority values.

APT Preferences files are kept in the ‘/etc/apt/preferences.d’ directory, and their filenames must either have no extension, or end in “.pref”. They contain one or more entries, each of which consists of multiple lines. Successive entries are separated by blank lines.

Two forms of entries are supported in an APT Preferences file:

  • The specific form assigns a priority to a specified version (or version range) of one or more specified package names.
  • The general form assigns a priority to all package versions provided either by a given distribution (i.e., “release”), or by a given fully-qualified domain name (i.e., “origin”).

Thus, to customise the priority of any package versions in the candidate/sid section of the Belgian eID software repository, you will have to create a general-form entry for a specific distribution. To identify the distribution, you can review what the apt-cache policy command had to say about it:

apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
   1 http://files.eid.belgium.be/debian candidate/sid/main amd64 Packages
     release o=FedICT,a=candidate,n=candidate,l=FedICT,c=main,b=amd64
     origin files.eid.belgium.be
 500 http://security.debian.org testing/updates/non-free amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian-Security,c=non-free,b=amd64
     origin security.debian.org
 500 http://security.debian.org testing/updates/main amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian-Security,c=main,b=amd64
     origin security.debian.org
 500 http://ftp.be.debian.org/debian testing/non-free amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=non-free,b=amd64
     origin ftp.be.debian.org
 500 http://ftp.be.debian.org/debian testing/contrib amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=contrib,b=amd64
     origin ftp.be.debian.org
 500 http://ftp.be.debian.org/debian testing/main amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=main,b=amd64
     origin ftp.be.debian.org
Pinned packages:

The ‘release’ line identifies the distribution, based on the values found in the ‘Release’ file:

o=FedICT
The value from the “Origin:” line.
a=candidate
The value from the “Archive:” or “Suite:” line;
n=candidate
The value from the “Codename:” line.
l=FedICT
The value from the “Label:” line.
c=main
A value taken from the “Components:” line.
b=amd64
A value taken from the “Architectures:” line.

With the exception of the binary system architecture (i.e., the b=amd64 parameter), these values can be copied into the Preferences file in order to uniquely specify the distribution.

Finally, to ensure that updates from the archive will be automatically installed whenever you run an APT upgrade, you will have to raise its priority. In principle, any value from 100 to 999 will do, but typically one of the following two settings will be selected:

Priority 100
The archive will be handled as if it were marked “NotAutomatic: yes” as well as “ButAutomaticUpgrades: yes” in its ‘Release’ file.
Priority 500
This is the default priority of a software archive.

Taking these considerations into account, an appropriate Preferences file may have the following contents:

Package: *
Pin: release o=FedICT, a=candidate, n=candidate, l=FedICT, c=main
Pin-Priority: 100

If you save these text lines into a file named, say, ‘eid_belgium_be.pref’ in the ‘/etc/apt/preferences.d’ directory, then the apt-cache policy command will subsequently display the newly set priority for the Belgian eID archive:

apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 100 http://files.eid.belgium.be/debian candidate/sid/main amd64 Packages
     release o=FedICT,a=candidate,n=candidate,l=FedICT,c=main,b=amd64
     origin files.eid.belgium.be
 500 http://security.debian.org testing/updates/non-free amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian-Security,c=non-free,b=amd64
     origin security.debian.org
 500 http://security.debian.org testing/updates/main amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian-Security,c=main,b=amd64
     origin security.debian.org
 500 http://ftp.be.debian.org/debian testing/non-free amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=non-free,b=amd64
     origin ftp.be.debian.org
 500 http://ftp.be.debian.org/debian testing/contrib amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=contrib,b=amd64
     origin ftp.be.debian.org
 500 http://ftp.be.debian.org/debian testing/main amd64 Packages
     release o=Debian,a=testing,n=stretch,l=Debian,c=main,b=amd64
     origin ftp.be.debian.org
Pinned packages:

Furthermore, the Belgian eID software packages will get automatically updated, along with the rest of the Debian system, whenever you run an APT upgrade.

References

Sunday, 11 June 2017

How to Disable ext4 Lazy Initialisation

Quick reminder to self: When an ext4 file system is created, it will by default use lazy inode table initialisation and lazy journal initialisation. While this feature significantly speeds up file system creation, the kernel will be forced to perform full initialisation afterwards, in the background. As a result, the disk that holds the file system will appear to be constantly busy, even when you do not use it, until its initialisation gets completed.

To disable this behaviour, you can turn lazy initialisation off when you create the file system, by specifying the extended options lazy_itable_init and lazy_journal_init, and setting them to 0:

mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 DEVICE

Of course, file system creation will, then, take noticeably longer, but it will be fully completed by the time the command finishes.

References:

Tuesday, 6 June 2017

Installing Belgian eID Software under Debian Testing

Introduction

The Belgian Federal Public Service Internal Affairs (“Federale Overheidsdienst Binnenlandse Zaken”“Service Public Fédéral Intérieur”“Föderaler Öffentlicher Dienst Inneres”) provides software to allow citizens to log in to online public services using their electronic identity card (“eID”). Official software packages are available for various platforms, including several Linux distributions. For instance, at the time of writing, the software repository officially supports Debian 7 (“wheezy”) and Debian 8 (“jessie”), but the packages for Debian 9 (“stretch”) have not yet been released, given that Debian 9 itself is still in testing state.

If, however, you are running a Linux release that is not officially supported by the Belgian eID software repository, then you have the option of downloading the source files and compiling the software yourself—see the Linux eID software installation page, under the “Downloads for unsupported distributions” header; for a detailed description of this process, you can see, e.g., Linux & networking: installing eid-viewer eid-middleware on debian 9 stretch R2.

How I Installed the Software under Debian Testing

Until recently, I had been using the Belgian eID software only under the Ubuntu Long-Term Support releases—currently, Ubuntu 16.04.

When I wanted to set up the eID software under Debian testing, I initially attempted to simply install the packages for the latest Debian version for which they were released—i.e., “jessie”—but that failed because of dependency issues.

Instead of compiling from source, I decided to look around the eID repository, and I found a candidate area that included a sid directory; given that sid stands for the Debian testing distribution, this seemed like a perfect fit for my situation.

The official Linux eID software installation procedure wants you to download and install an “eid-archive” package to enable the eID package repositories, after which you can install the “eid-viewer” and ”eid-mw” packages in the usual way (i.e., under Debian, through the APT system). This procedure, however, doesn’t let you select which package versions you wish to install—it simply works out which distribution and which release you are running, and, provided that the repository supports your release, it will use the corresponding package versions. If, on the other hand, the release is not officially supported, then, for Debian, it will blindly fall back to wheezy.

Since the official installation procedure was not suitable for me, I decided to download and unpack the “eid-archive.deb” package, and to study how exactly it operated, with the aim of finding out how I could develop a procedure better adapted to my needs. In the process, I realised that I didn’t really need the “eid-archive.deb” package, because it didn’t perform any actions that I couldn’t just as easily do manually: it simply loads the public keys for the repository, and it creates a “sources.list” entry. The public keys can be received easily enough from a Synchronising Key Server and loaded into the local APT keyring, and the “sources.list” entry is pretty simple to generate anyway.

My ‘install-be-eid.sh’ Script

In the end, I decided to write my own Bash script to set up the eID software repository and install the software packages from it—to view it, click: Bash Script ‘install-be-eid.sh’.

To be honest, the script is somewhat more complex than necessary, since it reuses some Python code with which I had been experimenting earlier on, and that I preferred to leave in mostly unchanged, in order to avoid the risk of messing it up beyond repair.

In addition to its bash source code, the script embeds the Python code that it will execute, plus a “sources.list” template that it will modify according to the Debian release under which it is run. I first got the idea of embedding a payload into a bash script from a Linux Journal article.

[UPDATE]

The script also imbeds an APT Preferences file that will raise the priority of the candidate/sid area of the repository, to allow updated packages to be automatically installed from it, together with other system updates, whenever you run an APT upgrade operation. For more information, see my later post, Belgian eID Packages on Debian Testing: Updates Ignored.

[END UPDATE]

Note that the script will refuse to run under anything but a Debian system, because that is what its Python code specifically assumes. If it is run under a Debian release that is supported by the eID repository, then it will set up a “sources.list” entry for that release. Otherwise, assuming that you are running what looks like a Debian pre-release version, it will ask you for confirmation to make the “sources.list” entry refer to the “candidate/sid” area of the repository.

Some notes about the “install-be-eid.sh” Script

  • The script runs its embedded Python code, which attempts to identify the “best” Debian release that is supported by the Belgian eID software repository. Preferably, this should be the currently running Debian release itself, but otherwise, it will be assumed to be the “highest supported release that is older than the currently running one.”

    If successful, then the Python code will output the codenames of two Debian releases: the supposedly “best” one, and the currently running one.

  • If the two codenames produced by the Python code are equal, then the running Debian release is supported by the eID repository, and the “sources.list” entry will be set up to use it.

    If the codenames are different, then the script will assume that your system will best be served by an entry that refers to the “candidate/sid” area of the repository, but it will ask you for confirmation to continue.

  • The public keys to verify the integrity of the repository, will be received from a Synchronising Key Server, and will subsequently be added to the list of keys trusted by the APT system.

  • Once the “sources.list” entry is in place and the public keys are loaded, the script will update your APT package indexes and install the “pcscd”, “pcsc-tools”, “eid-mw”, and “eid-viewer” packages.

  • The script will create an “opt/be-eid” subdirectory under your home location, and it will store its data there, so that it will not need “root” privileges unless and until it successfully collects all data that it requires.

  • To obtain “root” privileges, the script will use the “sudo” command.

Saturday, 27 May 2017

Python Module ‘lsb_release’

Provided that your Linux distribution supports it, you can run the lsb_release command to obtain certain distribution-specific values—e.g.:

lsb_release --all
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.0 (stretch)
Release:        9.0
Codename:       stretch

In addition to the lsb_release command, there exists a Python module of the same name, to allow you to easily obtain the same information from Python. (In fact, the lsb_release command is itself written in Python, and it calls upon said module in order to get this information.)

Note:

At the time of writing, migration from Python 2 to Python 3 is well underway, and most Linux distributions include Python 3, if not as the default Python release, then at least as an option. Therefore, the examples in this document will assume Python 3, and they are not guaranteed to work under Python 2 unchanged.

If you want to try out the lsb_release module interactively, then you will first have to start the appropriate version of the Python interpreter—e.g.:

python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.

You can then import the lsb_release module:

import lsb_release

To obtain the distribution-specific information, you can call the get_distro_information function that the module provides:

distro_info = lsb_release.get_distro_information()

The get_distro_information function returns the requested information in the form of a dictionary—as you can verify if you display the results:

print(distro_info)
{'CODENAME': 'stretch', 'DESCRIPTION': 'Debian GNU/Linux 9.0 (stretch)', 'OS': 'GNU/Linux', 'RELEASE': '9.0', 'ID': 'Debian'}

Presumably, for better readability, you may want to display the information in a clearer format:

print('\n'.join(['%-12s : %s' % (key, value) for (key, value) in distro_info.items()]))
CODENAME     : stretch
DESCRIPTION  : Debian GNU/Linux 9.0 (stretch)
OS           : GNU/Linux
RELEASE      : 9.0
ID           : Debian

Clearly, if your Linux distribution does not support the lsb_release module, then an exception will be raised when you attempt to import it. Thus, if you write a Python program that should be run only on a specific Linux distribution—e.g., Debian—that supports the module, but you want to ensure that it fails silently on a system that does not provide it, then you will have to catch the exception, and handle it in whichever way you deem appropriate.

As an example, the following Python program will either display the codename of the Debian release on which you run it, or produce no output whatsoever if it is run on any other system:

#!/usr/bin/python3

try:
   import lsb_release
   (id, codename) = (lsb_release.get_distro_information()[x] for x in ['ID', 'CODENAME'])
except:
   # The system does not provide the 'lsb_release' module.
   # Quit with a return code of 8.
   exit(8)

if id != 'Debian':
   # The system does provide the 'lsb_release' module, but it is not Debian.
   # Quit with a return code of 4.
   exit(4)

print('You are running %s \'%s\'.' % (id, codename))

Sunday, 17 February 2013

From my Photo Archives: Easter 2010—The Circle of Friends in La Roche-en-Ardenne

We spent the 2010 Easter weekend (from Friday, April 2 until Monday, April 5) in La Roche-en-Ardenne with the family. Here's a picture of my brother's son and me wearing our Ubuntu Circle of Friends t-shirts during our stay there.

Oh, by the way: He's the one with the hair...

Note:

I first posted this picture on Ubuntu Forums, back when I was still a regular visitor there.

Saturday, 1 December 2012

Firefox: How to Open New Tabs at the End of the Tab Bar

Modern browsers, including current Firefox versions, apparently love to insert new tabs, opened from a hyperlink, immediately to the right of the currently active tab.

The following picture, for instance, shows a Firefox window with two open tabs, the first of which is the currently active one:

If you right-click any link on the page, a menu pops up, from which you can select the “Open Link in New Tab” option:

The new tab will open right next to the one from which you selected the link:

Now obviously, I don’t know about you, but I, for one, find this behaviour pretty annoying—I keep expecting the new tab to appear at the rightmost position on the tab bar, like so:

Fortunately, Firefox allows you to reconfigure its tab behaviour in exactly this way, in just a few fairly simple steps:

  1. Type about:config in the address bar:

  2. When you hit the <ENTER> key, you may see the following warning:

    In this case, if you do not want to see this message again whenever you call up the about:config page, then deselect the “Show this warning next time” checkbox.

    To continue, just click the “I’ll be careful, I promise!” button.

  3. Type browser.tabs.insertRelatedAfterCurrent, or any part thereof that uniquely identifies this option, into the search filter:

  4. Right-click the browser.tabs.insertRelatedAfterCurrent entry, and from the menu that pops up, select the “Toggle” option:

    Alternatively, you can just double-click the entry—which has the exact same effect.

  5. The value will change from true to false:

Done! From now on, new tabs will open to the right of all the existing ones.

Monday, 15 October 2012

The hexdump Utility: A Tutorial

Introduction

The hexdump utility displays file contents in a user-specified format. This tutorial will take a look at some of its features to help you control the format of the output produced by the program. A more complete description of the command is available on its man page—which, of course, you can read with the following command:

man hexdump

Example Input File

If you want to follow along with the examples in this tutorial, then I suggest you run the following command to create the “mpoppins” input file:

echo 'supercalifragilisticexpialidocious' > mpoppins

The tutorial will attempt to explain enough about the hexdump utility to allow you to create the following output from this input file:

00000000    73 75 70 65 72 63 61 6C  69 66 72 61 67 69 6C 69    supercalifragili
00000010    73 74 69 63 65 78 70 69  61 6C 69 64 6F 63 69 6F    sticexpialidocio
00000020    75 73 0A                                            us.

Default Output

The simplest invocation of the hexdump utility displays a hexadecimal dump of the input file:

hexdump mpoppins
0000000 7573 6570 6372 6c61 6669 6172 6967 696c
0000010 7473 6369 7865 6970 6c61 6469 636f 6f69
0000020 7375 000a
0000023

Each output line displays the input offset in hexadecimal, followed by eight chunks of data. Each chunk shows two input bytes, processed as a 16-bit integer number, and is output with four hexadecimal digits.

Note that, since the x86 family of CPUs stores integers in little-endian byte order, integers will appear “byte-reversed”. For instance, since the first byte of the example file is hexadecimal ‘73’ and its second byte is hexadecimal ‘75’, the integer number formed by these two bytes will be hexadecimal ‘7573’, not ‘7375’.

Also note that, even though the size of the input file is uneven, the last chunk of input data will be padded with a null byte. Thus, the last chunk is processed as if the last input byte—hexadecimal ‘0A’—were followed by an additional null byte, so that it, too, can be displayed as a two-byte hexadecimal integer.

Finally, the last output line displays the input offset after the input file is completely processed—i.e., the input file size—also in hexadecimal. Therefore, even though the output shows an even number of bytes, the number displayed on the last output line indicates that the input file size is actually uneven.

The ‘-x’ Option

If you specify the ‘-x’ option to the hexdump utility, then the output will be equivalent to the default, even though the spacing will be different:

hexdump -x mpoppins
0000000    7573    6570    6372    6c61    6669    6172    6967    696c
0000010    7473    6369    7865    6970    6c61    6469    636f    6f69
0000020    7375    000a
0000023

The ‘-C’ Option

The ‘-C’ option produces what the program calls “canonical hex + ASCII” output:

hexdump -C mpoppins
00000000  73 75 70 65 72 63 61 6c  69 66 72 61 67 69 6c 69  |supercalifragili|
00000010  73 74 69 63 65 78 70 69  61 6c 69 64 6f 63 69 6f  |sticexpialidocio|
00000020  75 73 0a                                          |us.|
00000023

This time, each line displays sixteen bytes, one by one, in hexadecimal, followed by the same sixteen bytes in ASCII (where non-printable characters are replaced with a single “.”); the sequence of ASCII characters is enclosed in vertical bars (i.e., “|”).

Using a Format File

The output format that the hexdump utility produces, is specified through a set of format strings. Certain formats are built into the program, and can be activated with a simple command-line option—e.g., the ‘-x’ or ‘-C’ options demonstrated above.

The program is not limited to its built-in formats, however, but it allows you to specify a custom format instead. You can store a custom format specification into a format file, which you can then identify to the program using the ‘-f’ command-line option.

The format file consists of one or more text lines, where each line defines one format string.

The simplest format string consists simply of one format—which should be enclosed in double quotes. For example, a simple hexadecimal format, with a single space separating successive output chunks, can be specified as follows:

"%x "

To store this format into a format file, you can run, e.g., the following command:

echo '"%x "' > fmt-01

You can then pass this format file to the hexdump command:

hexdump -f fmt-01 mpoppins
65707573 6c616372 61726669 696c6967 63697473 69707865 64696c61 6f69636f a7375

As you can see, the “%x” format produces hexadecimal output, with four input bytes per chunk. Again, each chunk is processed as an integer number, which (on an x86-family computer) causes the bytes to appear in reversed order. Also, as the last chunk demonstrates, leading zeroes are suppressed.

If you want to prevent this zero suppression, then you can specify the desired field width, with a leading zero, on the format—e.g., for eight-character fixed-width output:

echo '"%08x "' > fmt-02
hexdump -f fmt-02 mpoppins
65707573 6c616372 61726669 696c6967 63697473 69707865 64696c61 6f69636f 000a7375

If you omit the leading zero on the field width, then the leading zeroes will be replaced with spaces:

echo '"%8x "' > fmt-03
hexdump -f fmt-03 mpoppins
65707573 6c616372 61726669 696c6967 63697473 69707865 64696c61 6f69636f    a7375

It should be noted that the field width determines the minimum number of output characters produced for the field. If a value doesn’t fit, then the output width will be increased as needed.

In addition to the field width, the format specification may include a precision, which, if present, must be preceded by a period (“.”), and identifies the minimum number of digits that must be produced:

echo '"%08.8x "' > fmt-04
hexdump -f fmt-04 mpoppins
65707573 6c616372 61726669 696c6967 63697473 69707865 64696c61 6f69636f 000a7375

Even though the precision may seem redundant (particularly if, as in the above example, it is set equal to the field width), it appears to be common practice to include it nonetheless, and this convention will also be used in the remainder of this tutorial.

By the way, the precision does make a difference if it is less than the field width—for example:

echo '"%08.7x "' > fmt-05
hexdump -f fmt-05 mpoppins
65707573 6c616372 61726669 696c6967 63697473 69707865 64696c61 6f69636f  00a7375

This time, all values are formatted into eight-character output fields, but—as the last value shows—the number of actual digits produced may be reduced to seven, if possible.

You can change the number of input bytes per chunk. In the case of the hexadecimal output format, four input bytes per chunk are assumed by default, but you can request either one or two input bytes per chunk instead. This byte count should be preceded by a forward slash (i.e., “/”) and followed by the format to which you want to apply it—e.g., for a hexadecimal format with a byte count of 2:

echo '/2 "%04.4x "' > fmt-06
hexdump -f fmt-06 mpoppins
7573 6570 6372 6c61 6669 6172 6967 696c 7473 6369 7865 6970 6c61 6469 636f 6f69 7375 000a

Similarly, with a byte count of 1:

echo '/1 "%02.2x "' > fmt-07
hexdump -f fmt-07 mpoppins
73 75 70 65 72 63 61 6c 69 66 72 61 67 69 6c 69 73 74 69 63 65 78 70 69 61 6c 69 64 6f 63 69 6f 75 73 0a

Clearly, displaying all of the output on one long line is not particularly elegant. You may, for example, prefer to display eight chunks of two input bytes (for a total of sixteen bytes) per line. This number of output chunks is called the iteration count, and should precede the forward slash—e.g.:

echo '8/2 "%04.4x "' > fmt-08
hexdump -f fmt-08 mpoppins
7573 6570 6372 6c61 6669 6172 6967 696c7473 6369 7865 6970 6c61 6469 636f 6f697375 000a

Note that all of the output is still displayed on one line, since you haven’t requested any newlines in your format just yet. You may, however, be surprised that after every eighth chunk, there appears to be a space missing—e.g., the eighth and ninth chunks run into one another and are displayed as ‘696c7473’. This is not a bug—it’s a feature: If the iteration count is greater than 1, then there will be no trailing spaces produced following the last iteration.

To specify a newline in your format, you should use the “\n” escape sequence; you can simply append it, enclosed in double quotes, to your format string—i.e.:

echo '8/2 "%04.4x " "\n"' > fmt-09
hexdump -f fmt-09 mpoppins
7573 6570 6372 6c61 6669 6172 6967 696c
7473 6369 7865 6970 6c61 6469 636f 6f69
7375 000a

The iteration count in this example is 8, and the byte count is 2. As a consequence, the format immediately following will process an input chunk of two bytes at a time, and will be applied eight times in a row. The format string will, therefore, work with blocks of sixteen input bytes in total.

Each two-byte chunk will be displayed as four zero-padded hexadecimal digits, followed by a space. However, since the iteration count is greater than 1, the space following the last iteration will be suppressed.

Finally, after eight iterations are executed, a newline will be output.

The format string will be applied to each sixteen-byte input block in succession, until the end of the input file is reached.

Note that the last block has only four input bytes left to process—the final three bytes from the input file, plus one padding null byte. In such a case, the remaining iterations will output spaces instead of actual values obtained from the input file. In other words, spaces will be appended to the last line of output, as shown below—where each ‘•’-sign represents a space that is output in place of a digit from a missing input value:

7573 6570 6372 6c61 6669 6172 6967 696c
7473 6369 7865 6970 6c61 6469 636f 6f69
7375 000a •••• •••• •••• •••• •••• ••••

Displaying the Current Input Offset

To display the current input offset in hexadecimal, the hexdump utility supports the special “%_ax” format specification. Just like the “%x” format, it supports a field width and a precision.

Thus you can, for instance, expand the previous example to include the current input offset, with seven hexadecimal digits, at the start of each line, like this:

echo '"%07.7_ax " 8/2 "%04.4x " "\n"' > fmt-10
hexdump -f fmt-10 mpoppins
0000000 7573 6570 6372 6c61 6669 6172 6967 696c
0000010 7473 6369 7865 6970 6c61 6469 636f 6f69
0000020 7375 000a

Note that, except for the final output line that displays the total number of bytes processed, this result is identical to the output that the program produces by default.

Displaying the Final Input Offset at the End of the Output

The hexdump utility also supports a “%_Ax” format specification (with an uppercase ‘A’), which will display the input offset only after the input file is completely processed. You can simply append it to your existing format string, like so:

echo '"%07.7_ax " 8/2 "%04.4x " "\n" "%07.7_Ax\n"' > fmt-11
hexdump -f fmt-11 mpoppins
0000000 7573 6570 6372 6c61 6669 6172 6967 696c
0000010 7473 6369 7865 6970 6c61 6469 636f 6f69
0000020 7375 000a
0000023

The result is now completely identical to the default output from the program.

Multiple Format Strings

All of the examples in this tutorial so far specified a single format string. If you want to use multiple format strings instead, then just enter each of them on a separate line in the format file.

It is, for example, common practice to create a separate format string for the “%_Ax” format specification—e.g.:

echo '"%07.7_ax " 8/2 "%04.4x " "\n"
"%07.7_Ax\n"' > fmt-12
hexdump -f fmt-12 mpoppins
0000000 7573 6570 6372 6c61 6669 6172 6967 696c
0000010 7473 6369 7865 6970 6c61 6469 636f 6f69
0000020 7375 000a
0000023

If multiple format strings are given, then they will be applied sequentially to each input block.

In fact, if you really want to, you can enter each format specification on its own, separate format string—as follows:

echo '"%07.7_ax "
8/2 "%04.4x "
"\n"
"%07.7_Ax\n"' > fmt-13
hexdump -f fmt-13 mpoppins
0000000 7573 6570 6372 6c61 6669 6172 6967 696c
0000010 7473 6369 7865 6970 6c61 6469 636f 6f69
0000020 7375 000a
0000023

In this example, it doesn’t really matter whether or not you split up the format into multiple format strings—the output is not affected. As later examples in this tutorial will show, however, this holds true only in the simplest of cases.

Displaying Bytes in Hexadecimal and as Printable ASCII

By now, it should be obvious how you can display bytes, one by one, in hexadecimal, with sixteen bytes per line:

echo '16/1 "%02.2x " "\n"' > fmt-14
hexdump -f fmt-14 mpoppins
73 75 70 65 72 63 61 6c 69 66 72 61 67 69 6c 69
73 74 69 63 65 78 70 69 61 6c 69 64 6f 63 69 6f
75 73 0a

If you want to output an extra space between the eighth and ninth bytes on each line, then you will have to specify two format units with an iteration count of 8, with two spaces in between—like so:

echo '8/1 "%02.2x " "  " 8/1 "%02.2x " "\n"' > fmt-15
hexdump -f fmt-15 mpoppins
73 75 70 65 72 63 61 6c  69 66 72 61 67 69 6c 69
73 74 69 63 65 78 70 69  61 6c 69 64 6f 63 69 6f
75 73 0a

Keep in mind that, if the iteration count on a format unit is greater than 1, then the space following the last iteration will be suppressed. Hence, the literal string following the first format unit needs two spaces instead of just one.

Note that this format string still processes the input data in 16-byte blocks, which will be output in two 8-byte halves.

Also, it is important to enter the format specifications on a single format string; if you split the format, then the result will different, and likely not what you want:

echo '8/1 "%02.2x " "  "
8/1 "%02.2x " "\n"' > fmt-16
hexdump -f fmt-16 mpoppins
73 75 70 65 72 63 61 6c  73 75 70 65 72 63 61 6c
69 66 72 61 67 69 6c 69  69 66 72 61 67 69 6c 69
73 74 69 63 65 78 70 69  73 74 69 63 65 78 70 69
61 6c 69 64 6f 63 69 6f  61 6c 69 64 6f 63 69 6f
75 73 0a                 75 73 0a

This time, both format strings consume eight input bytes. As a result, the input data will be processed in eight-byte blocks, which will be passed to both of the format strings sequentially; in other words, each eight-byte input block will be formatted twice.

Consequently, if you want to display the input data in two formats—e.g., both in hexadecimal and as printable ASCII characters—then you will have to specify two format strings: one for the hexadecimal output, and another one for the printable ASCII output. Given that the format specification for printable ASCII output is “%_p”, you can produce the desired result as follows:

echo '8/1 "%02.2x " "  " 8/1 "%02.2x " "    "
16/1 "%_p" "\n"' > fmt-17
hexdump -f fmt-17 mpoppins
73 75 70 65 72 63 61 6c  69 66 72 61 67 69 6c 69    supercalifragili
73 74 69 63 65 78 70 69  61 6c 69 64 6f 63 69 6f    sticexpialidocio
75 73 0a                                            us.

Both format strings in this example will process the input data in 16-byte blocks. The first format string will split the block in two 8-byte halves, and display the byte values in hexadecimal; the second format string will display the block as 16 printable ASCII characters.

Finally, you may want to output the current input offset at the start of each line:

echo '"%08.8_ax    " 8/1 "%02.2x " "  " 8/1 "%02.2x " "    "
16/1 "%_p" "\n"' > fmt-18
hexdump -f fmt-18 mpoppins
00000000    73 75 70 65 72 63 61 6c  69 66 72 61 67 69 6c 69    supercalifragili
00000010    73 74 69 63 65 78 70 69  61 6c 69 64 6f 63 69 6f    sticexpialidocio
00000020    75 73 0a                                            us.

You can probably guess by now what happens if you accidentally type the entire format on a single format string—as follows:

echo '"%08.8_ax    " 8/1 "%02.2x " "  " 8/1 "%02.2x " "    " 16/1 "%_p" "\n"' > fmt-19
hexdump -f fmt-19 mpoppins
00000000    73 75 70 65 72 63 61 6c  69 66 72 61 67 69 6c 69    sticexpialidocio
00000020    75 73 0a

This time, the input data will be processed in 32-byte blocks. The first 16 bytes of the block will be displayed in hexadecimal, while the last 16 bytes will be output as printable ASCII characters.

Typing the Format Strings on the Command Line

Instead of saving the format strings to a file, you may prefer to enter them directly on the command line when you run the hexdump program. In such a case, you should omit the ‘-f’ option, and replace it with one or more ‘-e’ options.

The ‘-e’ option requires a format string as a parameter, and one such option must be specified for each format string—e.g.:

hexdump -e '"%08.8_ax    " 8/1 "%02.2x " "  " 8/1 "%02.2x " "    "' -e '16/1 "%_p" "\n"' mpoppins
00000000    73 75 70 65 72 63 61 6c  69 66 72 61 67 69 6c 69    supercalifragili
00000010    73 74 69 63 65 78 70 69  61 6c 69 64 6f 63 69 6f    sticexpialidocio
00000020    75 73 0a                                            us.

Closing Notes

  • By default, the hexdump utility will collapse successive identical output lines, which will be indicated with an output line that contains just a single asterisk.

    To illustrate this feature, you can create a data file that contains 512 null bytes, and pass it to the program as the input file:

    dd if=/dev/zero of=nulls count=1 bs=512
    hexdump nulls
    0000000 0000 0000 0000 0000 0000 0000 0000 0000
    *
    0000200

    If you want to prevent this behaviour, and want to display all input data normally, then you can use the ‘-v’ option:

    hexdump -v nulls

  • The default output from the program is produced using the following format strings:

    "%07.7_Ax\n"
    "%07.7_ax " 8/2 "%04x " "\n"

  • The ‘-x’ option, which is equivalent to the default except for the spacing, uses the following format strings:

    "%07.7_Ax\n"
    "%07.7_ax " 8/2 "   %04x " "\n"

  • The ‘-C’ option, on the other hand, uses the following format strings:

    "%08.8_Ax\n"
    "%08.8_ax  " 8/1 "%02x " "  " 8/1 "%02x "
    "  |" 16/1 "%_p" "|\n"

  • Even though this tutorial should give you a pretty good idea about the operation of the hexdump utility, the program supports quite a few additional features. As noted above, more information is available on its man page.