TheHackersManual2015RevisedEdition

Linus Torvalds has the final say on what goes into the Linux kernel, and over the last couple of decades he has proven to be a good project manager (if a bit mouthy at times).

Consequently, there haven’t been any major forks of the kernel based on developer fall-outs, but separate branches of the kernel source code do exist. These incorporate experimental features or take the kernel in a different direction than intended by Torvalds et al, and the most notable include:

pf-kernel (http://pf.natalenko.name) A bunch of “awesome features not merged into mainline” (the official kernel source tree). This includes an alternative I/O scheduler, TuxOnIce for improved hibernation functionality, and the -ck patch to boost responsiveness.

TinyLinux (www.tinylab.org/project/tinylinux) This patchset was designed with resource-limited embedded systems in mind, and attempts to reduce the disk and memory footprint of the kernel.

RT Patch (https://rt.wiki.kernel.org) The standard Linux kernel isn’t brilliant at real-time operations, because on most servers and desktops, it doesn’t matter if a certain task takes an extra 0.01s to complete if the system is under load. But if you’re an audio professional or using Linux to control machinery, you want to guarantee that the kernel will do certain things to exact deadlines (to keep everything in sync) and this patch provides this ability.

Helpfully, the make install process also updates the Grub bootloader. If you look inside /boot/grub/grub.cfg you’ll see new entries for the version of the kernel that you just built and installed.

Now you’re ready to take the most exciting step: rebooting into your shiny new personalised kernel. Just select it from the Grub menu, cross your fingers, and see what happens. All being well, your system will boot up properly and you can explore the new features that you enabled – but if not, no worries. Just reboot again and select your old kernel (it’s likely to be under the ‘Advanced options’ submenu). Linux is

Note that many patchsets take a while to sync with the latest kernel tree, so you might not be able to find a patch to match the exact kernel version that you’re building.

Here’s what the RT Patch looks like – the highlighted part shows how lines are removed and added.

excellent at supporting multiple kernel versions on one machine, so it’s unlikely that your system will become completely unbootable.

If you need to make further changes to your kernel, run make xconfig again, then follow the procedure above. You can also remove compiled files by entering make clean, but that still leaves .config and some other files in place. To reset the kernel sources to their original, pristine form, use:

make mrproper

After entering this, all temporary files and settings will be removed, just as if you had extracted a fresh kernel tarball. Θ

Create your own kernel patch

If you fancy trying your hand as a kernel hacker, the first thing you’ll need to do is add something useful to the kernel. That kind of thing is entirely beyond the scope of this guide, but in order to share your work with the world, you’ll also need to create a .patch file that shows the difference between the original kernel and your souped-up one. To do this, you need two directories: one with the original, vanilla kernel source code, and one containing your changes. So, in /usr/src you could have linux-3.18.7 and linux-3.18.7-me, the latter being where you’ve hacked some code. To generate the patch file, enter this:

diff -uprN linux-3.18.7/ linux-3.18.7-me/ > myfile.patch

You can now compress the patch file and distribute it to others, and they can apply it using the instructions that have been described in the main text. If you’ve got something really cool to show and want to send it to a kernel developer, read

Documentation/SubmittingPatches in the kernel source code first – it contains lots of useful tips.

The big yellow button on www.kernel.org always points to the latest stable source code.

Among the plethora of goodies in your favourite Linux distro, the package manager is surely one of the most important. This guy

is responsible for so much, it’s hard to fathom the amount of behind-the-scenes grunt

work it does: updating package lists, resolving dependencies, deciding which order to install those dependencies in, fetching

and verifying package checksums and authenticity, making sure they are unpacked in the right place, performing various post-install chores, and finally recording the success or otherwise of these efforts. And all because you, on a

caprice, decide that you want to install KDE. So let’s take a closer look.

Let’s begin with some introductory nomenclature. You might know some of these words already. Good for you if so, but if not

we’ll explain the key ones. First off, a package is essentially a container for a bunch of files, together with some instructions on what to do with these files in order to make them useful. A package may have some dependencies – that is, other packages which are required so that it

can do its thing. There may also be some packages which improve functionality without being essential; these are sometimes called optional dependencies.

In order for things to move forward, a complete dependency graph must be figured out. This structure contains all the dependencies of our desired package, and all of their dependencies and so on. From here we need to calculate an

ordered and duplicate-free list so that no package will be installed before any of its dependencies, which is a problem requiring some mathematics. OpenSUSE and the new Hawkey back-end in Fedora offer the flashiest solution to this conundrum, by recoursing to a

superiority of any of these, but rather to elucidate the different approaches, features and quirks of these different systems and show how common tasks are performed therein. This is also entirely pertinent to derivative distributions (Ubuntu, Archbang, Sabayon), although some of these will also have their own package managers. Let’s begin with a short summary of each species.

Three package managers

There are those who would say Portage isn’t really a package manager, and there are those who would say Gentoo is not really a distribution – preferring instead to form dreadful descriptors using the prefix ‘meta’. No matter. Let’s not dwell on these semantics. It is certainly the case that Gentoo being a source-based (meta)distribution means that Portage has a great deal of responsibilities. Somewhat controversially, it is written in

Python, which detractors claim has an unnecessary impact on performance and poses the risk of an unrecoverable system should your Python install break, although we have recovered Gentoo systems suffering much worse. You can get binaries for major packages which can be shoehorned on without a working Portage, or from a different OS, if things are sufficiently messy.

Portage maintains a folder (hereafter the ‘portage tree’) of available packages grouped by category – for example, dev-libs/ and www-client/. Typically multiple versions of a given package are available, each having its own ebuild – a special script with all the required particulars. Core system packages are grouped together in a set called @system, and packages explicitly installed (that is, only stuff you specifically asked for and not their dependencies) end up in the @world set. The @s are a relatively new addition since

You can make lovely dependency graphs using garchdeps from the Arch User Repository.

Bread and butter tasks

Now let’s list some common garden tasks. It is useful to use Portage’s --pretend or apt-get’s

--dry-run options, which both do the same thing – that is, nothing – in order to see what havoc the command would wreak if it were run without this safety net. Pacman has no such option, but it does at least have the decency to require confirmation before it does anything. The indecent can turn that off with --noconfirm.

Install the latest available version of a package:

emerge packagename pacman -S packagename apt-get install packagename

Updating the system’s list of available packages:

emerge –sync # (or eix-sync) pacman -Sy

apt-get update

Upgrading all packages to the latest available version:

emerge -Duv @world

The -D (--deep) option upgrades dependencies as well, -v (--verbose) gives some more info about USE flags. You can also use the -N (--newuse) option to reinstall

packages where USE flags have changed. pacman -Su

Use pacman -Syu to do a combined update/upgrade, which is the recommended procedure.

apt-get upgrade

Use apt-get dist-upgrade for a more destructive upgrade, generally only needed for Unstable/Testing.

Searching for a package:

emerge --search packagename # (this is slow and eix is your friend)

pacman -Ss packagename # (or -Qs to search installed packages)

apt-cache search packagename

Removing a package: emerge -C packagename apt-get remove packagename

or apt-get purge to obliterate configuration files too.

pacman -R packagename

Use pacman -Rs to recursively remove dependencies too.

While doing a full package update should never be a big deal on Debian, and should only rarely be so on Arch, the situation on Gentoo is a little different. A good rule of thumb is that something will go awry when some packages undergo major upgrades. While this can often be blamed on using unstable versions of packages and unofficial overlays, sooner or later every Gentoo user will run into these sorts of troubles. Sometimes a USE flag of package A will demand that some version of package B is installed, but meanwhile package C insists that a different version of package B be present. You can usually work around this sort of thing by masking or unmasking different versions of A, B and C, or changing

the offending USE flag, but often this just results in different conflicts. Sometimes the blocks are much more complicated, and while Portage does try its best, bless, to tell you where the issues are arising, it’s not always easy to see how to proceed. The Gentoo Forums will usually have people suffering from your particular ailment, and sooner or later someone figures it out, or the problem is obviated by a new release.

Packagekit

The Packagekit project is an abstraction layer designed to enable a distribution-agnostic approach to package management. It aims to implement a ‘fire-and-forget’ mechanism for all your software installation requirements. The new Gnome

Software application is an example of a packagekit front-end, as is KDE’s Apper, as is the pkcon command-line tool. The front-end talks to

the packagekitd daemon, which talks to a distribution-specific back-end. All this talking takes place over Dbus, so if a user initiates

a lengthy package transaction and then logs out or their X session crashes, then the transaction will proceed unaffected. Backends exist for most distros, but not all are considered stable/useful. The full feature matrix is available at www.packagekit.org/ pk-matrix.html. For example, the whole Packagekit framework would need to be rewritten to support Portage in a sane manner, and Pacman often requires (educated) user intervention in times of package upheaval,

Soon geeks and zombies alike will be using Software-Centre-like application stores.

which violates Hughsie’s law regarding noninteractivity of transactions. That said, Packagekit is largely operational for Debian, Fedora and OpenSUSE, and it is entirely possible that users of these distros will be able to prosecute all their package-related affairs through Packagekit, without having any contact with their distro’s native tools.

Appstream is an XML schema backed by major Linux vendors for providing package screenshots and ratings and review data as well as a standardised database for storing repository data. The ultimate goal is to provide all the underlying gubbins so that anyone with the means and inclination could build a distribution-agnostic Software Centre, or dare I use the term ‘App Store’.

This type of convergence may seem like something of a Linux anathema, in the sense that we like lots of different ways of doing things and learning lots of commands for doing the same thing in different places. However, it should not in any way be seen as an attempt to homogenise the whole package management process. Each distro’s tried and tested tools aren’t going anywhere. Sure, they will evolve to better fit in with these new models, but only for the benefit of that distro. Packagekit and friends will provide the means to enable much needed consistency for casual package management from the desktop. Θ

Arch is aimed at intermediate or advanced Linux users, since terminal commands and manual editing of

configuration files are de rigueur. The Arch wiki provides thorough documentation though, so even if you’ve dabbled with Ubuntu or Mint and like a challenge, then feel free to give it a shot on a spare (or virtual) machine. If the install process doesn’t put you off then the chances are you’ll get on fine. Arch aims to expose as much of the system as possible to the user, but to have these internals arranged in a coherent manner. So with a bit of patience there is potential to really advance your skills. On the other hand, if you just tried Ubuntu and are looking for an alternative because you ran into difficulties, Arch is unlikely to solve your problems. However, Manjaro Linux, an Arch derivative with more of a focus on userfriendliness, might just be the answer.

Discerning Arch users enjoy up to the minute releases of graphics drivers (wholesome and proprietary), web browsers, and pretty much any software you would care to name. It’s often a sore point on the Ubuntu Forums that Canonical’s repos lag behind developer releases. Then again, those forums have plenty of sore points, and a conservative update strategy does allow for a more thoroughly tested and stable distribution. Speaking of which, the stable releases of Debian happen once every three years, so at the time of writing while Arch users are enjoying the shiny new 3.15 kernel, users of the current stable release of Debian (codenamed Wheezy) are still

rocking the 3.2 series. Of course, this branch is still maintained, and any security fixes are applied promptly. And in fairness, Debian users can use backports to get newer versions of certain software,

but not those – such as Gnome – that depend on so many new libraries. Likewise, Ubuntu has PPAs, Gentoo has Overlays and Fedora has other repos, such as RPM Fusion.

Currently, Arch Linux distributes packages for two architectures: i686 (32-bit) and x86_64 (64-bit). This greatly simplifies the

The Arch way

The ethos behind Arch Linux is summed up in five principles: simplicity, code-correctness over convenience, user-centricity, openness and freedom. It aims to provide a lightweight foundation upon which the user can build however they see fit. Linux is complex and sometimes complicated, and Arch makes no attempt to hide any of it behind layers of abstraction or GUI. Instead, it lays them out in as clutter-free and transparent a manner as possible. Software is kept as close to the

developers’ release as possible and patching is avoided unless absolutely necessary. For example, the stock kernel package usually contains only a handful of bug fixing patches (two at the time of writing) rather than the truckloads you will find in other distros.

Arch emphasises that the user is responsible for administering their system, and the provided tools aim to make this process efficient as opposed to easy. The thinking here is that by oversimplifying things, control is sacrificed.

There is a very strong community among which users are encouraged to share their problems (coherently!) and solutions. Arch users are free to configure their system to their every whim: Whether you want a headless server, a fullyfledged KDE desktop, or a real-time patched audio workstation, Arch is perfectly capable of adapting around you. Proprietary and other restrictively licensed software (such as Flash and mp3 codecs) is available in the repositories, placing functionality above ideology.