Hack 29 Live Install Your Own Programs

Write your own live installer scripts to install programs you need that aren't on Knoppix by default.

As already briefly discussed in [Hack #27], there are several techniques that allow live installation of software to the home directory. This hack discusses these techniques, shows you how you can create your own live installer modules, and gives an overview of possible future solutions.

3.10.1 The Problem

To understand how to write your own live installer scripts, you must understand the main problem with installing software to the home directory: most software expects to be installed in /. Most programs install their files in many different directories in Linux, with files in /usr, /etc, /usr/lib, /var, and other directories. Under Knoppix, the entire /usr directory is part of the read-only CD-ROM filesystem. Even if you copy everything to /home/knoppix/usr, /home/knoppix/etc, /home/knoppix/var, etc., it doesn't work because the program expects its files to be under the root directory. Even if that particular software does not need any other datafiles, most software depends on libraries, and these libraries are also searched for in /usr/lib.

3.10.2 Use the Source Luke

One solution to this problem is to use the source. With the source code of a program, it is possible to install any program in the home directory.

The paths that determine where a program looks for its files are set during compile time. And most programs support setting the root to another prefix. With an autotools-compliant package, you can change the prefix by running the following command from the root of the source directory:

knoppix@ttyp0[knoppix]$ ./configure --prefix=/home/knoppix/.dist/

knoppix@ttyp0[knoppix]$ make

knoppix@ttyp0[knoppix]$ make install

Unfortunately, compiling is not a perfect solution. First, compiling a program is slow (depending, of course, on the speed of the computer). If I want to install a program, I want it now, not after a long coffee break. I'm not a Gentoo masochist!

Second, compiling a program often requires many dependencies, which must also be compiled, and these dependencies may have even more dependencies that you must also compile, and so on.

3.10.3 Dependency Hell

To install new software in Debian, use the apt-get utility. It not only installs the software you want, but also all dependencies this software needs and all dependencies of the dependencies. Unfortunately, apt-get isn't designed to be run from CD-ROM but from hard disk, and expects certain parts to be writable that are not writable under Knoppix. Despite this problem, the live installer uses wrappers that execute the original programs with special parameters to use apt-get.

3.10.4 Patch Me if You Can

After you download the .deb you need, use dpkg to extract the files into your local ~/.dist directory:

knoppix@ttyp0[knoppix]$ dpkg -x file

.deb ~/.dist/

To start the program, you must set up a library path so that you can install libraries in places other than the read-only /usr/lib directory. Fortunately, this is possible without the need for root rights. To modify the library path for a program, type:

knoppix@ttyp0[knoppix]$ export PATH=$HOME/.dist/usr/bin:$PATH

knoppix@ttyp0[knoppix]$ export LD_LIBRARY_PATH=$HOME/.dist/usr/lib:$LD_LIBRARY_PATH

knoppix@ttyp0[knoppix]$ 

programname

Now the program searches for libraries and programs under your ~/.dist directory—at least most programs work this way. The live installer transparently writes a ~/.bashrc to set these paths so that starting programname in the shell also works. Games that need certain datafiles are still problematic. There isn't a set environment variable for these types of datafiles. Instead, you must determine which files the program is going to reference (dpkg -L packagename at least shows you which files the program includes). This is different for each program, and is one of the challenges that you must face when using live installs. Once you determine the paths that need to be changed, you must path the binary itself to change the hardcoded file paths. For example, tuxracer stores its datafiles in /usr/share/games/tuxracer. To change this path to a local path, run:

knoppix@ttyp0[knoppix]$ perl -pi -e \

's|/usr/share/games/tuxracer|./../share/games/tuxracer|' $DESTDIR/games/tuxracer

This changes all occurrences of the global path /usr/share/games/tuxracer to the local path /share/games/tuxracer. If you start the program from /home/knoppix/.dist/usr/bin/, it finds the datafiles in /home/knoppix/.dist/usr/share/games/tuxracer.

3.10.5 Do It Yourself

OK, now you want to create your own live installer module. The best method is to find one of the existing modules and modify it for your program. To use your own live installer modules, make a directory called modules in your home directory, and run the live installer:

knoppix@ttyp0[knoppix]$ cp -a /usr/share/knx-live-inst/modules ./

knoppix@ttyp0[knoppix]$ cp -a modules/gkrellm modules/myprog



knoppix@ttyp0[knoppix]$ knx-live-inst.sh

This command makes a mirror copy of the Knoppix live installer modules in your home directory, and then copies the gkrellm modules to use as a base for your program. The files in modules/myprog allow you to change different settings for your module:

description

Describes the live installer menu

exes

Is the main executable for the program

files

Shows which files to download and from where

install.sh

Explains how to install the program

menu

Is the name of the menu entry for the created Extra Software menu

While description, exes, and menu are simple, the one-line text file, files, has a special format, and install.sh is a shell script.

For a program that can be installed by apt, files is as simple as:

# Format of this is:

# basename location_to_download type md5sum

myprog         none    apt     none

And install.sh in most cases is as simple as:

#!/bin/bash

#

# File to actually install the software

# copy main prog



cp -fa myprog/usr/* $DESTDIR/

Sometimes (such as in the previous tuxracer example), you must add binary patches and wrappers that run when the program is installed to work around difficult file path issues. In these cases, use the tuxracer module as a base so that you can see how it handles patching and wrappers.

3.10.6 Advanced Techniques

Some programs, such as the Nvidia drivers, won't work with the previous techniques. The problem is usually that these programs must be in, and write to, /usr. One option is to build a huge symlink farm. In a symlink farm, at boot every directory on the CD-ROM is copied as a new directory in the filesystem, and files in those directories are not copied but symlinked. This makes everything in /usr writable. However, if you build a symlink farm when you create the live CD, you waste space; if you build it while you boot the live CD, it takes a very long time.

One solution to this problem is the dynamic creation of symlink farms. This is how the nvidia live installer works. The installer runs make -n install and checks the output of that program for files that need to be writable, and then builds the necessary symlink farm.

Another possible solution that isn't being implemented in the Knoppix live installer is to use overlay filesystems. The idea behind an overlay filesystem is simple. You can overlay /usr with /ramdisk/usr, and although you can read all files that lie in /usr, all file writes go to /ramdisk/usr. There are several of these filesystems, and they all have some problems:

Ovlfs

Is the oldest and best working overlay filesystem (development started with kernel 2.0), but its source code is very complex and not portable to kernel 2.6.

Translucency

Attempts to redirect files and directories if there is an open or readdir attempt. Translucency unfortunately still lacks support for symlinks, and directory entries that exist in both directories before the overlay are presented twice.

Linux User Space Filesystem (LUFS)

Is slow because it uses user space tools as opposed to a kernel module.

Device Mapper

Has a snapshot functionality that doesn't work reliably because it works on the so-called block-layer, which means the underlying filesystem does not know how much space is available.

What all of these solutions have in common is that they don't make use of the VFS Linux Virtual Filesystem Layer (VFS) very much, but reimplement it one way or another. It's possible that eventually a good overlay filesystem will be included in the main kernel tree.

—Fabian Franz