Hack 89 Distribute the Load with ClusterKnoppix

ClusterKnoppix is a live-CD distribution that takes advantage of the openMosix cluster technology to turn any room full of computers into a makeshift openMosix cluster.

Clusterknoppix (http://bofh.be/clusterknoppix) combines the functionality of openMosix with the ease of use of Knoppix. It contains an openMosix kernel and the openMosix userspace utilities, along with various other tools for monitoring and manipulating the cluster, like openMosixview (http://www.openmosixview.com). It is meant to be a bootable CD that demos the possibilities of openMosix; hard-disk installation isn't really supported. The CD runs like the normal Knoppix CD. The only difference is that it starts openMosix, the openMosixcollector (from openMosixview), and sshd.

8.11.1 What openMosix Is

The openMosix project (http://openmosix.sf.net) describes openMosix as:

A Linux kernel extension for single-system image clustering. This kernel extension turns a network of ordinary computers into a supercomputer for Linux applications.

The openMosix software makes it easy to create a supercomputer out of your spare machines for heavy number crunching, from brute-forcing encryption cracking to 3D rendering. Details about technologies behind computer clusters and openMosix clusters in particular are outside the scope of this hack. For a more thorough explanation, consult the openMosix project (http://openmosix.sf.net).

8.11.2 Set Up a Cluster

To set up a cluster, you need at least two systems networked to each other via a crossover cable or a switch. For small clusters, you can get away with 10 MB, but 100 MB is advised.

There are basically two ways to set up an instant cluster with Clusterknoppix—with a network boot or with multiple CDs. The easiest way is to use network boot (PXE or etherboot). PXE (Preboot eXecution Environment) downloads the bootstrapping code to load an operating system, the Linux kernel in our case, over the network. Most new computers support it look for "Booting from NIC" or PXE in the BIOS. If you don't have a PXE-capable network card, you can get etherboot to do almost the same thing by using a ROM on a floppy to download the bootstrapping code. More information about etherboot can be found at http://www.etherboot.org.

If your computer doesn't have a PXE or working Ethernet, you don't want to boot from the network, or you don't know which NICs (Network Interface Cards) are in your computers, simply put a Clusterknoppix CD in each computer.

8.11.3 The Network Boot Method

The first step to set up a cluster that boots from the network is to prepare the master node. With a master node running, you only need a single Clusterknoppix CD.

openMosix itself doesn't differentiate between master and client—every node is equal by default—but I use the terms master and client nodes to differentiate between the computer with the CD and the computers without it, respectively.

If you have a DHCP server on your network, you should receive an IP address once the master node boots from the CD; otherwise, you must configure the IP manually, as in:

knoppix@ttyp0[knoppix]$ su -

root@ttyp0[root]# ifconfig eth0 10.33.1.14 netmask 255.255.255.0

root@ttyp0[root]# route add default gw 10.33.1.254

root@ttyp0[root]# /etc/init.d/openmosix restart

Replace eth0 with your Ethernet card and the IP addresses with the values you want to use for your network.

If you have your own DHCP server on this network segment, you must shut it down for now, because the terminal server you are running starts its own (as mentioned in [Hack #25] ). Having two DHCP servers on the same network segment creates problems on the network, because you don't know which server answers a client's request for an address.

8.11.4 Configure openMosix

In X, click on K MenuKNOPPIXServicesStart KNOPPIX openMosix Terminal Server (see Figure 8-2).

Figure 8-2. Launching an openMosix terminal server

In the dialog that appears, select "setup (re)configure server and (re)start" and press OK.

Next, choose the NIC where your DHCP/TFTP/NFS server is listening and press OK. Now you can add the IP-address range you want to distribute to the client nodes. The default that the wizard uses should be sufficient for most applications.

Now you must select the correct network modules. You must select all the modules that your client nodes uses. For example, if you have three clients, each with a different NIC, select those three modules on this screen. They are probed automatically on each client node at boot time. A number of common NIC modules are already checked in this list, so if you aren't sure which module your NIC uses, try the default and skip ahead. Otherwise, you can try booting Knoppix directly on the machine and checking which module it uses for your NIC with the lsmod command run from a terminal.

Now select some miscellaneous options:

Secure

Disables the root access on client systems, which means that the default user knoppix won't be able to su to root.

Textmode

Boots the clients in text-mode, thus giving more memory for the migrating processes. Using textmode is the same as specifying 2 as an extra boot option (see [Hack #3]).

Masq

Sets the master node as a masquerading/forwarding server so that your clients can connect to the Internet/rest of the network.

DNS

Sets up a caching nameserver.

Squid

Sets up a web proxy/cache.

Finally, you get the option to specify any extra Knoppix cheat codes that you want to use on your clients. After the Knoppix cheat codes are chosen, the terminal server starts.

8.11.5 Boot Your Clients

The next step is to boot your clients from the network. This step will be very simple or require a bit of extra work, depending on whether your client supports PXE or requires Etherboot.

8.11.5.1 PXE

If you are using PXE, this step is easy. Select PXE in your BIOS and then boot the client. If you specified the correct network card driver in the terminal server setup, the client now boots ClusterKnoppix. You can pat yourself on the back now.

8.11.5.2 Etherboot

Etherboot takes a bit more work. First, go to http://www.rom-o-matic.net. Click on production and select your NIC. Next, click on Get ROM to download the ROM to your computer. Then put a formatted floppy in your floppy drive, go to a console, and type:

$ dd if=eb-5.2.4-yournic.zdsk of=/dev/fd0

Replace eb-5.2.4-yournic.zdsk with the path to your downloaded ROM image.

If you are not sure which NIC you have, you can download an image from http://drbl.nchc.org.tw/kernel/drbl_nic_detect.img, then open up a console and type:

$ dd if=drbl_nic_detect.img of=/dev/fd0

Boot the client from the floppy, and it will detect your NIC so that you know which ROM image to download.

Put the floppy disk with the corresponding ROM in the client. Make sure the BIOS is set to boot from the floppy first, and once the computer boots, the client is added to your cluster as a node.

8.11.6 The CD-ROM Boot Method

Put a ClusterKnoppix in every CD drive, and then let the computers boot. There you go: instant cluster. Unlike booting from the network, when booting each computer from the CD, openMosixview won't work out of the box, but processes are still migrating. After each node has booted, just run mosmon on the master node, and it should show a node for each computer. This ncurses tool displays the load, speed, and memory of openMosix nodes as a bar chart in a console.

openMosixview is the X counterpart of mosmon, a cluster-management GUI for monitoring and administrating the cluster. To use openMosixview, you must have key-based authentication over SSH working. Key-based authentication uses RSA or DSA encryption for special keys that allow you to log in to other SSH servers without entering a password by hand. This lets openMosixview communicate with all of the nodes securely, without requiring you to enter a password as it logs into each node.

If you use the CD-ROM boot method, you must complete a few extra steps before it works. Here is an example for two nodes. Node1 (IP address 10.33.1.14) is the node where openMosixview is running, and Node2 (IP address 10.33.1.6) represents a regular node on the network.

On Node1, open a shell, su to root, enter a password, and remember it:

knoppix@ttyp0[knoppix]$ su -

root@ttyp0[root]# passwd

Enter new UNIX password:

Retype new UNIX password:

passwd: password updated successfully

Now generate the keys. This DSA key has both a public and private counterpart. The public key exists on any remote machines this machine wants to connect to, and can be used to decrypt packets that this machine creates with the private key:

root@ttyp0[root]# ssh-keygen -t dsa -N "" -f /root/.ssh/id_dsa

Generating directory '/root/.ssh'.

Your identification has been saved in /root/.ssh/id_dsa.

Your public key has been saved in /root/.ssh/id_dsa.pub.

The key fingerprint is:

cb:b6:e2:6a:57:ca:c5:1d:42:4d:7a:83:9f:df:ff:22 root@Knoppix

root@ttyp0[root]# cp /root/.ssh/id_dsa.pub /root/.ssh/authorized_keys

root@ttyp0[root]# echo -n "* " >> /root/.ssh/known_hosts

On Node2, copy the ssh key from Node1, and place it in the list of authorized keys:

knoppix@ttyp0[knoppix]$ sudo mkdir /root/.ssh/

knoppix@ttyp0[knoppix]$ sudo scp root@10.33.1.14:/root/.ssh/

authorized_keys /root/.ssh/

The authenticity of host '10.33.1.14 (10.33.1.14)' can't be established.

RSA key fingerprint is e8:7b:56:eb:02:62:85:bb:93:c8:e7:49:0e:92:0b:f5.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.33.1.14' (RSA) to the list of known hosts.

Password:

authorized_keys                                 100%  668     0.7KB/s   00 00

Now go back to Node1, and confirm that the key has been copied correctly:

root@ttyp0[root]# ssh 10.33.1.6

The authenticity of host '10.33.1.6 (10.33.1.6)' can't be established.

RSA key fingerprint is 8a:80:b8:2b:fb:cd:a1:90:39:36:ca:03:57:6c:b0:5a.

Are you sure you want to continue connecting (yes/no)? yes



Warning: Permanently added '10.33.1.6' (RSA) to the list of known hosts.

root@0[root]# exit

root@ttyp0[root]# exit

Now you are ready to run openMosixview on Node1:

knoppix@ttyp0[knoppix]$ sudo openmosixview

You can now manage your nodes from the master node using openMosixview (see Figure 8-3).

Figure 8-3. openMosixview in action

8.11.7 Test the Cluster

There are a lot of tools to be found on the openMosix web site, from simple programs to stress test suites, but you can simply run the following script to test your cluster. This script creates a nested loop that counts from 0 to 10,000—10,000 times—and is an easy way to fully load your processor.

knoppix@ttyp0[knoppix]$ awk 'BEGIN {for(i=0; i<10000; 

i++) for(j=0; j<10000; j++);}' &

Run this script a number of times so that there are multiple processes for openMosix to distribute among the nodes, then run mosmon in another shell, and you should see the load being balanced between each node in your cluster.

8.11.8 See Also

• The official ClusterKnoppix page at http://bofh.be/clusterknoppix.

• The official openMosix page at http://openmosix.sf.net.

—Wim Vandersmissen