Set Up an MPICH2 Cluster With Ubuntu Server 11.04

This is a short note of setting up MPI cluster with Ubuntu system. Assume four machines named ub0-3, with ub0 being the master node. They are connected through private local network. At the time of writing this note, the operating system of these machines is Ubuntu Server 11.04.

1. Define hostnames in /etc/hosts

Edit /etc/hosts like this (change IP address accordingly):

127.0.0.1 localhost

192.168.133.100 ub0

192.168.133.101 ub1

192.168.133.102 ub2

192.168.133.103 ub3



2. Share files through NFS

2.1. Install NFS server on the master node:

user@ub0:~$ sudo apt-get install nfs-kernel-server

2.2. Make a directory to be exported:

user@ub0:~$ mkdir /mirror

2.3. Edit file /etc/exports by adding a line like this:

/mirror ub1_hostname(rw,sync,no_subtree_check) ub2_hostname(…) …

2.4. Restart the NFS service:

user@ub0:~$ sudo service portmap restart

user@ub0:~$ sudo service nfs-kernel-server restart

2.5. Configure firewall to open two ports 111 and 2049 if necessary:

user@ub0:~$ sudo iptables -I INPUT num -p tcp --dport 111 –s ub1_hostname --j ACCEPT

user@ub0:~$ sudo iptables -I INPUT num -p tcp --dport 2049 --s ub1_hostname --j ACCEPT

…

2.6. Mount the directory on client machines:

user@ub1:~$ sudo apt-get install nfs-common

user@ub1:~$ sudo mount ub0:/mirror /mirror

…

3. Set up an user account for running MPI

3.1. Create a user with same name and same userid on all nodes with a home directory in /mirror:

user@ub0:~$ sudo useradd -d /mirror/mpiuser -m -s /bin/bash mpiuser

user@ub0:~$ sudo passwd mpiuser

… #repeat the same process on other nodes (same password on all the nodes is not necessary)

3.2. Change the owner of /mirror to mpiuser:

user@ub0:~$ sudo chown mpiuser /mirror

…

4. Install SSH server on all nodes if SSH not available

user@ub0:~$ sudo apt--get install openssh--server

…

5. Setup SSH with no pass phrase for communication between nodes

5.1. Log in with the new user on the master node:

user@ub0:~$ su mpiuser

5.2. Generate ssh key:

mpiuser@ub0:~$ ssh-keygen -t dsa # Leave passphrase empty. A folder called .ssh will be created in the home directory. This folder will contain a file id_dsa.pub that contains the public key.
mpiuser@ub0:~$ ssh-keygen -t rsa # Leave passphrase empty. A folder called .ssh will be created in the home directory if not exist previously. This folder will have a file id_rsa.pub which contains the public key and a file id_rsa which contains the private identification.

(Update on 11/05/2014. SSH can use either "RSA" (Rivest-Shamir-Adleman) or "DSA" ("Digital Signature Algorithm") keys. Both of these were considered state-of-the-art algorithms when SSH was invented, but DSA has come to be seen as less secure in recent years. RSA is the only recommended choice for new keys. Please see ref 3)

5.3. If, as in the present note, the home directory is shared by NFS between master and slave nodes for user mpiuser, run the following command on the master node so as to copy public key to ~/.ssh/authorized_keys and then go to step 5.5: (Updated on 24 August 2012)

mpiuser@ub0:~$ cat ~/.ssh/id_dsa.pub >>~/.ssh/authorized_keys
mpiuser@ub0:~$ cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys

#please note the file ~/.ssh/authorized_keys should be in mode '644'

5.4. If the home directory is not shared by NFS between master and slave nodes for user mpiuser, transfer the public key file id_dsa.pub id_rsa.pub to slave nodes. On the slave nodes, log in as mpiuser, and then add the key generated from master node to file ~/.ssh/authorized_keys:

mpiuser@ub1:~$ cat id_dsa.pub >>~/.ssh/authorized_keys
mpiuser@ub1:~$ cat id_rsa.pub >>~/.ssh/authorized_keys
…
#please note the file ~/.ssh/authorized_keys should be in mode '644'


5.5. To test passwordless SSH login, run this command on the master node:

mpiuser@ub0:~$ ssh ub1 hostname

#It should return remote hostname without asking for passphrase.

(Update 2: to resolve file permission issue, ensure the user of accessing NFS shared files have same UID and GID on both server and client. Edit /etc/passwd and /etc/group when necessary)

6. Install GCC and other compilers

mpiuser@ub0:~$ sudo apt-get install build-essential

…

7. Install MPICH2

7.1. Install through package or using Synaptic:

mpiuser@ub0:~$ sudo apt-get install mpich2

7.2. Test installation by running:

mpiuser@ub0:~$ which mpiexec

mpiuser@ub0:~$ which mpirun

#If successful, paths to the two executables should be return

7.3. Set up Hydra Process Manager (which replaces MPD) on master node:

7.3.1. Create a file with the names of the nodes

mpiuser@ub0:~$ cat hosts

ub1:8 #8 is the number of cpu cores available

ub2:8

ub3:8

ub0:7

7.3.2. The hosts file can also be designated by setting variable HYDRA_HOST_FILE:

mpiuser@ub0:~$ echo HYDRA_HOST_FILE=/etc/hydra/hosts >> ~/.bashrc
#Note: do not set this environment variable HYDRA_HOST_FILE if the MPI environment is to be controlled by a cluster queueing system like SGE. (Updated on 17 May 2012)


7.4. Test parallel computation

7.4.1. Compile an application with mpicc:

mpiuser@ub0:~$ mpicc cpi.c -o cpi

#cpi.c is an example provided by the MPICH2 library

#On my machine, running mpicc got an error:

#/usr/bin/ld: cannot find –lcr

#collect2: ld returned 1 exit status

#This problem is due to missing package ‘libcr-dev’

#Install libcr-dev: mpiuser@ub0:~$ sudo apt-get install libcr-dev

7.4.2. Run the application using mpiexec:

mpiuser@ub0:~$ mpiexec -np 1 ./cpi #using single core

mpiuser@ub0:~$ mpiexec -np 16 ./cpi #using 16 cores

#Note the path to the executable (‘./’ here) should be specified, otherwise an error produced: HYDU_create_process (./utils/launch/launch.c:69): execvp error on file cpi (No such file or directory) (Updated on 17 October 2011)

#If hydra is not the default process manager for MPICH2, running mpi program with mpiexec may cause an error that all processes have rank 0. To overcome this error, use mpiexec.hydra instead of mpiexec (Updated on 17 October 2011)

#Configure firewall if connection is blocked, eg,

#mpiuser@ub0:~$ sudo iptables -I INPUT 1 -s ub1 -j ACCEPT

(Update 1: I've posted a relevant topic "Install Sun Grid Engine (SGE) on Ubuntu Server 11.04")

References
1. Setting Up an MPICH2 Cluster in Ubuntu (https://help.ubuntu.com/community/MpichCluster)
2. MPICH2 Developer Decumentation (http://wiki.mcs.anl.gov/mpich2/index.php/Developer_Documentation)
3. https://help.ubuntu.com/community/SSH/OpenSSH/Keys