High Performance Computing - 1

Many problems in physics and other fields like biology, medicine, etc can be solved using computational methods, but require prodigious computing resources. In this laboratory you will be introduced to the use of a high-performance computing (HPC) cluster. You will be solving the problem developed in the previous laboratory (Shooting the Sun) using a Monte Carlo method to explore the range of parameters for successfully launching a projectile into the Sun. The HPC cluster is a research-quality, linux-based system consisting a head node and an array of other computers that you can access to run calculations in parallel and dramatically increase the speed of your computations. This notebook will guide you in the use of this system.

1. Getting to the HPC cluster.

The HPC cluster consists of a head node (gluon.richmond.edu) and seven remote nodes with eight cores on each remote node. You will be able to run a total of 56 parallel programs on the remote nodes.To access the cluster from one of the physics, Windows-based, laboratory machines start the WinSCP application that can be found under the start menu.You should see a graphical user interface (GUI) like the one shown below. 


Figure 1. WinSCP startup GUI.

In the Host name field enter the IP address of the head node: gluon.richmond.edu. Under User name enter your Richmond NetID and under Password enter your Richmond NetID password. Click Login. You should see a new GUI with your local files on the left-hand side and the your files in the top directory of gluon on the right-hand side. See Figure 2.   

  
Figure 2. WinSCP file transfer GUI.

To copy files from one computer to another you simply have to click and drag between the two sides of the GUI. To edit a file just double-click on the file and a new GUI appears that is a text editor with the desired file in it. As an example, navigate to the compPhys/run folder by clicking on the appropriate folders and then double click on the file myMath.m. You should see a new GUI appear something like the next figure.


Figure 3. Text editor GUI in WinSCP showing myMath.m.

This is a special Mathematica input file that will be used later to run parallel computations on the cluster. Note the comments in the file (non-executable code) defined by the combinations ‘(*’ and ‘*)’. Explore the use of the editor by adding a comment to the file and then saving it. Each of the menu items at the top of the GUI has context help (putting the cursor over the button will make a message appear describing the function of the button). Saving the file (leftmost button) will automatically upload the modified file to the cluster head node.

Sometimes you will need to access the cluster via a terminal window to enter commands. Go to the WinSCP file transfer GUI and click on the  icon. A  new GUI will appear like the one shown below.
         


Figure 4. Terminal window in WinSCP.

Enter your Richmond NetID password and you will be logged into the cluster. You can now enter commands. See Part 2 for a description of Unix commands.

2. Unix Commands

Once you have opened a terminal window to the cluster you should explore your directories. In UNIX (and Linux) files are arranged in directories (‘folders’ in Windows or MacOS terminology). Some commands allowing a user to manipulate files are listed below. Your first task will be to copy your files into a separate directory for safekeeping. Create a directory in your home area called save (using the mkdir command). Navigate to your compPhys directory (using the cd command described below). Copy all the files there into your save directory (using the cp command). By doing this you can recover from possible mistakes later.

pwd		displays the name of the current directory (‘.’).

ls		lists the files in the current directory.

ls dir 		lists the files in the directory ‘dir’.

ls -lF		a useful extension of ls providing additional information about the files.

cp file1 file2	copies file1 to file2. To copy from one directory to another cp dir1/file1 dir2/file2.

cd dir1		go to specified directory that is located in the current working directory.

cd .. 		go to the directory above the current directory.

cd ../foo 	Go up one level, then down into directory foo. The current directory 
		is always denoted by ‘.’.

mv file1 file2	rename file1 as file2, within the same directory (changing the name of the file), or 
		move file1 into some other directory.

more file	display file onto standard output (screen), one screenful at a time.

cat file		concatenate files and print on the standard output.

mkdir dir	create a new subdirectory called dir in your current directory.

>		redirect output to a file; e.g. cat file1 > file2 copies file file1 into file2.

>>		redirect output to a file; e.g. cat file1 >> file2 appends file1 onto file2.

|		‘pipe’ standard output to standard input e.g. cmd1 | cmd2 sends the output of 
		command cmd1 into the input of cmd2.

rm file		delete a file.

man <command>	Prints the manual page for <command> to the screen.

UNIX also supports ‘wildcards.’ A ‘?’ will be replaced by any matching single character, while a ‘*’ will be replaced by any number of matching characters. For example, ls abc*.c will list all files whose name starts with ‘abc’ and ends in ‘.c’. 

More information on unix/linux commands can be found by clicking on the links below. It is worth noting here that there are a variety of ‘shells’ which are the programs that interpret your commands, i.e. all the commands listed above. We are using tcsh (pronounced tee-shell).

3. Using the Cluster

In this section we describe the process of running many parallel jobs on the cluster. The likely development cycle we will follow will start with you developing your Mathematica calculation on the physics lab machines. Once you are satisfied with the calculation, then you save your Mathematica notebook in a special way(see Part 4) and transfer it to the cluster head node (using the WinSCP file transfer GUI described above). From there you will submit your job ‘in batch’ to one of the remote nodes. A batch job is one that is running on a computer, but requires no input from the user. It is a self-contained program that starts, runs, generates output, and stops all on its own.

Navigate to the compPhys/run directory and list the files there. Three of the files in that directory are essential for running computing jobs on the cluster. You can view the files with your editor, but don’t save any changes unless you know what you are doing.

myMath.m		This is a special format output file from Mathematica that contains the code
			for your calculation. It will be used to run jobs on the remote nodes. The
			code you develop will be saved in a similar file and used for the parallel 
			computations. The easiest way to use the files in this directory may be to 
			rename the Mathematica file you developed on the physics lab machines 
			and uploaded to this directory to myMath.m.
			
submit.pl		Text file containing commands to submit jobs on the nodes of the cluster. 
			You may need to edit this file to change the parameters of the job submission
			like the number of parallel jobs to submit. More details on using the file
			are at the top of the file itself. You can view it with the WinSCP editor.

runMath.sh.template	This file contains the commands  needed to run Mathematica in batch on a
			particular remote node using myMath.m as the input. You may need to
			edit this file if you change how you handle input files, outputs, etc. More
			details are at the top of the file itself and can be viewed with the WinSCP
			editor.

4. Cluster Commands

There is a set of commands for interacting with the cluster and getting information from individual remote nodes. Those commands are listed below. We will study them in more detail in the next example.

qstat			Lists your submissions with jobID and CPU time.		
qdel <jobID>		Stop one of your jobs.
qstat -u			Lists everyone’s job submissions.
qstat -f -j <jobID>	Prints out additional information about that job.
act_exec		Execute a shell command on a remote node. For example, entering the following
			command
			
			 act_exec --nodes=node01 ls /tmp
			 
	  	           on the head node will list the files in the /tmp directory on node01. See the man
	  	            page using man act_exec.
act_cp		          Used to copy files to the nodes in the cluster. See also man act_cp.

More detailed information on the commands special to using the cluster can be found by clciking on the links below.

5. An Example

Now run a sample calculation to demonstrate the procedure for running parallel computations on the cluster. The code in the file myMath.m in your compPhys/run directory  was derived from the Mathematica code shown below. It generates two random number and uses the acceptance-rejection method to a get set of random numbers that follow a sinusoidal distribution over the range 0-π. The results are placed in an output file myMathOutput.dat

trials=100;
table={};
Do[x1=RandomReal[{0,π}];y1=RandomReal[{0,1}];y2=Sin[x1];
If[y1 < y2,AppendTo[table,x1]],
{i,1,trials}];
Export["myMathOutput.dat",table];

1. Run the code above on your local machine to get an idea of what your sample calculation will be doing. Read the output file into Mathematica (use the ReadList command) and make a histogram of the results (use the Histogram command). Check that your results are sensible. 

2. Compare your code (on your local machine) with the file myMath.m in your compPhys directory on the cluster. It should look familiar to you.

3. On the cluster navigate to your compPhys/run directory. You need to create two, new subdirectories for storing the results of the calculations you will perform. Find the Unix command in Part 2 for creating a subdirectory and create two subdirectories in your compPhys/run area named logs and results.

4. When you are ready enter submit.pl <your NetID> in your compPhys/run directory. You will see lots of messages stream by as each job is submitted.

5. Monitor what is happening to your jobs. Enter qstat on the cluster and you will see a listing of your jobs and how much time they have used (see Part 4).  In a web browser go to the following site

http://gluon.richmond.edu/ganglia/

and you can see the load on all the nodes in the cluster. Try some of the commands listed in Part 4 to see what is happening as your jobs progress.

6. When your jobs are complete (no output from the qstat command), navigate to the compPhys/run/logs directory. This directory contains the output from each individual job in a file tagged with an index. View one of the files (see the more command in Part 2). If you have questions, ask your instructor.

7. Navigate to the compPhys/run/results directory. This area contains the results of your calculations tagged with an index number. View one of the files to make sure it looks correct. You can add all these files together using the cat command (see Part 2).

cat myMathOutput*.dat > total.dat

Copy the file total.dat back to your local machine using the WinSCP file transfer GUI and generate a new histogram of your results. Compare it with your previous histogram. If there is not a vast increase in the number of events, alert your instructor.

8. Go back to the compPhys/run directory. Use the WinSCP editor to open and view the myMath.m file. Find and increase the number of Monte Carlo events or throws. Save the file. Redo steps 4-7 above. If you do not see an increase in the number of events, alert your instructor.