High Performance Computing - 2

We now take advantage of the computing cluster to study the phase space of our projectiles in the Earth-Sun system. Starting with your code for calculating the trajectory of an object that has been launched at the Sum, you will adapt that code for running on the cluster and then map out the dependence of the final distance from the center of mass (CM) after a suitable time on the initial angle and speed of the projectile (see the figure below).

To map the (Subscript[v, 0])-ϕ dependence of the distance of the projectile from the CM of the Earth-Sun system requires calculating many trajectories. To visualize the results we assign a color to each point in the (Subscript[v, 0])-ϕ plane related to the distance from the CM. The color scheme will be the following.

Blue points - (Subscript[d, c]) = -2; hit the ‘ Earth’
Black Point - (Subscript[d, c]) = -1; hit the Sun.
Green → Red - (Subscript[d, c]) = calculated value; red is far from the CM,  green is close.

Before getting started we consider a ‘simple’ example to learn the skills to take advantage of the computing power of the cluster. We will routinely refer to the previous laboratory entitled High Performance Computing -1 as HPC-1.

1. An Example for the Development Cycle.

Consider the following Monte Carlo code to calculate the value of π. One picks two different numbers at random in the range from zero to one to represent a point in a unit square with the lower, left corner at the origin. The Pythagorean theorem can be used to determine the distance of these two points from the origin. If that distance is less than one, then this ‘event’ is counted as a hit. If the distance exceeds one, then we ignore the event and go on to a new pair of random numbers. This process is repeated many, many times. The total number of pairs of random numbers is the number of throws. Taking  the ratio of the number of hits to the number of throws and multiplying by four gives one an estimate of π.

1. Copy the code below into Mathematica running on one of the Windows-based physics lab computers. Run it to make sure it behaves correctly.

(* starting parameters for Monte Carlo calculation of π. *)
nthrows = 10000;
radius=1.0;
nhits=0;
(* Monte Carlo loop. *)
Do[x0=RandomReal[{0,radius}];
y0=RandomReal[{0,radius}];
r0=(Sqrt[(x0^2) + (y0^2)]);
If[r0<=radius, nhits++],
{i,1,nthrows}
]; (* end of loop over throws. *)
(* get statistical uncertainties. *)
dnhits = (Sqrt[nhits]);
myPi=4*(nhits/nthrows);
dmyPi=4*(dnhits/nthrows);
ratio = (myPi/π);
dratio=(dmyPi/π);
results={nhits,nthrows};
Print["myPi=",N[myPi]," ± ",N[dmyPi],"; Ratio =",N[ratio]," ± ",N[dratio]]

2.  Save your results by adding the command Export["myMathOutput.dat",{results}]; to the end of your code. Run your code again and check that your results have been properly written into the output file which will most likely appear in your Documents area (on the Windows-based, Physics lab machines.

2. Preparing a Local File for Parallel Running

In this laboratory you will develop your calculation on the Windows-based, physics lab machines first. Once you are satisfied with your code follow the steps below to prepare the myMath.m file for batch submission to the cluster.

1. To prepare your Mathematica notebook for batch running you first have to collect the output from the calculation. We will do that here in a file we will call test.out. Starting with your code on the local Windows-based machine add the command Put["test.out"] at the top of the Mathematica cell. Add >>>"test.out" after the last command. Then add Quit[] at the very end of the cell to halt Mathematica. Run your final code interactively on the local machine to check it. You should see no output on your screen and the file test1.out should be created in your home directory.

2. Click on the cell bracket for your code and in the Mathematica menu  and then in the Mathematica menu go to Cell→Cell Properties and select Initialization Cell.

3. Save your file in the Mathematica package format by clicking on Save As in the file menu and then under Files of type: select Mathematica package (*.m). Use a unique filename so you can identify this project from others.

4. Upload the file to the cluster using the WinSCP file transfer GU or similar toolI. Put the file in your compPhys/run directory on the cluster.

3. Test Your Code.

Once you have a working code uploaded to the cluster, it is useful to test it before submitting it to the full cluster.

1.  Open a terminal window and connect to the cluster (see Part 1 of HPC-1). Navigate to the compPhys/run directory. 

2. Copy the Mathematica package file you made in the previous part to myMath.m (see Part 2 of HPC-1).  This step will replace the previous contests of myMath.m. If you want to save that file then copy it into another file with a unique name first.

3. You can now test to see if your code runs without error using the same command that will be used on the remote nodes. You might want to reduce the number of trials or events in your Monte Carlo to finish this test quickly. Use the file transfer GUI to edit your myMath.m file. Once you are ready enter the following command in the terminal window from the compPhys/run directory on the cluster. Note that you have to enter your own netID in the path.

/usr/local/bin/math -noprompt -pwfile /home/<your netID>/compPhys/run/math-license -run ‘<<myMath.m’

The anatomy of the command follows.

/usr/local/bin/math		Run Mathematica.
-noprompt			Mathematica option to suppress printing the banner or In/Out prompts 
				(useful for batch input)
-run ‘<<myMath.m’		Mathematica option to read in and execute the commands in the file.	
-pwfile /home/<your netID>/compPhys/run/math-license	Mathematica option to read the license from a file.	

4. Check to see your output looks reasonable.

4. Submitting Parallel Jobs to the Cluster.

To start the job running on the cluster open up a terminal window (see Part 1 of HPC-1) and navigate to the compPhys/run directory. Follow the procedure below.

1. The simplest way to submit your job to the cluster is to copy your Mathematica package file that you uploaded to your compPhys/run directory to myMath.m (see Part 2 above and Part 2 of HPC-1). This is the file that the runMath.sh script will use to run the batch job on one of the remote nodes. If you want to keep your original Mathematica package name you will have to modify the file runMath.sh.template.

2. Run the submit.pl script in your compPhys/run directory on the cluster from the command line using submit.pl <your NetID>. You’re off!

3. Monitor your batch jobs until they are all done (see Part 4 of HPC-1 for ways to do this). When you are finished, navigate to your compPhys/run/logs directory. You should see  files with names like log14, log37, etc. These files contain system information about each job submitted by the submit.pl script. Check a couple of files to see that things are alright (see the more command in Part 2 of HPC-1).

4. Navigate to your compPhys/run/results directory on the cluster. This is where the output from each batch job is stored and you should see files with names like myMath08.dat, myMath41.dat, etc. These files contain the results of the calculation. Check a couple of the files to see that things are alright (see the more command in Part 2 of HPC-1). These files have the results for the calculations on the individual nodes and we want to add these together to get a final result. The Unix cat command (short for concatenate not a reference to the mammal) can be used for this (see also Part 2 of HPC-1).

cat myMathOutput*.dat > total.dat

This command creates a new file total.dat that will have all the results in a single file. 

5. Download the file total.dat to your local machine and analyze this file to produce a final estimate of π.

5. Now do yours.

Now do  steps 1 - 5 for your code. Use the command in Part 6 of this laboratory for plotting.

6. A Plotting Command.

Here we re-introduce the plotting command first encountered in the laboratory entitled Shooting the Sun. Once you have obtained the data to plot the grid use the following command to make that plot. The assumptions that go into the code below are that (1) the input file is made of rows with three columns consisting of initial ϕ and (Subscript[v, 0]) and the final distance from the CM of the Earth-Moon system, (2) an entry in column three of ‘-2’ means the projectile hit the Earth, and (3) an entry of ‘-1’ means the projectile struck the Sun. A sample file that can be used to demonstrate the command is available on the course webpage.

(* Read in the data (phi, v0, dC) and measure some properties for the plot. *)
t1 = ReadList["/media/flash/215s12/examples/3body/threeBodyTest.dat",{Number,Number,Number}];
myNmax=Length[t1];
mymin=1;
mymin=Min[Table[t1[[i,3]],{i,1,myNmax}]];
mymax=Max[Table[t1[[i,3]],{i,1,myNmax}]];
range=mymax-mymin;
(* Make the final table with the color code for the final distance as the last element. *)
t2=Table[
{t1[[i,1]],
t1[[i,2]],
If[t1[[i,3]]<0,If[t1[[i,3]]==-1,0.9,0.7],0.25*(1-((t1[[i,3]]-mymin)/range))]},{i,1,myNmax}
];

The code below defines a new function ThreeBodyScatterPlot used to plot the results of the calculation and shows its usage.

(* Define the plotting function. *)
ThreeBodyScatterPlot[data_/;
   MatrixQ[data, NumberQ] &&
   Length[First[data]] === 3,fun_, opts___] :=
   Show[Graphics[
          {PointSize[0.005],
           Map[{fun[Last[#]],Point[Take[#, 2]]}&, data]
          }, opts
        ]
 ]

(* Make the plot. We also keep track of the time for future reference. *)
start=AbsoluteTime[];
p5=ThreeBodyScatterPlot[t2,
              Hue[#,1,If[#==0.9,0,1]]&,      
              Axes->True,
              AxesLabel->{"x","y","z"},
              AspectRatio->1,
              Frame->True,
              PlotRange->{ {0,360},
                           {0.7,1.3}},
              BaseStyle->FontSize->16,
              ImageSize->8*72,
              Frame->True,
              FrameLabel->{"phi (deg)","v0/vescape"}
             ]
stop=AbsoluteTime[];
Print["Elapsed time =",stop-start," s."]