Parallel Programming using Message Passing Interface (MPI)

(Part of HPC @ IU Workshop, Indiana University)

  Working on the AVIDD Cluster


  Message Passing Interface (MPI) - Simple Programming Examples


  MPI I/O - Parallel File Access


  Getting Ready for Graphical Applications from Compute Nodes


  Profiling Parallel Code using Intel Trace Analyzer/Collector (ITA/C)


  Debugging Parallel Code using TotalView


  Using MPICH (instead of LAM) to run MPI code (Appendix)


  More Documentation!

MPI - Introduction

Introduction

What will I learn in this tutorial?

You will learn about scientific parallel programming on the AVIDD Linux Cluster at Indiana University.
Previous


HPC @ IU Tutorial - Initial Setup

Initial Setup

Workshop training account on AVIDD

SSH onto AVIDD

Grab Interactive Compute Nodes using qsub -I

  • We will be using interactive nodes through out this workshop to run MPI jobs. Request two compute nodes (2 processors on each node) for 6 hours from PBS.
    Exercise: Do this! 
     [agopu@bh2 agopu]$ qsub -I -l nodes=2:ppn=2 -l walltime=6:00:00
    You can expect to see a message like this: 
     qsub: waiting for job 281988.aviss.avidd.iu.edu to start
 qsub: job 281988.aviss.avidd.iu.edu ready

 [agopu@bc56 agopu]$
    If you want to know what nodes have been allocated to your job, then you can find that information in a file pointed to, by $PBS_NODEFILE. 
     [agopu@bc81 agopu]$ cat $PBS_NODEFILE
 bc81
 bc53
 bc81
 bc53
    Note: Later in the workshop when you run example MPI code, you'll realize that each process' rank is determined by the order of the nodes in $PBS_NODEFILE. We will return to this later in the session.


HPC @ IU - Elementary MPI Programming Example

My First MPI Program

MPI API for SPMD programs

  • MPI stands for "message passing interface." API stands for "application programming interface."

  • MPI lets you write one single program (SP) that all processors execute together.

  • It also equips you with simple interprocessor communication functions for sharing multiple data (MD) among processors.

The "Hello, parallel worlds!" program


HPC @ IU Tutorial - Makefile

Tutorial Makefile

Makefile to compile your example programs

Couple of important points:

  • Make needs a tab, not spaces! If indeed you decided to build your own set of example programs and Makefiles, then after you copy and paste the above Makefile, you must change the spaces into tabs in the twelve or so indented lines. In your text editor backspace each of those lines to the left margin, then hit the tab key once. The make command requires that these command lines begin with exactly one tab; see the above listed tutorial or other documentation for more information.

  • Which compiler to use? If you are going to use the example C source codes from this tutorial, then you do not need to modify this makefile. If you code in C++ or Fortran 77 or Fortran 90, then you must change the CC variable in the following makefile to mpiCC, mpif77 or mpif90 respectively.

  • What is LDFLAGS defined for? In the latter half of this tutorial, you'll be using Intel Trace Analyzer to profile your parallel code. The libraries listed in LDFLAGS (-lVT and its dependencies) in the above Makefile cause profiling files to generated when you run your parallel code.



HPC @ IU - Basic Concepts in Parallel Programming

Basic Concepts in Parallel Programming

In this section, we will briefly look at parallel programming paradigms and the two popular types of architectures used by parallel computers.


Parallel Programming Paradigms

There are different parallel programming paradigms (the names are self-explanatory):

  • Data Parallel Paradigm

  • Message Passing Paradigm

  • Shared Memory Paradigm

  • Mixed Parallel Programming Paradigm

We will use a mixed parallel programming paradigm -- a combination of data parallel and message passing paradigms. We'll refer to it as "Single Program Multiple Data" or SPMD model in this tutorial. The reason it is called so:

  • Single Program (SP)

    • All processors will execute the same program, though they could be executing different sections of the code based on certain criteria. This will be explained in detail in later sections.

  • Multiple Data (MD)

    • Data will be decomposed into chunks, with each processor getting one or more of those chunks.

If you are interested in learning more about the paradigms mentioned above, Gustav Meglicki has some excellent notes explaining parallel programming paradigms in the I590 section of his website.


Hardware Architectures used by Parallel Computers

Distributed memory systems

Shared memory systems


Once again...

Check out "Low-down on AVIDD - Technical Details" subsection in the Primer to working on the AVIDD Linux Cluster guide, if you have not done so yet.




HPC @ IU - RoundRobin: Simple Message Passing Illustrated

RoundRobin - Simple Message Passing Illustrated

The roundRobin program

  • This is our second parallel program and it illustrates basic interprocessor communication.

  • Processor 0 composes a message (in this case, an integer array) and sends it to processor 1; Processor 1 receives the message and sends it on to 2; Processor 2 receives it and sends it to 3, and so forth. Finally the last processor sends it back to 0 who then compares it with the original message to check if the message passing worked correctly.


HPC @ IU - Simple MPI functions explained

The RoundRobin program -- Explanation of MPI function calls

Recall from a previous section...

  • In a distributed-memory machine each processor has access only to its own memory. There is no common or shared memory that processors can access.

  • Therefore, processors can interact only by explicitly transmitting packages of data between each other. In the roundRobin program this interaction was accomplised using MPI function calls MPI_Send () , MPI_Recv () and MPI_Bcast () ; these functions are explained below.

Simple MPI Sends/Receives/Broadcasts

  • Before a processor can send information, it has to pack the relevant data into a "send buffer" located in its own memory.

  • Similary, when a processor receives data from another process, it has to unpack the data sitting in its own "receive buffer".

  • In C the MPI function to send data is: 

     int MPI_Send( void*        sendBuf,
	       int          count,
	       MPI_Datatype datatype,
	       int          destinationRank,
	       int          tag,
	       MPI_Comm     comm )

    • sendBuf is the send buffer or array packed with the data to be sent.

    • count is the length of the sendBuf array.

    • datatype is the datatype of the sendBuf array, typically MPI_INT, MPI_FLOAT , or MPI_DOUBLE.

    • destinationRank is the rank of the intended receiving processor.

    • tag is an identifier that may be used to distinguish among multiple transmissions between the same two processors.

    • comm is the communication domain. In this workshop, we will always specify the default communicator MPI_COMM_WORLD which consists of all processors.

      Note: See Gustav Meglicki's Parallel Diffusion example if you are interested in learning about constructing your own communicator.

  • Be aware that a processor executing an MPI_Send will "block", i.e. it will not proceed executing subsequent code until the send is completed.

  • The MPI receive function in C looks as shown in the roundRobin program is: 

     int MPI_Recv( void*        recvBuf,
	       int          count,
	       MPI_Datatype datatype,
	       int          sourceRank,
	       int          tag,
	       MPI_Comm     comm,
	       MPI_Status   *status )

    • recvBuf is the receive buffer or array into which the MPI API will place the transmitted data.

    • count is the length of the recvBuf array.

    • datatype is the datatype of the recvBuf array, typically MPI_INT, MPI_FLOAT , or MPI_DOUBLE .

    • sourceRank is the rank of the sending processor.

    • tag is an identifier that may be used to distinguish among multiple transmissions between the same two processors.

    • comm is the communication domain.

    • status is a pointer to information about the state of the receive. In this workshop, we will not make use of this status information.

  • Similar to what was explained in the case of MPI_Send () , a processor executing an MPI_Recv will "block", i.e. it will not proceed executing subsequent code, until the receive is completed.

  • And finally, if a process is expected to send some information to all the other processes, then we can use the MPI_Bcast () command. 

     int MPI_Bcast ( void* buffer,
                 int count,
                 MPI_Datatype datatype,
                 int root,
                 MPI_Comm comm )

    The parameters to the above MPI function are similar to the ones in MPI_Send () and MPI_Recv () functions.

  • MPI_Comm_Size () tells the participating processes about the total number of processes in this communicator. This is also something they need to know. For example, if the program is to work on a very long array, the processes will have to know how many of them there are in the pool in order to work out for themselves which portion of the array they should work on.

  • Misc: MPI has a function called MPI_Wtime () that can be used to time your program (or sections of it). It measures wall-clock time down to a microsecond, very accurate indeed. Of course using a tool like Intel Trace Analyzer would give you an even better idea about how your program is performing.



HPC @ IU - Data Decomposition

Data Decomposition

What is Data Decomposition?

  • The roundRobin program was meant only to illustrate basic interprocessor communication, so we did not really focus on data. Processor 0 generated the data. Then each processor received and sent the data back and forth.

  • But in real-world programming, it's never that simple! In most cases, data will have to be shared among the processors so that each cares about only its own portion of the data. "Data decomposition" refers to the sharing data between parallel processes in the best possible (i.e. most efficient) fashion. No process should step on another process' data either.

Parallelism (vs) InterProcessor Communication (IPC) Overhead

It is very important to figure out how suitable an algorithm/piece of code is to parallelization; you should guard against a tendency to parallelize code without enough consideration to the InterProcessor Communication, which is very expensive on distributed memory parallel systems. In general, the following guidelines are recommended:

  • More parallelism is good; maximum amount of CPU time must be spent by each processor on computation (and not message passing)

  • Least interprocessor communication is desirable.

  • Objective: How can we decompose data in our SPMD model in order to maximize parallelization while minimizing interprocessor communication?

Thinking of Parallelization? First write the serial program...

In most cases, when you wish to parallelize any algorithm, then it might be worthwhile to start with the serial version of the code. Then you could follow the steps listed below:




HPC @ IU - Computing PI by Elementary Numerical Integration

A Naturally Parallelizable Computation

Problem statement

  • In this example we would like to compute PI using the following integration:

  • Consider the figure below. To do the integration, we'll divide the domain [0,1] into intervals, compute the integral at the mid point of each interval, then multiply those values by the width of each rectangle and finally add up the areas of the all the rectangles to get an approximate value of PI.

  • In the following sections, we'll show how you could write serial code to perform the above integration and then how you could go about parallelizing the code.


HPC @ IU - Computing PI using Elementary Numerical Integration: Serial version

Computing PI using Serial Code

Program serialPi

  • The serial version of our program to compute PI, written in C, is displayed below: 

    #include <stdio.h>
#include <stdlib.h>
#include <math.h>

const double pi = 3.1415926535897932385;

int main( int argc, char *argv[] ) {
  /*  Arguments required for executing serialPi:     */
  /*  0                         -> serialPi          */
  /*  1                         -> numberOfIntervals */

  int numberOfIntervals, interval;
  double intervalWidth, intervalMidPoint, area = 0.0;

  /*  The number of intervals is a command line argument. */
  numberOfIntervals = atoi( argv[1] );

  /*  Compute the interval width. */
  intervalWidth = 1.0 / numberOfIntervals;

  /*  Now compute the area. */
  for ( interval = 0; interval < numberOfIntervals; interval++ ) {
      intervalMidPoint = (interval + 0.5) * intervalWidth;
      area += 4.0 / ( 1.0 + intervalMidPoint*intervalMidPoint );
  }
  area *= intervalWidth;

  /*  Print the results. */
  printf( "The computed value of the integral is %.15f\n", area );
  printf( "The exact value of the integral is    %.15f\n", pi );

  return 0;
}

    Note: If you did not copy example programs from HPC's account in the initial section of this workshop, then copy the above source code as serialPi.c to your SerialPi directory.

  • Exercise: Do this!

    To compile the serialPi program, at your shell prompt in the SerialPi directory, do: 

     [agopu@bh2 agopu]$ cd ~/MPI_Tutorial/SerialPi/
 [agopu@bh2 SerialPi]$ icc -o serialPi serialPi.c

    Then to run the program on interactive nodes you got through qsub -I (bcXX), do: 

     [agopu@bc81 agopu]$ cd ~/MPI_Tutorial/SerialPi/
 [agopu@bc81 SerialPi]$ ./serialPi 20000
  The computed value of the integral is 3.141592654423135
  The exact value of the integral is    3.141592653589793

    Note: Since this is not a MPI program, we've not used the Makefile. We could have if we'd wanted to, but we would have needed to change the compiler to icc or gcc.

  • Since the command line argument used above is 20000, the domain would have been divided into 20,000 intervals.



HPC @ IU - Computing PI using Elementary Numerical Integration: Parallel version

Computing PI using Parallel Code

Program parallelPi - Actual Parallelization of serialPi


HPC @ IU - Solving Diffusion Equation

The 2-D Diffusion Equation

The equation

  • The next example is more interesting than parallelPi. Our objective is to solve a two-dimensional diffusion PDE:

    where u(x,y,t) is the temperature of the surface being modeled, and D is the so-called diffusion coefficient.

  • Let us choose as the surface a rectangular plate:

Discretized Numerical Solution

  • In this workshop, we will not worry about the exact solution (an infinite sum). Instead we will look at an approximate solution using a discrete grid. We don't bother about the real math because our objective is to learn parallel computing. In any case, the exact solution to the above equation is shown at the end of this page.

  • Going back to the rectangular place shown above, let us superimpose a computational grid upon the plate we are modeling:

    which shows the numbering of the x and y grid indices i and j, respectively. The number of interior grid points in the x-direction is nni, in the y-direction nnj. The boundaries are i = 0, i = nni + 1, j = 0, and j = nnj + 1.

  • Each point in the grid can be computed using the following equation: (based on a finite-difference equation):

    • Let us call the expressions outside of the parentheses mu, c and nu. Using temperature at time step n, the temperature at time step n + 1 can be computed.

  • For the above finite-difference representation of the two-dimensional diffusion equation, we will have to consider a 5-point stencil that must be computed to obtain u_ij at the next time step. The stencil considers the closest neighbors (diagonal points are not considered because they are not the closest).

    Later when considering parallelization we will be especially interested in the stencil. The stencil is the set of grid points needed to compute the single ij grid point at the next time step.

Exact Solution



HPC @ IU - Solving Diffusion - Program serialDiffusion

Serial Numerical Solution of the Diffusion Equation

Program: serialDiffusion

  • As we'd mentioned earlier, if you are beginning a parallel project from scratch, writing the serial code is typically the first step.

  • In C, our serialDiffusion program is shown below: 

    #include <stdio.h>
#include <stdlib.h>
#include <math.h>

const double pi = 3.1415926535897932385;

int main( int argc, char *argv[] ) {
  /*  Arguments required for executing argv[0]:      */
  /*  0                         -> serialDiffusion   */
  /*  1                         -> nni               */
  /*  2                         -> nnj               */
  /*  3                         -> numTimeSteps      */


  double **uo; /* solution at previous time step */
  double **u;  /* solution at current time step  */

  double mu, nu, c;    /* parameters of the FDE                              */
  double ell1,ell2;    /* x,y grid dimensions, 0 <= x <= ell1, etc.          */
  double dx,dy;        /* x,y grid spacing, or spacial step size             */
  int    nni,nnj;      /* x,y number of interior grid points; excl. bdy. pts.*/
  int    numTimeSteps; /* when to end simulation                             */
  double dt;           /* time step size                                     */
  double dcoeff;       /* diffusion coefficient D                            */
  double u_0;          /* uniform initial conditions                         */
  double actual;       /* the true solution                                  */

  int    i,j,m,n,timeStep;

  /*  Get the number of interior grid points and  */
  /*  number of time steps from the command line. */
  nni = atoi( argv[1] );
  nnj = atoi( argv[2] );
  numTimeSteps = atoi( argv[3] );

  /*  Allocate memory for the solution arrays; include boundaries. */
  /*  As declared above, u is a pointer to a pointer to a double.  */
  /*  Allocate memory so u points to the first element of an       */
  /*  (nni+2)-long array of pointers to doubles. Likewise for uo.  */
  u  = (double **) malloc( (size_t) ( (nni+2) * sizeof(double*) ) );
  uo = (double **) malloc( (size_t) ( (nni+2) * sizeof(double*) ) );

  /*  u[0] is a pointer to a double.                               */
  /*  Allocate memory so u[0] points to the first element of an    */
  /*  (nni+2)*(nnj+2)-long array of doubles. Likewise for uo[0].   */
  u[0]  = (double *) malloc( (size_t) ( ((nni+2)*(nnj+2)) * sizeof(double) ) );
  uo[0] = (double *) malloc( (size_t) ( ((nni+2)*(nnj+2)) * sizeof(double) ) );

  /*  Now set pointer u[1] to point to *(u[0] + (nnj+2)).          */
  /*  And set pointer u[2] to point to *(u[0] + 2*(nnj+2))         */
  /*      or *(u[1] + (nnj+2)).                                    */
  /*  And so forth. Likewise for uo[1], uo[2], etc.                */
  for ( i = 1; i < nni+2; i++ )  {
      u[i]  = u[i-1] + (nnj+2);
      uo[i] = uo[i-1] + (nnj+2);
  }

  /*  We have now dynamically allocated memory for the arrays      */
  /*  u[nni+2][nnj+2] and uo[nni+2][nnj+2] which are large         */
  /*  enough to include the boundary points.                       */

  /*  Set the diffusion coefficient, grid dimensions, */
  /*  and the uniform initial conditions.             */
  dcoeff = 1.0;
  ell1 = 7.0;
  ell2 = 1.0;
  u_0 = 1.0;

  /*  Compute dx and dy, and set dt so that the Courant condition */
  /*  is satisfied. */
  dx = ell1 / ( nni + 1 );
  dy = ell2 / ( nnj + 1 );
  dt = 0.49 / ( dcoeff*( 1.0/(dx*dx) + 1.0/(dy*dy) ) );

  /*  Set uniform initial conditions. */
  for ( i = 1; i <= nni; i++ )   {
      for ( j = 1; j <= nnj; j++ )	{
	  u[i][j] = u_0;
      }
  }

  /*  Set boundary conditions. */
  /*  Assume Dirichlet boundary conditions equal to 0. */
  /*  First the y = 0 boundary. */
  for ( i = 0; i <= nni+1; i++ )    {
      u[i][0] = 0.0;
  }
  /*  Second the y = ell2 boundary. */
  for ( i = 0; i <= nni+1; i++ )  {
      u[i][nnj+1] = 0.0;
  }
  /*  Third the x = 0 boundary. */
  for ( j = 1; j <= nnj; j++ )    {
      u[0][j] = 0.0;
  }
  /*  Finally the x = ell1 boundary. */
  for ( j = 1; j <= nnj; j++ )    {
      u[nni+1][j] = 0.0;
  }


  /*  Solve the 2-D diffusion equation by the explicit FTCS method. */

  /*  Pre-compute the FDE coefficients. */
  mu = dcoeff*dt / (dx*dx);
  nu = dcoeff*dt / (dy*dy);
  c  = 1.0 - 2.0*( mu + nu );

  for ( timeStep = 1; timeStep <= numTimeSteps; timeStep++ )    {
      /*  Update previous time step data. */
      for ( i = 0; i <= nni+1; i++ )	{
	  for ( j = 0; j <= nnj+1; j++ )	    {
	      uo[i][j] = u[i][j];
	  }
      }

      /*  Now compute the stencil for each i,j pair. */
      for ( i = 1; i <= nni; i++ )	{
	  for ( j = 1; j <= nnj; j++ )	    {
	      u[i][j] = mu*(uo[i+1][j]+uo[i-1][j]) + c*uo[i][j]
		+ nu*(uo[i][j+1]+uo[i][j-1]);
	  }
      }
      /*  Now proceed to the next time step. */
  }

  /*  Okay, the numerical computation of u is completed. */
  /*  Let's look now at a specific grid point.           */
  i = (int) nni/8;
  j = 10;
  /*  Compute the actual temperature at this point.  */
  actual = 0.0;
  for ( m = 1; m < 1000; m += 2 )    {
      for ( n = 1; n < 1000; n += 2 )	{
	  actual += 1.0/(m*n) * exp( - dcoeff * pi*pi * 
				     ( m*m/(ell1*ell1) + n*n/(ell2*ell2) ) *
				     numTimeSteps*dt )
	    * sin( m*pi*i*dx/ell1 ) * sin( n*pi*j*dy/ell2 );
      }
  }
  actual *= 16.0*u_0 / (pi*pi);
  /*  Compare the actual with the numerical result at this point. */
  printf( "After %d time steps some results are:\n", numTimeSteps );
  printf( "actual   u[%d][%d] = %f\n", i, j, actual );
  printf( "computed u[%d][%d] = %f\n", i, j, u[i][j] );

  /*  And let's look at another specific grid point. */
  i = (int) nni/4;
  /*  Compute the actual temperature at this point.  */
  actual = 0.0;
  for ( m = 1; m < 1000; m += 2 )    {
      for ( n = 1; n < 1000; n += 2 )	{
	  actual += 1.0/(m*n) * exp( - dcoeff * pi*pi * 
				     ( m*m/(ell1*ell1) + n*n/(ell2*ell2) ) *
				     numTimeSteps*dt )
	    * sin( m*pi*i*dx/ell1 ) * sin( n*pi*j*dy/ell2 );
      }
  }
  actual *= 16.0*u_0 / (pi*pi);
  /*  Compare the actual with the numerical result at this point. */
  printf( "After %d time steps some results are:\n", numTimeSteps );
  printf( "actual   u[%d][%d] = %f\n", i, j, actual );
  printf( "computed u[%d][%d] = %f\n", i, j, u[i][j] );

  return 0;
}

    Note: If you didn't copy source files from HPC's tutorial directory, then copy the above source code into a file called serialDiffusion.c inside your ~/MPI_Tutorial/SerialDiffusion/ directory.

  • Exercise: Do this!

    To compile the code, change to the ~/MPI_Tutorial/SerialDiffusion/ directory on the head node (bhX) and then type [since this is not a MPI program, we are not going to use the Makefile we've been using all along]: 

     [agopu@bh2 agopu]$ cd ~/MPI_Tutorial/SerialDiffusion/
 [agopu@bh2 SerialDiffusion]$ icc serialDiffusion.c -o serialDiffusion -lm
    Note: Our serialDiffusion program uses functions like sin () and exp (). That's whay we've used the -lm flag in the above call to icc ...to link with the Math library.

  • To run this program on a interactive node you got through qsub -I (bcXX), do: 

     [agopu@bc81 agopu]$ cd ~/MPI_Tutorial/SerialDiffusion/
 [agopu@bc81 SerialDiffusion]$ ./serialDiffusion 8000 1000 100

The results of serialDiffusion

  • The output of serialDiffusion 8000 1000 100 is: 

      After 100 time steps some results are:
  actual   u[1000][10] = 0.874806
  computed u[1000][10] = 0.873978
  After 100 time steps some results are:
  actual   u[2000][10] = 0.874775
  computed u[2000][10] = 0.873978

    The computed and actual values differ by one part in a thousand, which is not too bad considering we used 8000 and 1000 as parameters (not huge values).



HPC @ IU - Solving Diffusion - Parallelizing Serial Program

Parallelizing the Numerical Solution of the Diffusion Equation

Domain Decomposition

  • In our problem the easiest way to allocate grid points among the processese is to decompose along the x-axis:

  • At each time step, each process computes only it's portion of the grid.

Ghost Points

  • However, when a process computes a grid point that lies adjacent to the edge of its portion, one of the stencil points is actually owned by the neighboring process. This point is referred to as the process' "ghost points".

  • A process never computes the ghost points by itself; these points are computed during the prior time step by neighboring processes.

  • To compute the stencil at any given time step, each processor must receive values (of the grid point) at its ghost points from its neighbors at the conclusion of the previous time step. Thus, processes must share their ghost points between themselves after each time step, i.e. interprocessor communication is required at the conclusion of each time step.

  • A process' ghost points are u[0][j], which are received from its left neighbor's u[mynni][j], and u[mynni + 1][j], which are received from its right neighbor's u[1][j], for all j.

Interprocessor Exchange of Ghost Points

  • We will use the MPI_Send () and MPI_Recv () functions to transfer ghost points between the processors. These are blocking functions; they do not return until the communication is complete (in most installations of MPI). So we will need to make sure that a deadlock situation does not arise.

  • A deadlock arises in a perfectly symmetric situation when, for instance, two processes are sending to each other, and each process' call to MPI_Send () does not return, something which needs to happen for those processes to call their respective MPI_Recv () calls!

    • It's better to program asymmetrically, whereby one process first calls MPI_Send () and its respective receiving process calls MPI_Recv ().

  • To prevent deadlocks we'll choose odd-numbered processes to do MPI_Send () first and MPI_Recv () second, while even-numbered processes will do MPI_Recv () first and MPI_Send () second.



HPC @ IU - Solving Diffusion - Program parallelDiffusion

Parallelized Solution for Diffusion Equation

Program parallelDiffusion.c


HPC @ IU - Simple Illustration of MPI I/O and Minimal Error Handling

Simple MPI I/O and Error Handling

In this section, we will delve into few aspects of MPI we have not looked at before. You'll need to pay careful attention since it introduces multiple (and up to some extent unrelated/tangentially related) concepts.

  • We will see how MPI I/O can be used by multiple processes to read and write to files. All processes will read from/write to the same file using different offsets.

    Note: MPI I/O can only be used with parallel file-systems like GPFS. It is not designed to work with NFS and the like. Refer the MPI distribution's documentation to check what file-systems are supported.

  • Within the program illustrating MPI I/O (shown below), we also introduce very basic error handling mechanisms.

    Important: Do note that, error handling is especially important when you are dealing with files. For example, if you open a file for writing and encounter "write" errors, then it is your responsibility to close the file (handle) and also unlink (delete) it before your program terminates. Refer to the I/O section of the MPI-2 specification.


Program: mpi_io.c


The mpi_io Program Explained

Now that you've seen the program work (hopefully!), let us see how the code worked and what the various MPI functions mean...

  • The MPI_File_open () function opens a file identified by filename on all processes in the MPI_COMM_WORLD communicator. This is a collective function meaning that all processes must do it together and the values of all parameters passed to it must be identical on all processes too.

    The third parameter specifies how the file should be opened, e.g., for writing, reading or both, and whether it should be created if it doesn't exist. In our example, the input file is opened in read-only mode. All file-open modes are listed on the MPI-2 specification for MPI_File_open (). The options can be combined with the boolean ``or'', |, operator.

    The fourth parameter can be used to give the operating system additional hints about how and where the file should be opened - too advanced, so let us not worry about it.

  • The MPI_File_get_size () function gives the file size, which will be used later on to determine the offset for each process.

  • The MPI_File_seek () function points to the position in the file where each process will start reading data. The seek can be performed in one of three ways: MPI_SEEK_SET (the pointer is set exactly to my_offset), MPI_SEEK_CUR (the pointer is advanced by my_offset from its current position), MPI_SEEK_END (the pointer is advanced by my_offset from the end of the file). In general my_offset can be positive and negative, i.e., you can use it to move in both directions within the file.

  • The MPI_File_read () function reads data into the buffer specified as the second parameter. The size to be read is defined by the third parameter.

  • The MPI_File_write_at () function will write data from the buffer, specified by the third parameter, into a specific position on the file specified by the second parameter.

  • The MPI_File_close function closes the file opened by the function MPI_File_open.

  • The MPI_Allreduce () function is similar to MPI_Reduce () but all processes do the reduction and know the result of the operation (rather than just the master process knowing the result)

  • The MPI_Abort () function terminates all the MPI processes associated with the communicator specified. It also allows you to specify a return error code.

    For Kicks

  • Have you spotted a bug in the above shown code?

    Hint: Try running the code on a different input file while using the same output filename. What do you see? Is there a bug? If yes, what is it???

    More hints: Check out the various file open modes in the MPI_File_open () specification while paying special attention to MPI_MODE_EXCL; you'll need to do "ORing" of modes.

  • For more comprehensive information on MPI I/O and (associated) error handling, refer to the MPI  I/O section of Gustav Meglicki's I590 notes.



HPC @ IU Tutorial - Tweaking security settings to allow X-traffic from AVIDD

Tweaking Security Settings

Change security settings to allow X traffic from AVIDD (For TotalView and Intel Trace Analyzer)



HPC @ IU Tutorial - Hoops to jump on Linux (firewall setting, etc.) to open graphical applications from a Unix based server (specifically AVIDD)

Hoops to jump on Linux to open graphical applications from AVIDD

If you are interested in opening GUI applications from AVIDD compute nodes on a Windows workstation then go to the How do I run AVIDD-based graphical applications on my Windows computer? page.

  • Tweak IPTables Rules on your workstation

    • Wait! Skip this part if you are not running a firewall on your Linux box!

    • If you are indeed running a firewall on your Linux box, then you will need to modify your firewall rules on a Linux workstation to allow X-traffic from AVIDD's firewall node bh3.

      Note: The following information is provided for your convenience; there is no guarantee it will enable you to open X applications from AVIDD or any other *n*x based server (ultimately your other rules also have to be taken into consideration)

    • As root , add the rules shown below (check to see if you already have the last two rules): 

       $ /sbin/iptables -A INPUT -i eth0 -p tcp -s 149.165.228.0/255.255.255.0 -d <your_IP_address> \
   --dport 6000 -m state --state NEW -j ACCEPT
 $ /sbin/iptables -A INPUT -i eth0 -p tcp -s 149.165.229.0/255.255.255.0 -d <your_IP_address> \
   --dport 6000 -m state --state NEW -j ACCEPT  	    
 $ /sbin/iptables -A INPUT -i eth0  -m state --state EST,REL -j ACCEPT
 $ /sbin/iptables -A OUTPUT -m state --state EST,REL -j ACCEPT

      All you are doing with the above rules is: allowing NEW packets IN from AVIDD compute nodes (both Bloomington and Indy) into your workstation on TCP ports 6000 as well as established and related packets.

    • If the above rules work, then you can go ahead and save them to your default IP tables ruleset, so they will be applied every time your computer starts!

  • Append Xhost list on your workstation

    • On your workstation you will need to allow X packets from AVIDD compute nodes. To accomplish this, you may want to use the modify-xhost-for-avidd-x-traffic.pl script which is listed below for your convenience.
    • Exercise: Do this!
      Right-click on the above link and save the script to your workstation (or you could copy and paste above text if you wish). Let us assume you call it modify-xhost-for-avidd-x-traffic.pl; Run the script by doing: 
       $ perl modify-xhost-for-avidd-x-traffic.pl b add
      You should see something like: 
        149.165.228.1 being added to access control list
  149.165.228.2 being added to access control list
  . . .
    • If you are using the Indy cluster nodes, then use i instead of b as the first parameter. I.e. do "modify-xhost-for-avidd-x-traffic.pl i add".
    • Once you are done with using graphical applications, use remove as the second parameter to clear out the xhost list back to its original state. I.e. do "modify-xhost-for-avidd-x-traffic.pl b remove".

  • Find your workstation's IP address (assuming you are using the Ethernet interface eth0):

    • On Linux:
       [agopu@peart agopu]$ /sbin/ifconfig eth0 | grep "inet addr" | cut -f2 -d: | cut -f1 -d " " | head -1
    • On Sun Solaris: 
       bash-2.04$ /usr/sbin/ifconfig -a | grep -v "127.0.0.1" | grep "inet" | cut -f2 -d" "
      If you are unable to access ifconfig, then you could use the fully qualified hostname instead. You can get it by doing: 
       bash-2.04$ cat /etc/hosts | grep -v "#" | grep -v "localhost" | cut -f3 | head -1
    You'll be using the result of the above command (whichever was applicable) in the next step.

  • Set the DISPLAY environment variable on the compute node

    • You will need to finally set a DISPLAY environment variable on the compute node that you got using qsub -I; it should point to your workstation's IP address and display unit. Switch to the compute node terminal and do (replace the xx.xx.xx.xx with the real IP address.): 
       [agopu@bc81 agopu]$ export DISPLAY=xxx.xxx.xxx.xxx:0.0


  • For example

    1. On my laptop running Linux: 

       [agopu@laptop agopu]$ perl modify-xhost-for-avidd-x-traffic.pl b add
  149.165.228.1 being added to access control list
  149.165.228.2 being added to access control list
   . . .
  149.165.228.96 being added to access control list

 [agopu@laptop agopu]$ /sbin/ifconfig eth0 | grep "inet addr" | cut -f2 -d: | cut -f1 -d " " | head -1
   156.56.18.241

    2. Then on AVIDD's compute node: 

        [agopu@bc81 agopu]$ export DISPLAY=156.56.18.241:0.0

  • Note: If you are using wireless and VPN (for example, on your laptop) then you'll need to find that interface's IP address. Let us know if you have trouble doing this.

If you are interested in opening GUI applications from AVIDD compute nodes on a Windows workstation then go to the How do I run AVIDD-based graphical applications on my Windows computer? page.
Previous Next


HPC @ IU Tutorial - Using Windows and XLiveCD (especially to open graphical applications from a Unix based server)

Using Windows and XLiveCD

If you are interested in opening GUI applications on a Linux workstation then go to the Tweaking Linux firewall... page.

How do I use XLiveCD on my Windows (XP) computer?

  • Download XLiveCD and burn the ISO image onto a recordable CD.

  • Assuming the CD is inserted into your CD drive, wait to see if the CD auto-loads. If it does not, then go to My Computer or Windows Explorer and double click on the CD rom drive. Then double-click on "X.bat". This should open a command window and run some Cygwin stuff (might take a while).

  • Once the previous step completes, you should see a Xterm window. If you don't see one, then look for a "New Xterm" shortcut on your Desktop and double-click on it.

    Note: At this stage, the Windows XP firewall might prompt you about blocking/unblocking Xwin.exe. You can select "Unblock" if you are comfortable with doing that - see the Tweak Windows XP SP2 firewall section below for more information.

  • To login to AVIDD without default X-forwarding, do: 

     Desktop> ssh -x -l hpctrnXX avidd-b.iu.edu
    (Replace XX with your training account #; Once you finish this workshop replace the whole login account name with your own AVIDD account)

  • Note:

    If you do not expect to use an graphical applications like TotalView or Intel Trace Analyzer, then you can go back to where you came from...most probably the Initial Setup: Login to AVIDD webpage or the Tweaking Security Settings webpage.

How do I run AVIDD-based graphical applications on my Windows computer?