\documentclass[10pt]{article}
\usepackage{../pplmanual,pst-node}
\input{../pplmanual}

\title{Adaptive MPI Manual}
\version{1.0}
\credits{
AMPI has been developed by Milind Bhandarkar with inputs from Gengbin Zheng and
Orion Lawlor. The derived data types (DDT) library, which AMPI uses for the
derived data types support, has been developed by Neelam Saboo. The current 
version of AMPI is maintained by Chao Huang.
}

\begin{document}
\maketitle

\section{Introduction}

This manual describes Adaptive MPI~(\ampi{}), which is an implementation of a
significant subset\footnote{Currently, 110 MPI-1.1 Standard functions have 
been implemented.} of MPI-1.1 Standard over \charmpp{}. \charmpp{} is a
\CC{}-based parallel programming library being developed by Prof. L. V. Kal\'{e} 
and his students back from 1992 until now at University of Illinois.

We first describe our philosophy behind this work (why we do what we do).
Later we give a brief introduction to \charmpp{} and rationale for \ampi{}
(tools of the trade). We then describe \ampi{} in detail. Finally we summarize the
changes required for original MPI codes to get them working with \ampi{}
(current state of our work). Appendices contain the gory details of installing
\ampi{}, building and running \ampi{} programs.

\subsection{Overview}

Developing parallel Computational Science and Engineering (CSE) applications is
a complex task. One has to implement the right physics, develop or choose and
code appropriate numerical methods, decide and implement the proper input and
output data formats, perform visualizations, and be concerned with correctness
and efficiency of the programs. It becomes even more complex for multi-physics
coupled simulations such as the solid propellant rocket simulation application. In addition, many applications
 are dynamic and adaptively refined so load imbalance is a major challenge.
Our philosophy is to lessen the burden of the application developers by
providing advanced programming paradigms and versatile runtime systems that can
handle many common programming and performance concerns automatically and let the application
programmers focus on the actual application content.

Many of these concerns can be addressed using processor virtualization and 
over-decomposition philosophy of \charmpp{}. Thus, the developer only sees 
virtual processors and lets the runtime system deal with underlying physical 
processors. This is implemented in \ampi{} by mapping MPI ranks to \charmpp{} user-level
 threads as illustrated in Figure\ref{fig_virt}. As an immediate and simple benefit, the programmer can use as 
many virtual processors ("MPI ranks") as the problem can be easily decomposed
 to them. For example, suppose the problem domain has $n*2^n$ parts that can be
  easily distributed but programming for general number of MPI processes is burdensome, 
  then the developer can have $n*2^n$ virtual processors on any number of physical ones using \ampi{}.
  
\begin{figure}[h]
\centering
\includegraphics[width=4.6in]{figs/virtualization.png}
\caption{MPI processes are implemented as user-level threads in \ampi{}}
\label{fig_virt}
\end{figure}

\ampi{}'s execution model consists of multiple user-level threads per process and,
typically, there is one process per physical processor. \charmpp{} scheduler coordinates
execution of these threads (also called Virtual Processors or VPs) and controls execution as shown in Figure \ref{fig_ratio}. These
VPs can also migrate between processors because of load balancing or other reasons.
The number of VPs per processor specifies the virtualization ratio (degree of over-decomposition). For example, in Figure \ref{fig_ratio} virtualization ratio is four (there are four VPs per each processor). Figure \ref{fig_prac} show how the problem domain is over-decomposed in \ampi{}'s VPs as opposed to other MPI implementations.

\begin{figure}[h]
\centering
\includegraphics[width=1.8in]{figs/ratio.png}
\caption{VPs are managed by \charmpp{} scheduler}
\label{fig_ratio}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=4.6in]{figs/prac.png}
\caption{Problem domain is over-decomposed to more VPs}
\label{fig_prac}
\end{figure}

Another benefit of virtualization is communication and computation overlap which 
is automatically achieved without programming effort. Techniques such as software 
pipelining require significant programming 
effort to achieve this goal and improve performance. However, one can use \ampi{} to 
have more virtual processors than physical processors to overlap communication and 
computation. Each time a VP is blocked for communication, \charmpp{} scheduler
 picks the next VP among those that are ready to execute. In this manner, while
 some of the VPs of a physical processor are 
waiting for a message to arrive, others can continue their execution. Thus, performance 
will be improved without any change to the source code.

A potential benefit is that of better cache utilization. With over-decomposition, a smaller subdomain is accessed by a VP repeatedly in different function calls before getting blocked by communication and switching to another VP. That smaller subdomain may fit into cache if over-decomposition is enough. This concept is illustrated in Figure \ref{fig_cache} where each \ampi{} subdomain (such as 12) is smaller than corresponding MPI subdomain (such as 3) and may fit into cache memory. Thus, there is a potential performance improvement without changing the source code.

\begin{figure}[h]
\centering
\includegraphics[width=4.6in]{figs/cache.png}
\caption{Smaller subdomains may fit into cache and result in better performance}
\label{fig_cache}
\end{figure}

One important concern is that of load imbalance. New generation parallel applications are 
dynamically varying, meaning that processors' load is shifting during execution. In a dynamic simulation application
such as rocket simulation, burning  solid fuel, sub-scaling for a certain part
of the mesh, crack propagation, particle flows all contribute to load
imbalance. Centralized load balancing strategy built into an application is
impractical since each individual modules are developed almost independently by
various developers. In addition, embedding a load balancing strategy in the code 
complicates it and programming effort increases significantly. Thus, the runtime 
system support for load balancing becomes even more critical. Figure \ref{fig_migrate} shows migration of a VP because of load imbalance. For instance, this domain may correspond to a weather forecast model where there is a tornado in top-left side, which requires more computation to simulate. \ampi{} will then migrate VP 13 to balance the division of work across processors and improve performance. Note that incorporating this sort of load balancing inside the application code may take a lot of effort and complicates the code.

\begin{figure}[h]
\centering
\includegraphics[width=4.6in]{figs/migrate.png}
\caption{\ampi{} migrates VPs across processors for load balancing}
\label{fig_migrate}
\end{figure}

There are different load balancing strategies built into \charmpp{} that can be 
selected. Among those, some may fit better for an application depending on its 
characteristics. Moreover, one can write a new load balancer, best suited for an 
application, by the simple API provided inside \charmpp{} infrastructure. Our approach is
based on actual measurement of load information at runtime, and on migrating
computations from heavily loaded to lightly loaded processors.

For this approach to be effective, we need the computation to be split into
pieces many more in number than available processors. This allows us to
flexibly map and re-map these computational pieces to available processors.
This approach is usually called ``multi-domain decomposition''.

\charmpp{}, which we use as a runtime system layer for the work described here,
simplifies our approach. It embeds an elaborate performance tracing mechanism,
a suite of plug-in load balancing strategies, infrastructure for defining and
migrating computational load, and is interoperable with other programming
paradigms.

\subsection{Terminology}

\begin{description}

\item[Module] A module refers to either a complete program or a library with an
orchestrator subroutine\footnote{Like many software engineering terms, this
term is overused, and unfortunately clashes with Fortran 90 module that denotes
a program unit. We specifically refer to the later as ``Fortran 90 module'' to
avoid confusion.} . An orchestrator subroutine specifies the main control flow
of the module by calling various subroutines from the associated library and
does not usually have much state associated with it.

\item[Thread] A thread is a lightweight process that owns a stack and machine
registers including program counter, but shares code and data with other
threads within the same address space. If the underlying operating system
recognizes a thread, it is known as kernel thread, otherwise it is known as
user-thread. A context-switch between threads refers to suspending one thread's
execution and transferring control to another thread. Kernel threads typically
have higher context switching costs than user-threads because of operating
system overheads. The policy implemented by the underlying system for
transferring control between threads is known as thread scheduling policy.
Scheduling policy for kernel threads is determined by the operating system, and
is often more inflexible than user-threads. Scheduling policy is said to be
non-preemptive if a context-switch occurs only when the currently running
thread willingly asks to be suspended, otherwise it is said to be preemptive.
\ampi{} threads are non-preemptive user-level threads.

\item[Chunk] A chunk is a combination of a user-level thread and the data it
manipulates. When a program is converted from MPI to \ampi{}, we convert an MPI
process into a chunk. This conversion is referred to as chunkification.

\item[Object] An object is just a blob of memory on which certain computations
can be performed. The memory is referred to as an object's state, and the set
of computations that can be performed on the object is called the interface of
the object.

\end{description}

\section{\charmpp{}}

\charmpp{} is an object-oriented parallel programming library for \CC{}.  It
differs from traditional message passing programming libraries (such as MPI) in
that \charmpp{} is ``message-driven''. Message-driven parallel programs do not
block the processor waiting for a message to be received.  Instead, each
message carries with itself a computation that the processor performs on
arrival of that message. The underlying runtime system of \charmpp{} is called
\converse{}, which implements a ``scheduler'' that chooses which message to
schedule next (message-scheduling in \charmpp{} involves locating the object
for which the message is intended, and executing the computation specified in
the incoming message on that object). A parallel object in \charmpp{} is a
\CC{} object on which a certain computations can be asked to performed from
remote processors.

\charmpp{} programs exhibit latency tolerance since the scheduler always picks
up the next available message rather than waiting for a particular message to
arrive.  They also tend to be modular, because of their object-based nature.
Most importantly, \charmpp{} programs can be \emph{dynamically load balanced},
because the messages are directed at objects and not at processors; thus
allowing the runtime system to migrate the objects from heavily loaded
processors to lightly loaded processors. It is this feature of \charmpp{} that
we utilize for \ampi{}.

Since many CSE applications are originally written using MPI, one would have to
do a complete rewrite if they were to be converted to \charmpp{} to take
advantage of dynamic load balancing and other \charmpp{} benefits. This is indeed impractical. However,
\converse{} -- the runtime system of \charmpp{} -- came to our rescue here,
since it supports interoperability between different parallel programming
paradigms such as parallel objects and threads. Using this feature, we
developed \ampi{}, an implementation of a significant subset of MPI-1.1
standard over \charmpp{}.  \ampi{} is described in the next section.

\section{AMPI}

\ampi{} utilizes the dynamic load balancing and other capabilities of \charmpp{} by
associating a ``user-level'' thread with each \charmpp{} migratable object.
User's code runs inside this thread, so that it can issue blocking receive
calls similar to MPI, and still present the underlying scheduler an opportunity
to schedule other computations on the same processor. The runtime system keeps
track of computation loads of each thread as well as communication graph
between \ampi{} threads, and can migrate these threads in order to balance the
overall load while simultaneously minimizing communication overhead. 

\subsection{AMPI Status}

Currently all the MPI-1.1 Standard functions are supported in \ampi{}, with a
collection of our extentions explained in detail in this manual. One-sided
communication calls in MPI-2 are implemented, but they are not taking advantage
of RMA features yet. Also ROMIO\footnote{http://www-unix.mcs.anl.gov/romio/} 
has been integrated to support parallel I/O features. Link with {\tt -lampiromio}
to take advantage of this library.

Following MPI-1.1 basic datatypes are supported in \ampi{}. (Some are not 
available in Fortran binding. Refer to MPI-1.1 Standard for details.)
\begin{alltt}
MPI_DATATYPE_NULL  MPI_BYTE            MPI_UNSIGNED_LONG MPI_LONG_DOUBLE_INT
MPI_DOUBLE         MPI_PACKED          MPI_LONG_DOUBLE   MPI_2FLOAT
MPI_INT            MPI_SHORT           MPI_FLOAT_INT     MPI_2DOUBLE
MPI_FLOAT          MPI_LONG            MPI_DOUBLE_INT    MPI_LB
MPI_COMPLEX        MPI_UNSIGNED_CHAR   MPI_LONG_INT      MPI_UB
MPI_LOGICAL        MPI_UNSIGNED_SHORT  MPI_2INT
MPI_CHAR           MPI_UNSIGNED        MPI_SHORT_INT
\end{alltt}

Following MPI-1.1 reduction operations are supported in \ampi{}.

\begin{alltt}
MPI_MAX   MPI_MIN   MPI_SUM   MPI_PROD  MPI_MAXLOC  MPI_MINLOC
MPI_LAND  MPI_LOR   MPI_LXOR  MPI_BAND  MPI_BOR     MPI_BXOR
\end{alltt}

Following are AMPI extension calls, which will be explained in detail in this
manual.
\begin{alltt}
MPI_Migrate     MPI_Checkpoint  MPI_Restart     MPI_Register    MPI_Get_userdata
MPI_Ialltoall   MPI_Iallgather  MPI_Iallreduce  MPI_Ireduce     MPI_IGet
\end{alltt}


\subsection{Name for Main Program}

To convert an existing program to use AMPI, the main function or program may need to be renamed. The changes should be made as follows:

\subsubsection{Fortran}

You must declare the main program as a subroutine called ``MPI\_MAIN''. Do not declare the main subroutine as a \textit{program} because it will never be called by the AMPI runtime.

\begin{alltt}

program pgm -> subroutine MPI_Main
	...		    		      ...
end program -> end subroutine
\end{alltt}

\subsubsection{C or C++}

The main function can be left as is, if \texttt{mpi.h} is included before the main function. This header file has a preprocessor macro that renames main, and the renamed version is called by the AMPI runtime by each thread.


\subsection{Global Variable Privatization}

For the before-mentioned benefits to be effective, one needs to map multiple
user-level threads onto each processor. 
Traditional MPI programs assume that the
entire processor is allocated to themselves, and that only one thread of
control exists within the process's address space. So, they may use global and static variables in the program.
However, global and static variables are problematic for multi-threaded environments such as \ampi{} or OpenMP.
This is because there is a single instance of those variables so they will be 
shared among different threads in the single address space and a wrong result may be produced by the program.
Figure \ref{fig_global} shows an example of a multi-threaded application with 
two threads in a single process. $var$ is a global or static variable in this 
example. Thread 1 assigns a value to it, then it gets blocked for communication 
and another thread can continue. Thereby, thread 2 is scheduled next and 
accesses $var$ which is wrong. Semantics of this program needs separate 
instances of $var$ for each of the threads. Thats where the need arises
to make some transformations to the original MPI program in order to run
correctly with \ampi{}.

\begin{figure}[h]
\centering
\includegraphics[width=4.6in]{figs/global.png}
\caption{Global or static variables are an issue for \ampi{}}
\label{fig_global}
\end{figure}

The basic transformation needed to port the MPI program to \ampi{} is
privatization of global variables.\footnote{Typical Fortran MPI programs
contain three types of global variables.

\begin{enumerate}

\item Global variables that are ``read-only''. These are either
\emph{parameters} that are set at compile-time. Or other variables that are
read as input or set at the beginning of the program and do not change during
execution. It is not necessary to privatize such variables.

\item Global variables that are used as temporary buffers. These are variables
that are used temporarily to store values to be accessible across subroutines.
These variables have a characteristic that there is no blocking call such as
\texttt{MPI\_recv} between the time the variable is set and the time it is ever
used. It is not necessary to privatize such variables either. 

\item True global variables. These are used across subroutines that contain
blocking receives and therefore the possibility of a context switch between the
definition and use of the variable. These variables need to be privatized.

\end{enumerate}
}
With the MPI process model, each MPI node can keep a copy of its own
``permanent variables'' -- variables that are accessible from more than one
subroutines without passing them as arguments.  Module variables, ``saved''
subroutine local variables, and common blocks in Fortran 90 belong to this
category. If such a program is executed without privatization on \ampi{}, all
the \ampi{} threads that reside on one processor will access the same copy of
such variables, which is clearly not the desired semantics.  To ensure correct
execution of the original source program, it is necessary to make such
variables ``private'' to individual threads. We are two choices: automatic 
global swapping and manual code modification.

\subsubsection{Automatic Globals Swapping}
Thanks to the ELF Object Format, we have successfully automated the procedure 
of switching the set of user global variables when switching thread contexts.
Executable and Linkable Format (ELF) is a common standard file format for Object Files in Unix-like operating systems.
ELF maintains a Global Offset Table (GOT) for globals so it is possible to
switch GOT contents at thread context-switch by the runtime system.


The only thing that the user needs to do is to set flag {\tt -swapglobals}
at compile and link time (e.g. ``ampicc -o prog prog.c -swapglobals"). It does not need 
any change to the source code and works with any language (C, C++, Fortran, etc).
However, it does not handle static variables and has a context switching overhead that grows with the number of global variables.
Currently, this feature only works on x86 and x86\_64
 (e.g. amd64) platforms that fully support ELF. Thus, it may not work on PPC or
  Itanium, or on some microkernels such as Catamount. When this feature does
   not work for you,
you can try other ways of handling global or static variables, which are detailed in the
following sections.

\subsubsection{Manual Change}
We have employed a strategy of argument passing to do this privatization
transformation. That is, the global variables are bunched together in a
single user-defined type, which is allocated by each thread dynamically. Then a
pointer to this type is passed from subroutine to subroutine as an argument.
Since the subroutine arguments are passed on a stack, which is not shared
across all threads, each subroutine, when executing within a thread operates on
a private copy of the global variables. 

This scheme is demonstrated in the following examples. The original Fortran 90 
code contains a module \texttt{shareddata}. This module is used in the main 
program and a subroutine \texttt{subA}.

\begin{alltt}
!FORTRAN EXAMPLE
MODULE shareddata
  INTEGER :: myrank
  DOUBLE PRECISION :: xyz(100)
END MODULE

SUBROUTINE MPI_MAIN
  USE shareddata
  include 'mpif.h'
  INTEGER :: i, ierr
  CALL MPI_Init(ierr)
  CALL MPI_Comm_rank(MPI_COMM_WORLD, myrank, ierr)
  DO i = 1, 100
    xyz(i) =  i + myrank
  END DO
  CALL subA
  CALL MPI_Finalize(ierr)
END PROGRAM

SUBROUTINE subA
  USE shareddata
  INTEGER :: i
  DO i = 1, 100
    xyz(i) = xyz(i) + 1.0
  END DO
END SUBROUTINE

//C Example
#include <mpi.h>

int myrank;
double xyz[100];

void subA();
int main(int argc, char** argv)\{
  int i;
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
  for(i=0;i<100;i++)
    xyz[i] = i + myrank;
  subA();
  MPI_Finalize();
\}

void subA()\{
  int i;
  for(i=0;i<100;i++)
    xyz[i] = xyz[i] + 1.0;
\}
\end{alltt}

\ampi{} executes the main subroutine inside a user-level thread as a subroutine. 
 
Now we transform this program using the argument passing strategy. We first group the
shared data into a user-defined type.

\begin{alltt}
!FORTRAN EXAMPLE
MODULE shareddata
  \emph{TYPE chunk}
    INTEGER :: myrank
    DOUBLE PRECISION :: xyz(100)
  \emph{END TYPE}
END MODULE

//C Example
struct shareddata\{
  int myrank;
  double xyz[100];
\};
\end{alltt}

Now we modify the main subroutine to dynamically allocate this data and change the
references to them. Subroutine \texttt{subA} is then modified to take this data
as argument. 

\begin{alltt}
!FORTRAN EXAMPLE
SUBROUTINE MPI_Main
  USE shareddata
  USE AMPI
  INTEGER :: i, ierr
  \emph{TYPE(chunk), pointer :: c}
  CALL MPI_Init(ierr)
  \emph{ALLOCATE(c)}
  CALL MPI_Comm_rank(MPI_COMM_WORLD, c\%myrank, ierr)
  DO i = 1, 100
    \emph{c\%xyz(i) =  i + c\%myrank}
  END DO
  CALL subA(c)
  CALL MPI_Finalize(ierr)
END SUBROUTINE

SUBROUTINE subA(c)
  USE shareddata
  \emph{TYPE(chunk) :: c}
  INTEGER :: i
  DO i = 1, 100
    \emph{c\%xyz(i) = c\%xyz(i) + 1.0}
  END DO
END SUBROUTINE

//C Example
void MPI_Main\{
  int i,ierr;
  struct shareddata *c;
  ierr = MPI_Init();
  c = (struct shareddata*)malloc(sizeof(struct shareddata));
  ierr = MPI_Comm_rank(MPI_COMM_WORLD, c.myrank);
  for(i=0;i<100;i++)
    c.xyz[i] = i + c.myrank;
  subA(c);
  ierr = MPI_Finalize();
\}

void subA(struct shareddata *c)\{
  int i;
  for(i=0;i<100;i++)
    c.xyz[i] = c.xyz[i] + 1.0;
\}
\end{alltt}

With these changes, the above program can be made thread-safe. Note that it is
not really necessary to dynamically allocate \texttt{chunk}. One could have
declared it as a local variable in subroutine \texttt{MPI\_Main}.  (Or for a
small example such as this, one could have just removed the \texttt{shareddata}
module, and instead declared both variables \texttt{xyz} and \texttt{myrank} as
local variables). This is indeed a good idea if shared data are small in size.
For large shared data, it would be better to do heap allocation because in
\ampi{}, the stack sizes are fixed at the beginning (can be specified from the
command line) and stacks do not grow dynamically.

\subsubsection{Source-to-source Transformation}
Another approach is to do the changes described in the previous 
scheme automatically. It means that we can use a tool to transform 
the source code to move global or static variables in an object and pass them around.
This approach is portable across systems and compilers and may also 
improve locality and hence cache utilization. It also does not have the 
context-switch overhead of swapping globals. However, it requires a new 
implementation of the tool for each language. Currently, there is a tool called \emph{Photran}\footnote{http://www.eclipse.org/photran}
for refactoring Fortran codes that can do this transformation. It is Eclipse-based and works by 
constructing Abstract Syntax Trees (ASTs) of the program.

\subsubsection{TLS-Globals}
Thread Local Store (TLS) was originally employed in kernel threads to localize variables and thread safety.
It can be used by annotating global/static variables with \emph{\_\_thread} in the source code.
Thus, those variables will have one instance per extant thread. This keyword is not an official
extension of the C language, however compiler writers are encouraged to
implement this feature. Currently, the ELF file format supports Thread Local
Storage.

It handles both global and static variables and has no context-switching
 overhead. Context-switching is just changing the TLS segment register
 to point to the thread's local copy. However, although it is popular, it  
is not supported by all compilers. Currently, \charmpp{} supports it for x86/x86\_64 platforms. 
A modified \emph{gfortran} is also available to use this feature.
To use TLS-Globals, one has to add \emph{\_\_thread} before all the global variables. For the example
above, the following changes to the code handles the global variables:
\begin{alltt}
__thread int myrank;
__thread double xyz[100];
\end{alltt}

The runtime system also should know that TLS-Globals is used at compile time:

\begin{alltt}
ampiCC -o example example.C -tlsglobals
\end{alltt}
Table \ref{tab:portability} shows portability of different schemes.

\begin{table*}[!t]
\begin{center}
\begin{tabular}{|c||c|c|c|c|c|c|c|c|c|}
\hline
Privatization
Scheme     & X86 & IA64 & Opteron & Mac OS X & IBM SP & SUN & BG/P & Cray/XT & Windows \\
		   \hline
		   \hline
		   Transformation  & Yes  & Yes   & Yes     & Yes     & Yes  & Yes  & Yes & Yes & Yes  \\
			   \hline
			   GOT-Globals  & Yes  & Yes  & Yes     & No       & No    & Maybe  & No  & No  & No \\
				   \hline
				   TLS-Globals  &  Yes & Maybe  & Yes     & No    & Maybe & Maybe & No  & Yes & Maybe  \\
					   \hline
					   \end{tabular}
					   \caption{Portability of current implementations of three privatization schemes.
						   ``Yes'' means we have implemented this technique.
							   ``Maybe'' indicates there are no theoretical problems, but no implementation exists.
							   ``No'' indicates the technique is impossible on this platform.}
							   \label{tab:portability}
							   \vspace{-1.0cm}
							   \end{center}
							   \end{table*}
\subsection{Extensions for Migrations}

For MPI chunks to migrate, we have added a few calls to \ampi{}. These include
ability to register thread-specific data with the run-time system, to pack all
the thread's data, and to express willingness to migrate.

\subsubsection{Registering Chunk data}

When the \ampi{} runtime system decides that load imbalance exists within the
application, it will invoke one of its internal load balancing strategies,
which determines the new mapping of \ampi{} chunks so as to balance the load.
Then \ampi{} runtime has to pack up the chunk's state and move it to its new
home processor. \ampi{} packs up any internal data in use by the chunk,
including the thread's stack in use. This means that the local variables
declared in subroutines in a chunk, which are created on stack, are
automatically packed up by the \ampi{} runtime system. However, it has no way
of knowing what other data are in use by the chunk. Thus upon starting
execution, a chunk needs to notify the system about the data that it is going
to use (apart from local variables.) Even with the data registration, \ampi{}
cannot determine what size the data is, or whether the registered data contains
pointers to other places in memory. For this purpose, a packing subroutine also
needs to be provided to the \ampi{} runtime system along with registered data.
(See next section for writing packing subroutines.) The call provided by
\ampi{} for doing this is \texttt{MPI\_Register}. This function takes two
arguments: A data item to be transported alongwith the chunk, and the pack
subroutine, and returns an integer denoting the registration identifier. In
C/\CC{} programs, it may be necessary to use this return value after migration
completes and control returns to the chunk, using function
\texttt{MPI\_Get\_userdata}. Therefore, the return value should be stored in a
local variable.

\subsubsection{Migration}

The \ampi{} runtime system could detect load imbalance by itself and invoke the
load balancing strategy. However, since the application code is going to
pack/unpack the chunk's data, writing the pack subroutine will be complicated
if migrations occur at a stage unknown to the application. For example, if the
system decides to migrate a chunk while it is in initialization stage (say,
reading input files), application code will have to keep track of how much data
it has read, what files are open etc. Typically, since initialization occurs
only once in the beginning, load imbalance at that stage would not matter much.
Therefore, we want the demand to perform load balance check to be initiated by
the application.

\ampi{} provides a subroutine \texttt{MPI\_Migrate} for this purpose. Each
chunk periodically calls \texttt{MPI\_Migrate}. Typical CSE applications are
iterative and perform multiple time-steps. One should call
\texttt{MPI\_Migrate} in each chunk at the end of some fixed number of
timesteps. The frequency of \texttt{MPI\_Migrate} should be determined by a
tradeoff between conflicting factors such as the load balancing overhead, and
performance degradation caused by load imbalance. In some other applications,
where application suspects that load imbalance may have occurred, as in the
case of adaptive mesh refinement; it would be more effective if it performs a
couple of timesteps before telling the system to re-map chunks. This will give
the \ampi{} runtime system some time to collect the new load and communication
statistics upon which it bases its migration decisions. Note that
\texttt{MPI\_Migrate} does NOT tell the system to migrate the chunk, but
merely tells the system to check the load balance after all the chunks call
\texttt{MPI\_Migrate}. To migrate the chunk or not is decided only by the
system's load balancing strategy.

\subsubsection{Packing/Unpacking Thread Data}

Once the \ampi{} runtime system decides which chunks to send to which
processors, it calls the specified pack subroutine for that chunk, with the
chunk-specific data that was registered with the system using
\texttt{MPI\_Register}. This section explains how a subroutine should be
written for performing pack/unpack.

There are three steps for transporting the chunk's data to other processor.
First, the system calls a subroutine to get the size of the buffer required to
pack the chunk's data. This is called the ``sizing'' step. In the next step,
which is called immediately afterward on the source processor, the system
allocates the required buffer and calls the subroutine to pack the chunk's data
into that buffer. This is called the ``packing'' step. This packed data is then
sent as a message to the destination processor, where first a chunk is created
(along with the thread) and a subroutine is called to unpack the chunk's data
from the buffer. This is called the ``unpacking'' step.

Though the above description mentions three subroutines called by the \ampi{}
runtime system, it is possible to actually write a single subroutine that will
perform all the three tasks. This is achieved using something we call a
``pupper''. A pupper is an external subroutine that is passed to the chunk's
pack-unpack-sizing subroutine, and this subroutine, when called in different
phases performs different tasks. An example will make this clear:

Suppose the chunk data is defined as a user-defined type in Fortran 90:

\begin{alltt}
!FORTRAN EXAMPLE
MODULE chunkmod
  TYPE, PUBLIC :: chunk
      INTEGER , parameter :: nx=4, ny=4, tchunks=16
      REAL(KIND=8) t(22,22)
      INTEGER xidx, yidx
      REAL(KIND=8), dimension(400):: bxm, bxp, bym, byp
  END TYPE chunk
END MODULE

//C Example
struct chunk\{
  double t;
  int xidx, yidx;
  double bxm,bxp,bym,byp;
\};
\end{alltt}

Then the pack-unpack subroutine \texttt{chunkpup} for this chunk module is
written as:

\begin{alltt}
!FORTRAN EXAMPLE
SUBROUTINE chunkpup(p, c)
  USE pupmod
  USE chunkmod
  IMPLICIT NONE
  INTEGER :: p
  TYPE(chunk) :: c

  call pup(p, c\%t)
  call pup(p, c\%xidx)
  call pup(p, c\%yidx)
  call pup(p, c\%bxm)
  call pup(p, c\%bxp)
  call pup(p, c\%bym)
  call pup(p, c\%byp)
end subroutine

//C Example
void chunkpup(pup_er p, struct chunk c)\{
  pup_double(p,c.t);
  pup_int(p,c.xidx);
  pup_int(p,c.yidx);
  pup_double(p,c.bxm);
  pup_double(p,c.bxp);
  pup_double(p,c.bym);
  pup_double(p,c.byp);
\}
\end{alltt}

There are several things to note in this example. First, the same subroutine
\texttt{pup} (declared in module \texttt{pupmod}) is called to size/pack/unpack
any type of data. This is possible because of procedure overloading possible in
Fortran 90. Second is the integer argument \texttt{p}. It is this argument that
specifies whether this invocation of subroutine \texttt{chunkpup} is sizing,
packing or unpacking. Third, the integer parameters declared in the type
\texttt{chunk} need not be packed or unpacked since they are guaranteed to be
constants and thus available on any processor.

A few other functions are provided in module \texttt{pupmod}. These functions
provide more control over the packing/unpacking process. Suppose one modifies
the \texttt{chunk} type to include allocatable data or pointers that are
allocated dynamically at runtime. In this case, when the chunk is packed, these
allocated data structures should be deallocated after copying them to buffers,
and when the chunk is unpacked, these data structures should be allocated
before copying them from the buffers.  For this purpose, one needs to know
whether the invocation of \texttt{chunkpup} is a packing one or unpacking one.
For this purpose, the \texttt{pupmod} module provides functions
\verb+fpup_isdeleting+(\verb+fpup_isunpacking+). These functions return logical value
\verb+.TRUE.+ if the invocation is for packing (unpacking), and \verb+.FALSE.+
otherwise. Following example demonstrates this:

Suppose the type \texttt{dchunk} is declared as:

\begin{alltt}
!FORTRAN EXAMPLE
MODULE dchunkmod
  TYPE, PUBLIC :: dchunk
      INTEGER :: asize
      REAL(KIND=8), pointer :: xarr(:), yarr(:)
  END TYPE dchunk
END MODULE

//C Example
struct dchunk\{
  int asize;
  double* xarr, *yarr;
\};
\end{alltt}

Then the pack-unpack subroutine is written as:

\begin{alltt}
!FORTRAN EXAMPLE
SUBROUTINE dchunkpup(p, c)
  USE pupmod
  USE dchunkmod
  IMPLICIT NONE
  INTEGER :: p
  TYPE(dchunk) :: c

  pup(p, c\%asize)
  \emph{
  IF (fpup_isunpacking(p)) THEN       !! if invocation is for unpacking
    allocate(c\%xarr(asize))
    ALLOCATE(c\%yarr(asize))
  ENDIF
  }
  pup(p, c\%xarr)
  pup(p, c\%yarr)
  \emph{
  IF (fpup_isdeleting(p)) THEN        !! if invocation is for packing
    DEALLOCATE(c\%xarr(asize))
    DEALLOCATE(c\%yarr(asize))
  ENDIF
  }

END SUBROUTINE

//C Example
void dchunkpup(pup_er p, struct dchunk c)\{
  pup_int(p,c.asize);
  if(pup_isUnpacking(p))\{
    c.xarr = (double *)malloc(sizeof(double)*c.asize);
    c.yarr = (double *)malloc(sizeof(double)*c.asize);
  \}
  pup_doubles(p,c.xarr,c.asize);
  pup_doubles(p,c.yarr,c.asize);
  if(pup_isPacking(p))\{
    free(c.xarr);
    free(c.yarr);
  \}
\}
\end{alltt}

One more function \verb+fpup_issizing+ is also available in module \texttt{pupmod}
that returns \verb+.TRUE.+ when the invocation is a sizing one. In practice one
almost never needs to use it.

\subsection{Extensions for Checkpointing}

The pack-unpack subroutines written for migrations make sure that the current
state of the program is correctly packed (serialized) so that it can be
restarted on a different processor. Using the \emph{same} subroutines, it
is also possible to save the state of the program to disk, so that if the 
program were to crash abruptly, or if the allocated time for the program
expires before completing execution, the program can be restarted from the
previously checkpointed state. Thus, the pack-unpack subroutines act as the 
key facility for checkpointing in addition to their usual role for migration.

A subroutine for checkpoint purpose has been added to AMPI:
\texttt{void MPI\_Checkpoint(char *dirname);}
This subroutine takes a directory name as its argument. It is a collective 
function, meaning every virtual processor in the program needs to call this 
subroutine and specify the same directory name. (Typically, in an
iterative AMPI program, the iteration number, converted to a character string,
can serve as a checkpoint directory name.) This directory is created, and the
entire state of the program is checkpointed to this directory.  One can restart
the program from the checkpointed state by specifying \texttt{"+restart
dirname"} on the command-line. This capability is powered by the \charmpp{} 
runtime system. For more information about \charmpp{} checkpoint/restart
mechanism please refer to \charmpp{} manual. 

\subsection{Extensions for Memory Efficiency}

MPI functions usually require the user to preallocate the data buffers needed before the
functions being called. For unblocking communication primitives, sometimes the user would
like to do lazy memory allocation until the data actually arrives, which gives the
oppotunities to write more memory efficient programs.     
We provide a set of AMPI functions as an extension to the standard MPI-2 one-sided calls,
where we provide a split phase MPI\_Get called MPI\_IGet. MPI\_IGet preserves the similar
semantics as MPI\_Get except that no user buffer is provided to hold incoming data.
MPI\_IGet\_Wait will block until the requested data arrives and runtime system takes
care to allocate space, do appropriate unpacking based on data type, and return.
MPI\_IGet\_Free lets the runtime system free the resources being used for this get request
including the data buffer. And MPI\_IGet\_Data is the utility program that returns the
actual data.     
 

\begin{alltt}

int MPI_IGet(MPI_Aint orgdisp, int orgcnt, MPI_Datatype orgtype, int rank,
             MPI_Aint targdisp, int targcnt, MPI_Datatype targtype, MPI_Win win,
             MPI_Request *request);

int MPI_IGet_Wait(MPI_Request *request, MPI_Status *status, MPI_Win win);

int MPI_IGet_Free(MPI_Request *request, MPI_Status *status, MPI_Win win);

char* MPI_IGet_Data(MPI_Status status);

\end{alltt}



\subsection{Extensions for Interoperability}

Interoperability between different modules is essential for coding coupled
simulations.  In this extension to \ampi{}, each MPI application module runs
within its own group of user-level threads distributed over the physical
parallel machine.  In order to let \ampi{} know which chunks are to be created,
and in what order, a top level registration routine needs to be written. A
real-world example will make this clear. We have an MPI code for fluids and
another MPI code for solids, both with their main programs, then we first
transform each individual code to run correctly under \ampi{} as standalone
codes. This involves the usual ``chunkification'' transformation so that
multiple chunks from the application can run on the same processor without
overwriting each other's data. This also involves making the main program into
a subroutine and naming it \texttt{MPI\_Main}.

Thus now, we have two \texttt{MPI\_Main}s, one for the fluids code and one for
the solids code. We now make these codes co-exist within the same executable,
by first renaming these \texttt{MPI\_Main}s as \texttt{Fluids\_Main} and
\texttt{Solids\_Main}\footnote{Currently, we assume that the interface code,
which does mapping and interpolation among the boundary values of Fluids and
Solids domain, is integrated with one of Fluids and Solids.} writing a
subroutine called \texttt{MPI\_Setup}.

\begin{alltt}
!FORTRAN EXAMPLE
SUBROUTINE MPI_Setup
  USE ampi
  CALL MPI_Register_main(Solids_Main)
  CALL MPI_Register_main(Fluids_Main)
END SUBROUTINE

//C Example
void MPI_Setup()\{
  MPI_Register_main(Solids_Main);
  MPI_Register_main(Fluids_Main);
\}
\end{alltt}

This subroutine is called from the internal initialization routines of \ampi{}
and tells \ampi{} how many number of distinct chunk types (modules) exist, and
which orchestrator subroutines they execute.

The number of chunks to create for each chunk type is specified on the command
line when an \ampi{} program is run. Appendix B explains how \ampi{} programs
are run, and how to specify the number of chunks (\verb|+vp| option). In the
above case, suppose one wants to create 128 chunks of Solids and 64 chunks of
Fluids on 32 physical processors, one would specify those with multiple
\verb|+vp| options on the command line as:

\begin{alltt}
> charmrun gen1.x +p 32 +vp 128 +vp 64
\end{alltt}

This will ensure that multiple chunk types representing different complete
applications can co-exist within the same executable. They can also continue to
communicate among their own chunk-types using the same \ampi{} function calls
to send and receive with communicator argument as \texttt{MPI\_COMM\_WORLD}.
But this would be completely useless if these individual applications cannot
communicate with each other, which is essential for building efficient coupled
codes.  For this purpose, we have extended the \ampi{} functionality to allow
multiple ``\texttt{COMM\_WORLD}s''; one for each application. These \emph{world
communicators} form a ``communicator universe'': an array of communicators
aptly called \emph{MPI\_COMM\_UNIVERSE}. This array of communicators is 
indexed [1 . . . \texttt{MPI\_MAX\_COMM}]. In the current implementation,
\texttt{MPI\_MAX\_COMM} is 8, that is, maximum of 8 applications can co-exist
within the same executable.

The order of these \texttt{COMM\_WORLD}s within \texttt{MPI\_COMM\_UNIVERSE}
is determined by the order in which individual applications are registered in
\texttt{MPI\_Setup}.

Thus, in the above example, the communicator for the Solids module would be
\texttt{MPI\_COMM\_UNIVERSE(1)} and communicator for Fluids module would be
\texttt{MPI\_COMM\_UNIVERSE(2)}.

Now any chunk within one application can communicate with any chunk in the
other application using the familiar send or receive \ampi{} calls by
specifying the appropriate communicator and the chunk number within that
communicator in the call. For example if a Solids chunk number 36 wants to send
data to chunk number 47 within the Fluids module, it calls:

\begin{alltt}
!FORTRAN EXAMPLE
INTEGER , PARAMETER :: Fluids_Comm = 2
CALL MPI_Send(InitialTime, 1, MPI_Double_Precision, tag, 
              \emph{47, MPI_Comm_Universe(Fluids_Comm)}, ierr)

//C Example
int Fluids_Comm = 2;
ierr = MPI_Send(InitialTime, 1, MPI_DOUBLE, tag,
                \emph{47, MPI_Comm_Universe(Fluids_Comm)});
\end{alltt}

The Fluids chunk has to issue a corresponding receive call to receive this
data:

\begin{alltt}
!FORTRAN EXAMPLE
INTEGER , PARAMETER :: Solids_Comm = 1
CALL MPI_Recv(InitialTime, 1, MPI_Double_Precision, tag, 
              \emph{36, MPI_Comm_Universe(Solids_Comm)}, stat, ierr)

//C Example
int Solids_Comm = 1;
ierr = MPI_Recv(InitialTime, 1, MPI_DOUBLE, tag,
                \emph{36, MPI_Comm_Universe(Solids_Comm)}, &stat);
\end{alltt}

\subsection{Extensions for Sequential Re-run of a Parallel Node}
In some scenarios, a sequential re-run of a parallel node is desired. One
example is instruction-level accurate architecture simulations, in which case
the user may wish to repeat the execution of a node in a parallel run in the
sequential simulator. AMPI provides support for such needs by logging the change
in the MPI environment on a certain processors. To activate the feature, build 
AMPI module with variable ``AMPIMSGLOG'' defined, like the following command in
charm directory. (Linking with zlib ``-lz'' might be required with this, for
generating compressed log file.)

\begin{alltt}
> ./build AMPI net-linux -DAMPIMSGLOG
\end{alltt}

The feature is used in two phases: writing (logging) the environment and
repeating the run. The first logging phase is invoked by a parallel run of the
AMPI program with some additional command line options. 

\begin{alltt}
> ./charmrun ./pgm +p4 +vp4 +msgLogWrite +msgLogRank 2 +msgLogFilename "msg2.log"
\end{alltt}

In the above example, a parallel run with 4 processors and 4 VPs will be
executed, and the changes in the MPI environment of processor 2 (also VP 2,
starting from 0) will get logged into diskfile "msg2.log". 

Unlike the first run, the re-run is a sequential program, so it is not invoked
by charmrun (and omitting charmrun options like +p4 and +vp4), and additional
comamnd line options are required as well. 

\begin{alltt}
> ./pgm +msgLogRead +msgLogRank 2 +msgLogFilename "msg2.log"
\end{alltt}

\subsection{Communication Optimizations for AMPI}
AMPI is powered by the \charmpp{} communication optimization support now!
Currently the user needs to specify the communication pattern by command
line option. In the future this can be done automatically by the system.

Currently there are four strategies available: USE\_DIRECT, USE\_MESH,
USE\_HYPERCUBE and USE\_GRID. USE\_DIRECT sends the message directly. 
USE\_MESH imposes a 2d Mesh virtual topology on the processors so each 
processor sends messages to its neighbors in its row and column of the 
mesh which forward the messages to their correct destinations. USE\_HYPERCUBE 
and USE\_GRID impose a hypercube and a 3d Grid topologies on the processors. 
USE\_HYPERCUBE will do best for very small messages and small number of 
processors, 3d has better performance for slightly higher message sizes 
and then Mesh starts performing best. The programmer is encouraged to try 
out all the strategies. (Stolen from the CommLib manual by Sameer :)

For more details please refer to the CommLib paper \footnote{L. V. Kale and 
Sameer Kumar and Krishnan Vardarajan, 2002. 
http://finesse.cs.uiuc.edu/papers/CommLib.pdf}. 

Specifying the strategy is as simple as a command line option +strategy. For
example:
\begin{alltt}
> ./charmrun +p64 alltoall +vp64 1000 100 +strategy USE\_MESH
\end{alltt}
tells the system to use MESH strategy for CommLib. By default USE\_DIRECT is
used.

\subsection{User Defined Initial Mapping}
                                                                                
You can define the initial mapping of virtual processors (vp) to physical 
processors (p) as a runtime option. You can choose from predefined initial 
mappings or define your own mappings. Following predefined mappings are 
available:
                                                                                
\begin{description}

\item[Round Robin]
                                                                                
This mapping scheme, maps virtual processor to physical processor in round-robin
fashion, i.e. if there are 8 virtual processors and 2 physical processors then
virtual processors indexed 0,2,4,6 will be mapped to physical processor 0 and 
virtual processors indexed 1,3,5,7 will be mapped to physical processor 1. 

\begin{alltt}
> ./charmrun ./hello +p2 +vp8 +mapping RR\_MAP
\end{alltt}
                                                                                
\item[Block Mapping]
                                                                                
This mapping scheme, maps virtual processors to physical processor in chunks, 
i.e. if there are 8 virtual processors and 2 physical processors then virtual 
processors indexed 0,1,2,3 will be mapped to physical processor 0 and virtual 
processors indexed 4,5,6,7 will be mapped to physical processor 1.
                                                                                
\begin{alltt}
> ./charmrun ./hello +p2 +vp8 +mapping BLOCK\_MAP
\end{alltt}
                                                                                
\item[Proportional Mapping]
                                                                                
This scheme takes the processing capability of physical processors into account
for mapping virtual processors to physical processors, i.e. if there are 2 
processors with different processing power, then number of virtual processors 
mapped to processors will be in proportion to their processing power.
                                                                                
\begin{alltt}
> ./charmrun ./hello +p2 +vp8 +mapping PROP\_MAP
> ./charmrun ./hello +p2 +vp8
\end{alltt}

\end{description}

If you want to define your own mapping scheme, please contact us for help.

\subsection{Compiling AMPI Programs}

\charmpp{} provides a cross-platform compile-and-link script called \charmc{}
to compile C, \CC{}, Fortran, \charmpp{} and \ampi{} programs.  This script
resides in the \texttt{bin} subdirectory in the \charmpp{} installation
directory. The main purpose of this script is to deal with the differences of
various compiler names and command-line options across various machines on
which \charmpp{} runs. While, \charmc{} handles C and \CC{} compiler
differences most of the time, the support for Fortran 90 is new, and may have
bugs. But \charmpp{} developers are aware of this problem and are working to
fix them. Even in its alpha stage of Fortran 90 support, \charmc{} still
handles many of the compiler differences across many machines, and it is
recommended that \charmc{} be used to compile and linking \ampi{} programs. One
major advantage of using \charmc{} is that one does not have to specify which
libraries are to be linked for ensuring that \CC{} and Fortran 90 codes are
linked correctly together. Appropriate libraries required for linking such
modules together are known to \charmc{} for various machines.

In spite of the platform-neutral syntax of \charmc{}, one may have to specify
some platform-specific options for compiling and building \ampi{} codes.
Fortunately, if \charmc{} does not recognize any particular options on its
command line, it promptly passes it to all the individual compilers and linkers
it invokes to compile the program.

\appendix

\section{Installing AMPI}

\ampi{} is included in the source distribution of \charmpp{}. 
To get the latest sources from PPL, visit:
	http://charm.cs.uiuc.edu/

and follow the download link.
Now one has to build \charmpp{} and \ampi{} from source.

The build script for \charmpp{} is called \texttt{build}. The syntax for this
script is:

\begin{alltt}
> build <target> <version> <opts>
\end{alltt}

For building \ampi{} (which also includes building \charmpp{} and other
libraries needed by \ampi{}), specify \verb+<target>+ to be \verb+AMPI+. And
\verb+<opts>+ are command line options passed to the \verb+charmc+ compile
script.  Common compile time options such as \texttt{-g, -O, -Ipath, -Lpath,
-llib} are accepted. 

To build a debugging version of \ampi{}, use the option: ``\texttt{-g}''. 
To build a production version of \ampi{}, use the options: ``\texttt{
--with-production}''.

\verb+<version>+ depends on the machine, operating system, and the underlying
communication library one wants to use for running \ampi{} programs.
See the charm/README file for details on picking the proper version.
Following is an example of how to build AMPI under linux and ethernet
environment, with debugging info produced:

\begin{alltt}
> build AMPI net-linux -g
\end{alltt}

\section{Building and Running AMPI Programs}
\subsection{Building}
\charmpp{} provides a compiler called charmc in your charm/bin/ directory. 
You can use this compiler to build your AMPI program the same way as other
compilers like cc. Especially, to build an AMPI program, a command line 
option \emph{-language ampi} should be applied. All the command line 
flags that you would use for other compilers can be used with charmc the 
same way. For example:

\begin{alltt}
> charmc -language ampi -c pgm.c -O3
> charmc -language ampi -o pgm pgm.o -lm -O3 
\end{alltt}

Shortcuts to the AMPI compiler are provided. If you have added charm/bin 
into your \$PATH environment variable, simply type \emph{mpicc, mpiCC, 
mpif77,} and \emph{mpif90} as provided by other MPI implementations.

\begin{alltt}
> mpicc -c pgm.c -g
\end{alltt}

\subsection{Running}
\charmpp{} distribution contains a script called \texttt{charmrun} that makes
the job of running \ampi{} programs portable and easier across all parallel
machines supported by \charmpp{}. \texttt{charmrun} is copied to a directory
where an \ampi{} prgram is built using \charmc{}. It takes a command line
parameter specifying number of processors, and the name of the program followed
by \ampi{} options (such as number of chunks to create, and the stack size of
every chunk) and the program arguments. A typical invocation of \ampi{} program
\texttt{pgm} with \texttt{charmrun} is:

\begin{alltt}
> charmrun pgm +p16 +vp32 +tcharm_stacksize 3276800
\end{alltt}

Here, the \ampi{} program \texttt{pgm} is run on 16 physical processors with
32 chunks (which will be mapped 2 per processor initially), where each
user-level thread associated with a chunk has the stack size of 3,276,800 bytes.

\end{document}
