\documentclass[10pt]{article}
\usepackage{../pplmanual}
\input{../pplmanual}

\makeindex

\title{\charmpp\\ Finite Element Method (FEM) Framework\\ Manual}
\version{1.2}
\credits{
Initial version of \charmpp{} FEM Framework was developed
by Milind Bhandarkar with inputs from Timothy Hinrichs and Karthikeyan
Mahesh. The current version is almost completely rewritten by
Orion Lawlor. The most recent version is being called by the name ParFUM. 
ParFUM is short for Parallel Framework for Unstructured Meshes. 
This version has been mostly written by Nilesh Choudhury, Terry Wilmarth,
Sayantan Chakravorty and Issac Dooley.}

\begin{document}

\maketitle

\section{Introduction}

The Finite Element Method (FEM) approach is used in many engineering
applications with irregular domains, from elastic deformation problems to
crack propagation to fluid flow.  \charmpp{} is a free message-passing parallel
runtime system for machines from clusters of workstations to tightly-coupled
SMPs.  The \charmpp{} FEM framework allows you to write a parallel FEM program,
in C or Fortran 90, that closely resembles a serial version but includes
a few framework calls.

Using the FEM framework also allows you to take advantage of all the
features of \charmpp, including run-time load balancing,  performance
monitoring and visualization, and checkpoint/restart, with no additional
effort. The FEM framework also combines naturally with other \charmpp
frameworks built on TCHARM.

The FEM framework has been undergoing a wave of recent improvements. A choice to rename the new version ParFUM has been adopted.ParFUM is short for Parallel Framework for Unstructured Meshes. Section \ref{sec:ParFUM} describes some of the new features included in ParFUM that were not present in FEM.


\subsection{Philosophy}

The \charmpp{} FEM framework is designed to be flexible, in that it
provided a few very general operations, such as loading and partitioning 
a ``mesh.''  
In describing these operations, we draw on examples from structural analysis,
but in fact the same calls can be used for other applications, including
fluid dynamics or partial differential equations solvers, or
even general-purpose graph manipulation.

For example, the FEM framework does not specify the number of spatial
dimensions.  Node locations are treated as just another kind of node data,
with no restrictions on the number of data items.
This allows the FEM framework to work with problems having any number 
of spatial dimensions.


\subsection{Terminology}
\label{sec:terminology}

A FEM program manipulates elements and nodes. An \term{element} is a portion of
the problem domain, also known as a cell, and is typically some simple shape 
like a triangle, square, or hexagon in 2D; or tetrahedron or rectangular solid in 3D.  
A \term{node} is a point in the domain, and is often the vertex of several elements.  
Together, the elements and nodes form a \term{mesh}, which is the 
central data structure in the FEM framework.

An element knows which nodes surround it via the element's
\term{connectivity table}, which lists the nodes adjacent to each element.

\begin{figure}[h]
\begin{center}
\includegraphics[width=4in]{fig/simple_mesh}
\end{center}
\caption{3-element, 5 node mesh.}
\label{fig:simplemesh}
\end{figure}

\begin{table}[h]
\begin{center}
\begin{tabular}{||l||l|l|l||}\hline
Element & \multicolumn{3}{c||}{Adjacent Nodes} \\\hline
e1 & n1 & n3 & n4 \\
e2 & n1 & n2 & n4 \\
e3 & n2 & n4 & n5 \\
\hline
\end{tabular}
\end{center}
\caption{Connectivity table for mesh in figure~\ref{fig:simplemesh}.}
\label{table:simplemesh}
\end{table}

A typical FEM program performs some element-by-element calculations which
update adjacent node values; then some node-by-node calculations.  For
example, a material dynamics program has the structure:

\begin{alltt}
     time loop
          element loop-- Element deformation applies forces to
          surrounding nodes
          node loop-- Forces and boundary conditions change node
          positions
     end time loop
\end{alltt}

We can parallelize such FEM programs by partitioning the serial mesh
elements into several smaller meshes, or \term{chunks}.  There is normally
at least one chunk per processor; and often even more.  During partitioning, 
we give nodes and elements new, \term{local} numbers within that chunk.
In the figure below, we have partitioned the mesh above into two chunks, A and B.

\begin{figure}[h]
\begin{center}
\includegraphics[width=4in]{fig/partitioned_mesh}
\end{center}
\caption{Partitioned mesh.}
\label{fig:partitionedmesh}
\end{figure}

\begin{table}[hh]
\begin{center}
\begin{tabular}{||l||l|l|l||}\hline
Element & \multicolumn{3}{c||}{Adjacent Nodes} \\\hline
e1 & n1 & n3 & n4 \\
e2 & n1 & n2 & n4 \\
\hline
\end{tabular}
\end{center}
\caption{Connectivity table for chunk A in figure~\ref{fig:partitionedmesh}.}
\label{table:chunkA}
\end{table}

\begin{table}[hh]
\begin{center}
\begin{tabular}{||l||l|l|l||}\hline
Element & \multicolumn{3}{c||}{Adjacent Nodes}\\\hline
e1 & n1 & n2 & n3 \\
\hline
\end{tabular}
\end{center}
\caption{Connectivity table for chunk B in figure~\ref{fig:partitionedmesh}.}
\label{table:chunkB}
\end{table}

Note that chunk A's node n2 and B's node n1 were actually the same node in
the original mesh-- partitioning split this single node into two shared
copies (one on each chunk).  However, since adding forces is associative, we
can handle shared nodes by computing the forces normally (ignoring the
existence of the other chunk), then adding both chunks' net force for the
shared node together.  This ``node update'' will give us the same resulting
force on each shared node as we would get without partitioning, thus the
same positions, thus the same final result.  

For example, under hydrostatic pressure, each chunk might compute a local
net force vector for its nodes as shown in Figure~\ref{fig:forcedecomp}
(a).  After adding forces across chunks, we have the consistent global forces
shown in Figure~\ref{fig:forcedecomp} (b).

\begin{figure}[h]
\begin{center}
\includegraphics[height=3in]{fig/forcedecomp}
\end{center}
\caption{A force calculation decomposed across chunks: (a) before update
(b) after updating forces across nodes.}
\label{fig:forcedecomp}
\end{figure}

Hence, each chunk's time loop has the structure:

\begin{alltt}
     chunk time loop
          element loop-- Element deformation applies forces to
          surrounding nodes
          <update forces on shared nodes>
          node loop-- Forces and boundary conditions change node
          positions
     end time loop
\end{alltt}

This is exactly the form of the time loop for a \charmpp{} FEM framework
program.  The framework will accept a serial mesh, partition it, distribute
the chunks to each processor, then you run your time loop to perform
analysis and communication.


\subsection{Structure of a Classic FEM Framework Program}

A classic FEM framework program consists of two subroutines: 
\kw{init()} and \kw{driver()}.
\kw{init()} is called by the FEM framework
only on the first processor -- this routine typically does specialized I/O,
startup and shutdown tasks.  \kw{driver()} is called for every chunk on every
processor, and does the main work of the program.  In the language of the
TCHARM manual, \kw{init()} runs in the serial context, 
and \kw{driver()} runs in the parallel context.

\begin{alltt}
     subroutine init
          read the serial mesh and configuration data
     end subroutine
/* after init, the FEM framework partitions the mesh */
     subroutine driver
          get local mesh chunk
          time loop
               FEM computations
               communicate boundary conditions
               more FEM computations
          end time loop
     end subroutine
\end{alltt}

In this mode, the FEM framework sets up a default writing
mesh during \kw{init()}, partitions the mesh after \kw{init()},
and sets up the partitioned mesh as the default reading mesh 
during \kw{driver()}. 


\subsection{Structure of an AMPI FEM Framework Program}

In addition to the classic init/driver structure above,
you can write an FEM framework program using the MPI style.
This is a more general, more flexible method of running
the program, but it is more complicated than the classic mode.
All FEM framework calls are available in either mode.

\begin{alltt}
   main program
      MPI_Init
      FEM_Init(MPI_COMM_WORLD)
      if (I am master processor)
         read mesh
      partition mesh
      time loop
          FEM computations
          communicate boundary conditions
          more FEM computations
      end time loop
   end main program
\end{alltt}

In this mode, the FEM framework does not set a default
reading or writing mesh, and does no partitioning;
so you must use the FEM\_Mesh routines to create and 
partition your mesh.
See the AMPI manual for details on how to declare
the main routine.

The driver() portion of a classic FEM program
strongly resembles an MPI mode main routine---in fact, a classic
FEM program can even make MPI calls from its \kw{driver()} 
routine, because the FEM framework is implemented directly on
top of MPI.  

There is even a special shell script for collecting
up the FEM framework source code to build a non-Charm, 
MPI-only version of the FEM framework.
To build FEM in this manner, you first build Charm++ normally,
then run a script to collect up the neccessary source files
(the FEM framework, a small number of Charm configuration 
and utility files, and the METIS library),
and finally build the library using the usual MPI compiler 
commands:
\begin{alltt}
 > cd charm/
 > ./src/libs/ck-libs/fem/make_fem_alone.sh
 > cd fem_alone/
 > mpicc -I. -DFEM_ALONE=1 -c *.c *.C 
 > ar cr libfem_alone.a *.o
\end{alltt}
You will then have to build your application with the MPI
compilers, and manually point to this ``fem\_alone'' 
directory to find include files and the new FEM library.
A typical compiler invocation would be:
\begin{alltt}
 > mpif90 -I$HOME/charm/fem_alone -L$HOME/charm/fem_alone foo.f90 -lfem_alone -o foo
\end{alltt}
This ``standalone'', non-Charm++ method of building the 
FEM framework prevents the use of load balancing or the other
features of Charm++, so we do not recommend it for normal use.


\subsection{Compilation and Execution}

A FEM framework program is a \charmpp\ program, so you must begin by
downloading the latest source version of \charmpp\ from
{\tt http://charm.cs.uiuc.edu/}.  Build the source with 
{\tt ./build FEM version} or {\tt cd} into the build directory, 
{\tt version/tmp}, and type {\tt make FEM}.
To compile a FEM program, pass the {\tt -language fem} (for C) or 
{\tt -language femf} (for Fortran) option to {\tt charmc}.
You can also build using the ``fem\_alone'' mode described
at the end of the section above.

In a charm installation, see charm/version/pgms/charm++/fem/
for several example and test programs.

At runtime, a Charm++/FEM framework program accepts the following
options, in addition to all the usual Charm++ options described in 
the Charm++ ``Installation and Usage Manual''.

\begin{itemize}
\item {\tt +vp} $v$  

Create $v$ mesh chunks, or ``virtual processors''.
By default, the number of mesh chunks is equal to the number of 
physical processors (set with {\tt +p} $p$).


\item {\tt -write}

Skip \kw{driver()}.
After running \kw{init()} normally, the framework partitions the mesh, 
writes the mesh partitions to files, and exits.  As usual, the
{\tt +vp} $v$ option controls the number of mesh partitions.

This option is only used in the classic mode---MPI-style programs
are not affected.


\item {\tt -read}

Skip \kw{init()}.
The framework reads the partitioned input mesh from files
and calls \kw{driver()}.  Together with {\tt -write}, this option
allows you to separate out the mesh preparation and partitioning 
phase from the actual parallel solution run.

This can be useful, for example, if \kw{init()} requires more memory 
to hold the unpartitioned mesh than is available on one processor of 
the parallel machine.  To avoid this limitation, you can run the program
with {\tt -write} on a machine with a lot of memory to prepare the input
files, then copy the files and run with {\tt -read} on a machine with 
a lot of processors.

{\tt -read} can also be useful during debugging or performance tuning, 
by skipping the (potentially slow) mesh preparation phase.
This option is only used in the classic mode---MPI-style programs
are not affected.


\item {\tt +tcharm\_trace fem}

Give a diagnostic printout on every call into the FEM framework.
This can be useful for locating a sudden crash, or understanding
how the program and framework interact.  Because printing the 
diagnostics can slow a program down, use this option with care.

\end{itemize}


\section{FEM Framework API Reference}

Some of the routines in the FEM framework have different requirements or meanings
depending on where they are called from.  When a routine is described
as being ``called from driver'', this means it is called in the parallel
context---from \kw{driver()} itself, any subroutine called by \kw{driver()},
or from whatever routine is run by the FEM-attached TCHARM threads.
When a routine is described as being ``called from init'', this means it is 
called in the serial context---from \kw{init()} itself, from any subroutine
called from \kw{init()}, from a routine called by \kw{FEM\_Update\_mesh},
or from whatever TCHARM code executes before the \kw{FEM\_Attach}.


\subsection{Utility}

\prototype{FEM\_Num\_partitions}
\function{int FEM\_Num\_partitions();}
\function{INTEGER FUNCTION :: FEM\_Num\_partitions()}

     Return the number of mesh chunks in the current computation.  Can
     only be called from the driver routine.

\prototype{FEM\_My\_partitions}
\function{int FEM\_My\_partition();}
\function{INTEGER FUNCTION :: FEM\_My\_partition()}

     Return the number of the current chunk, from 0 to
     \kw{num\_partitions}-1.  Can only be called from the driver routine.

\prototype{FEM\_Timer}
\function{double FEM\_Timer();}
\function{DOUBLE PRECISION FUNCTION :: FEM\_Timer()}

     Return the current wall clock time, in seconds.  Resolution is
     machine-dependent, but is at worst 10ms.

\prototype{FEM\_Print\_partition}
\function{void FEM\_Print\_partition();}
\function{SUBROUTINE FEM\_Print\_partition()}

     Print a debugging representation of the current chunk's mesh.
     Prints the entire connectivity array, and data associated with
     each local node and element.

\prototype{FEM\_Print}
\function{void FEM\_Print(const char *str);}
\function{SUBROUTINE FEM\_Print(str)}
\args{CHARACTER*, INTENT(IN) :: str}

     Print the given string, with "[<chunk number>]" printed 
     before the text.  

     This routine is no longer required: you can now use 
     the usual printf, PRINT, or WRITE statements.


\input{mesh}

\input{idxl}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Old Communication Routines}

(This section is for backward compatability only.  The IDXL routines
are the new, more flexible way to perform communication.)

The FEM framework handles the updating of the values of shared nodes-- that
is, it combines shared nodes' values across all processors.  The basic
mechanism to do this update is the ``field''-- numeric data items associated
with each node. We make no assumptions about the meaning of the node data,
allow various data types, and allow a mix of communicated and non-communicated 
data associated with each node.  The framework uses IDXL layouts to find the data items
associated with each node in memory.

Each field represents a (set of) data records stored in a contiguous array,
often indexed by node number.  You create a field once, with the IDXL layout 
routines or \kw{FEM\_Create\_field},
then pass the resulting field ID to \kw{FEM\_Update\_field} (which does the
shared node communication), \kw{FEM\_Reduce\_field} (which applies a
reduction over node values), or one of the other routines described below.


\prototype{FEM\_Update\_field}
\function{void FEM\_Update\_field(int Fid,void *nodes);}
\function{SUBROUTINE FEM\_Update\_field(Fid,nodes)}
  \args{INTEGER, INTENT(IN)  :: Fid}
  \args{varies, INTENT(INOUT) :: nodes}

     Combine a field of all shared nodes with the other chunks.  Sums
     the value of the given field across all chunks that share each
     node.  For the example above, once each chunk has computed the net
     force on each local node, this routine will sum the net force
     across all shared nodes.

     \kw{FEM\_Update\_field} can only be called from driver, and to be useful,
     must be called from every chunk's driver routine.

     After this routine returns, the given field of each shared node
     will be the same across all processors that share the node.
     
     This routine is eqivalent to an \kw{IDXL\_Comm\_Sendsum} operation.

\prototype{FEM\_Read\_field}
\function{void FEM\_Read\_field(int Fid,void *nodes,char *fName);}
\function{SUBROUTINE FEM\_Read\_field(Fid,nodes,fName)}
  \args{INTEGER, INTENT(IN)  :: Fid}
  \args{varies, INTENT(OUT) :: nodes}
  \args{CHARACTER*, INTENT(IN) :: fName}

     Read a field out of the given serial input file.  The serial input
     file is line-oriented ASCII-- each line begins with the global
     node number (which must match the line order in the file),
     followed by the data to be read into the node field.  The
     remainder of each line is unread.  If called from Fortran, the
     first line must be numbered 1; if called from C, the first line
     must be numbered zero.  All fields are separated by white space
     (any number of tabs or spaces).

     For example, if we have called \kw{Create\_field} to describe 3 doubles,
     the input file could begin with

\begin{alltt}
          1    0.2    0.7    -0.3      First node
          2    0.4    1.12   -17.26    another node
          ...
\end{alltt}

     \kw{FEM\_Read\_field} must be called from driver at any time, independent
     of other chunks. 
     
     This routine has no IDXL equivalent.

\prototype{FEM\_Reduce\_field}
\function{void FEM\_Reduce\_field(int Fid,const void *nodes,void *out,int op);}
\function{SUBROUTINE FEM\_Reduce\_field(Fid,nodes,outVal,op)}
  \args{INTEGER, INTENT(IN)  :: Fid,op}
  \args{varies, INTENT(IN) :: nodes}
  \args{varies, INTENT(OUT) :: outVal}

Combine one record per node of this field, according to op, across all chunks.
Shared nodes are not double-counted-- only one copy will contribute to the
reduction.  After \kw{Reduce\_field} returns, all chunks will have identical
values in \kw{outVal,} which must be \kw{vec\_len} copies of \kw{base\_type}.

     May only be called from driver, and to complete, must be called
     from every chunk's driver routine.

     \kw{op} must be one of:

\begin{itemize}
        \item \kw{FEM\_SUM}-- each element of \kw{outVal} will be the sum 
of the corresponding fields of all nodes
        \item \kw{FEM\_MIN}-- each element of \kw{outVal} will be the 
smallest value among the corresponding field of all nodes
        \item \kw{FEM\_MAX}-- each element of \kw{outVal} will be the largest 
value among the corresponding field of all nodes
\end{itemize}

     This routine has no IDXL equivalent.


\prototype{FEM\_Reduce}
\function{void FEM\_Reduce(int Fid,const void *inVal,void *outVal,int op);}
\function{SUBROUTINE FEM\_Reduce(Fid,inVal,outVal,op)}
  \args{INTEGER, INTENT(IN)  :: Fid,op}
  \args{varies, INTENT(IN) :: inVal}
  \args{varies, INTENT(OUT) :: outVal}

     Combine one record of this field from each chunk, acoording to \kw{op}, 
 across all chunks.
\kw{Fid} is only used for the \kw{base\_type} and \kw{vec\_len}-- offset and
\kw{dist} are not used.  After this call returns, all chunks will have
identical values in \kw{outVal}.  Op has the same values and meaning as
\kw{FEM\_Reduce\_field}.

     May only be called from driver, and to complete, must be called
     from every chunk's driver routine.

\begin{alltt}
! C example
   double inArr[3], outArr[3];
   int fid=IDXL_Layout_create(FEM_DOUBLE,3);
   FEM_Reduce(fid,inArr,outArr,FEM_SUM);

! f90 example
   DOUBLE PRECISION :: inArr(3), outArr(3)
   INTEGER fid
   fid=IDXL_Layout_create(FEM_DOUBLE,3)
   CALL FEM_Reduce(fid,inArr,outArr,FEM_SUM)
\end{alltt}

     This routine has no IDXL equivalent.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Ghost Communication}

It is possible to get values for a chunk's ghost nodes and elements from the neighbors. To do this, use:

\prototype{FEM\_Update\_ghost\_field}
\function{void FEM\_Update\_ghost\_field(int Fid, int elTypeOrMinusOne, void *data);}
\function{SUBROUTINE FEM\_Update\_ghost\_field(Fid,elTypeOrZero,data)}
  \args{INTEGER, INTENT(IN)  :: Fid,elTypeOrZero}
  \args{varies, INTENT(INOUT) :: data}

This has the same requirements and call sequence as \kw{FEM\_Update\_field}, except it applies to ghosts. You specify which type of element to exchange using the elType parameter. Specify -1 (C version) or 0 (fortran version) to exchange node values.  


\subsection{Ghost List Exchange}

It is possible to exchange sparse lists of ghost elements between FEM chunks.

\prototype{FEM\_Exchange\_ghost\_lists}
\function{void FEM\_Exchange\_ghost\_lists(int elemType,int nIdx,const int *localIdx);}
\function{SUBROUTINE FEM\_Exchange\_ghost\_lists(elemType,nIdx,localIdx)}
  \args{INTEGER, INTENT(IN)  :: elemType,nIdx}
  \args{INTEGER, INTENT(IN) :: localIdx[nIdx]}

This routine sends the local element indices in localIdx to those neighboring chunks that connect to its ghost elements on the other side.  That is, if the element
\kw{localIdx[i]} has a ghost on some chunk \kw{c}, \kw{localIdx[i]} will be sent to 
and show up in the ghost list of chunk \kw{c}.

\prototype{FEM\_Get\_ghost\_list\_length}
\function{int FEM\_Get\_ghost\_list\_length(void);}
Returns the number of entries in my ghost list---the number of my ghosts that
other chunks passed to their call to \kw{FEM\_Exchange\_ghost\_lists}.

\prototype{FEM\_Get\_ghost\_list}
\function{void FEM\_Get\_ghost\_list(int *retLocalIdx);}
\function{SUBROUTINE FEM\_Get\_ghost\_list(retLocalIdx)}
  \args{INTEGER, INTENT(OUT) :: retLocalIdx[FEM\_Get\_ghost\_list\_length()]}

These routines access the list of local elements sent by other chunks.  
The returned indices will all refer to ghost elements in my chunk.



% Permanently undocumented (i.e., officially denied) features:
%   FEM_Serial_Split(ndom) ! Partition into ndom domains
%   FEM_Serial_Begin(idom) ! Begin accessing the idom'th domain
%   
% ``There has to be a better way:''
%   FEM_Composite_elem, FEM_Combine_elem


\input{parfum}

\input{index}
\end{document}
