\documentclass{report}
\usepackage{graphics}
\usepackage[dvipdfm]{hyperref}
%
% This is the new (September 2005) MPICH2 design document.  The plan is
% to make this available both as a PDF document and split on the web in
% an easy to read fashion.  We can use latex2html to tohtml to provide a 
% simple version of this, and that may be enough.  However, to retain the
% option of generating web pages directly from this source, only simple 
% Latex should be used, and only the new forms (e.g., \texttt{...} instead of 
% {\tt ...}
%


\makeindex

\begin{document}

\markright{MPICH Design Document}

\title{MPICH2 Design Document}
\author{William D. Gropp and Rajeev Thakur}
\maketitle

\pagenumbering{roman}
\tableofcontents
\clearpage


%\raggedright
%% raggedright resets parindent
%\parindent 1em
%% no parskip when parindent used
%\parskip 0pt

\pagenumbering{arabic}
\pagestyle{headings}

\part{MPICH2 organization for users}

\chapter{Overview}
     <<up here, explain that MPICH2 is a framework for MPI implementation with many possible choices, including specialized communication devices and interfaces to process managers, not ch3+mpd.  Duplicate this discussion in the top-level entry for developers>>

\chapter{Communication Devices}

\chapter{Process Managers}

    other options 
    <<this chapter partially overlaps the installation manual, but that manual doesn't cover much of the motivation for the design>>

\chapter{Help Resources}

\section{Frequently Asked Questions (FAQ)}

\section{Diagnostic Programs}

\section{Buglist archives}


\part{MPICH2 organization for developers and hackers}


    <<each section (as appropriate) contains a rationale for the choices and a brief discussion of alternatives and why they were not chosen>>

\chapter{Goals of MPICH2}


\section{Principles}
         No duplicated code (no cut and paste programming) 
         Uniform appearance of code
         Uniform user interface for controls (e.g., standardized parameter handling, output, naming)
         Coverage and Testing

\section{Major components}
         devices
              the ch3 device and channels
         process managers and PM interface
         logging
         collectives
         topology

\section{Build System}
         rationale
              why not automake
              why not libtool
         using the build system
         using your own configure/make/build scripts

\section{Coding Style}
         Code template for uniform error checking and reporting (e.g.,
         common \texttt{fn\_fail} target) 
         Macros for common operations
              error reporting
              memory allocation
              debugging
         Safer or more efficient replacements for common routines
              safe string routines
              memcpy hooks
         Error reporting
              Rationale
               Adding new error classes and codes
         Tags for coverage analysis
         Take advantage of compiler features to identify potential
         problems, including warnings about easily misused features in
         C (such as assignments within if tests).  
	 Need to put somewhere - don't fix missing prototype messages
         by adding a prototype to the C file; there should always be
         one prototype in a header file somewhere.

\section{Major Structures}
      request, comm, group, etc.

\section{Selecting Features at Compile Time}
     use of macros to control which code is used
     list of all macros

\section{Language Bindings}
           rationale for buildiface
           Outstanding issues

\section{Scripts and Standards for user interfaces}
           command line and environment variables

\chapter{Adding your own implementation of a component}
         General mechanism (--with-<<component-name>>=directory, e.g., --with-pm=/home/me/mypm)
              configure and setup scripts
              standardized variables (\texttt{MPI\_CFLAGS}, CFLAGS, etc.)
         Specific components (describe, explain the API/ABI, and how to work with it):

Two types 

configure/compile-time

link/runtime

\section{Specifying Components during Configure}

\section{Specifying Components at Runtime}

It is possible to override the default routines for any component at
runtime by calling one of the following routines.

\begin{verbatim}
    MPIX_SetMethod( component, object, name, function, communicator,
    version )
\end{verbatim}

where
    component - identifier for the component (e.g., topology,
    collectives)
    object    - object to change; null if changing the defaults for
    all objects
    name      - name of the method (probably character string for function)
    function  - function to use
    communicator - communicator scope of this change (MPI\_COMM\_SELF
    and MPI\_COMM\_WORLD are popular choices)
    version   - version of the interface (see below)

    This call is collective over the communicator.  The intent of the 
    communicator option is to allow consistency checking for changes
    that must be consistent across processes (e.g., changing the
    collective algorithms).

    The version number is used to ensure that the version of the
    interface matches the one used in the library

A related call is

\begin{verbatim}
    MPIX_SetAllMethods( component, object, struct-of-functions*,
    communicator, version )
\end{verbatim}

This is similar to SetMethod, except all functions are set from those
defined in the struct-of-functions.  The version is very important
here as it ensures that the struct is known.

One additional call is

\begin{verbatim}
    MPIX_DLLLoadMethods( component, object, dll-name, communicator,
                         version )
\end{verbatim}

This loads the methods from a dynamic loaded shared object.  It is
collective over the communicator

Implementation note: the ``component'' in these calls can be the name
of a routine that 

\section{Collectives}
\section{Topology}
\section{Logging}
    --with-logging=/pathname
looks for \texttt{setup\_logging} script.  In addition, the
\texttt{Makefile} will be invoked.

    clean, dist-clean, maintainer-clean, library target?

    Must provide an mpilogging.h file (contents defined as...)

    Example (give example directory on web)

    Builtin versions...

\section{PM and PMI}

mpiexec 
    -pmiargs host port executable
special args used to allow singleton init.

\section{Name Server}

    --with-nameserver=...

\section{mpid}
\section{ch3-channel}

          Components still evolving (a placeholder for things that we want to do)

\chapter{Working with MPICH2 itself}
          <<for people that edit MPICH2 rather than use the component interface>>
          Coding standards
          Keeping configure clean
          Keeping MPICH modular
          

\section{Parameters within the MPICH2 code}

   The purpose of this part is to enable the configure, compile, and
   run-time control of parameters in the collective implementation.

   Goals for the parameter handling routines:



\begin{enumerate}
\item All values (at least in comm\_world) must be the same (the
      collective routines expect the same parameter values)


      For communicators in different comm worlds (e.g., created by
      spawn or connect/accept), it may be necessary to perform a
      separate step when the communicator is created to negotiate
      parameter choices.  This may require a communicator-creation
      hook.

      The most likely implementation of this step is to check that the
      different comm\_worlds have compatible (typically identical)
      values for the parameters.  There may be a
      \texttt{MPIR\_Parm\_check\_consistent}(new intercomm) call for this.

\item The parameter routine should provide the following choices, in
      order of decreasing priority:
     
\begin{enumerate}
\item Explicit control from within the routine (i.e., by a routine
      call)
\item Command line parameter on mpiexec
\item Environment variable on the process with rank 0 in MPI\_COMM\_WORLD.
\item Value is user-specified parameter file (e.g., ~/.mpichrc)
\item Value in site-specified parameter file (e.g.,
      /usr/local/mpich2/.mpichrc)
\item Compile-time value set at configure time (e.g.,
      --enable-collective-config=file) 
\item Compile-time value set within the collective code (the
      default values)
\end{enumerate}
      The choice of parameter value should also have some configure-time
      control to allow a trade-off between generality and absolute
      best performance (particular for startup, if we want to allow
      configurations files to be read).

      Note that even process managers that can deliver the same
      environment variable to all processes often allow the user to
      change that behavior (e.g., with a command-line option to
      prevent environment variables from being sent to other
      processes, so we cannot assume that the values of the
      environment variables are the same on all processes without some
      additional information (e.g., the process manager could tell us
      that the environment variables are the same on all processes).

\item Parameter documentation should (at least in part) be provided
      where the parameters are used.

\item Overhead of using parameter routines should be low; particularly 
      after the first use (if that matters; that is, the first time
      may involve an initialization phase)

\item Parameter routines should be a component, cleanly initialized by
      MPI\_Init/MPI\_Init\_thread and shut down by MPI\_Finalize.

\item Parameter types that must be handled include integer (e.g.,
      message size, group size).  Others should be allowed for (such
      as arrays of integers, characters) as needed.  We should avoid
      floating-point values because of possible problems in using them
      consistently, particularly in a heterogeneous environment.

      Note that this does not say anything about how the values may be
      represented in a parameter database, as that is an
      implementation issue.  However, it does say that the value must
      be delivered to the using routine in the form that it needs.  We
      do not want to call strtol() everytime a routine needs an
      integer (atoi should not be used because it has no way to
      indicate that the value contains non-digit characters).

\item The design should ensure that changes need be made in only one
      place or alternately that inconsistencies (e.g., between an
      initialization and a use) are detectable before runtime (before
      runtime so that the problem can be detected and fixed without
      depending on running a particular test case).

\end{enumerate}
Proposed Design that meets these goals

\begin{enumerate}
\item => Some setup at MPI\_Init time to ensure all processes have
consistent values.  Also requires hook for communicators that span
multiple MPI\_COMM\_WORLDs.
\item => registration of names so that the data can be acquired scalably
(e.g., read from file, environment variable on process with rank zero)
\item => a description string either in the call or in the structured
   comment at the point of use.
\item also => an initialization step 
\item => using the finalize callback.  To make them a component for the
initialization step, there needs to be some mechanism to load modules
\item ?
\item => Either a single point of use (which introduces efficiency
problems) or some sort of source-code preprocessing.  
\end{enumerate}

Based on these, I propose the following:

Initialization module:\\
This routine is called from MPI\_Init.  It details all parameters used
by the collective routines, the associated environment variables, and
the description strings.  It might look something like:

\begin{verbatim}
int MPIR_Parm_register_collective( ) {
    rc = MPIR_Parm_register_int( "MPICH:SCATTER_THRESHOLD", 
			         &MPIR_Scatter_threshold, 
                                 2048,
    "Maximum size of messages sent using \
    doubling in scatter algorithm with a default of %d" );
    ...

    return MPI_SUCCESS;
}
\end{verbatim}

The purpose of this routine is to communicate, at runtime, the names
of the parameters that some part of the system may need.  The
parameters to this routine are:
\begin{description}
\item[\texttt{MPICH:SCATTER\_THRESHOLD}]The name of the parameter.  The
    environment variable that might correspond to this is
    \texttt{MPICH\_SCATTER\_THRESHOLD}

\item[\texttt{\&MPIR\_Scatter\_threashold}]The address of an int (since this is
  \texttt{MPIR\_Parm\_register\_int}) that will contain the value.
  This integer can be accessed directly by the code; see the example of the use
    below.

\item[2048]The default value if no value is provided

\item["Maximum ... \%d..."]A documentation string including a format
    specifier for the default value.  This will be used to create
    help information on the parameter directly from the source code.
\end{description}
Possible additional features would be a valid range (e.g., 0 to 64K)
or a routine to test for valid input.

Once all of these routines are called (e.g., for any set of modules
that choose to use this method to acquire parameters), the routine

\begin{verbatim}
int MPIR_Parm_init( )
\end{verbatim}

is called from within MPI\_Init/MPI\_Init\_thread.  This is collective
over COMM\_WORLD and performs the 
necessary file reads, environment variable reads and broadcasts the
results to all processes.  It handles all registered names, from all
modules.

In the code, all parameters are accessed through a macro that allows
the values to be either compile-time or run-time constants

\begin{verbatim}
MPIR_PARM_GET_INT( "MPICH:SCATTER_THRESHOLD", MPIR_Scatter_threshold,2048 )
\end{verbatim}

This expands into either:
\begin{verbatim}
    2048 (compile-time only)
\end{verbatim}
or
\begin{verbatim}
    MPIR_Scatter_threshold (run-time)
\end{verbatim}
or even (runtime, lazy evaluation):
\begin{verbatim}
    (MPIR_Scatter_threshhold_not_set ? \
        (MPIR_Scatter_threshhold_not_set=1,\
        MPIR_Scatter_threshold=MPIR_ScatterMPIRParmGetInt(\
          "MPICH:SCATTER_THRESHOLD", 2048 )) : MPIR_Scatter_threshhold )
\end{verbatim}

The name (the first argument) is provided to allow for consistency
checking against the registered names.   And in practice, the "2048"
would itself be a macro, set in an include file in the coll directory
(not in src/include/mpiimpl.h).  

The parm init routine would use the name in the register routine (the
first argument) in the following way:

\begin{verbatim}
The : in the name separates the prefix from the rest of the name.
The name after the : may be used in an init file
The name with the : replaced by an \_ may be used as an environment
variable or in the init file.
\end{verbatim}

A more sophisticated approach would use code similar to (and shared
with) the error message extraction to synthesize the
\texttt{MPIR\_Parm\_register\_collective()} routine.  This would be requried to
address goal 7 above.  An alternative would be to simply detect an
inconsistency by having a program read the source code and check that
the registration and use calls match.  

The script
\texttt{maint/extractstrings}, combined with code similar to that for 
error messages in \texttt{maint/extracterrmsgs}, can be used to
automate the collection of registration information from the source code.
% ---
% Old text from December 2004
%% 1) Control of parameterization 
%%     The purpose of this part is to enable the configure, compile, and 
%%     run-time control of parameters in the collective implementation. 

%%     Goals for the parameter handling routines: 

%%     1) All values (at least in comm_world) must be the same (the 
%%        collective routines expect the same parameter values) 

%%        a) For communicators in different comm worlds (e.g., created by 
%%        spawn or connect/accept), it may be necessary to perform a 
%%        separate step when the communicator is created to negotiate 
%%        parameter choices.  This may require a communicator-creation 
%%        hook. 

%%     2) The parameter routine should provide the following choices, in 
%%        order of decreasing priority: 

%%        a) Explicit control from within the routine (i.e., by a routine 
%%        call) 
%%        b) Environment variable on rank-0 communicator 
%%        c) Value is user-specified parameter file (e.g., ~/.mpichrc) 
%%        d) Value in site-specified parameter file (e.g., 
%%        /usr/local/mpich2/.mpichrc) 
%%        e) Compile-time value set at configure time (e.g., 
%%        --enable-collective-config=file) 
%%        f) Compile-time value set within the collective code (the 
%%        default values) 

%%        The choice of parameter value should also have some configure-time 
%%        control to allow a trade-off between generality and absolute 
%%        best performance (particular for startup, if we want to allow 
%%        configurations files to be read). 

%%     3) Parameter documentation should (at least in part) be provided 
%%        where the parameters are used. 

%%     4) Overhead of using parameter routines should be low; particularly 
%%        after the first use (if that matters; that is, the first time 
%%        may involve an initialization phase) 

%%     5) Parameter routines should be a component, cleanly initialized by 
%%        MPI_Init/MPI_Init_thread and shut down by MPI_Finalize. 

%%     6) Parameter types that must be handled include integer (e.g., 
%%        message size, group size).  Others should be allowed for (such 
%%        as arrays of integers, characters).  We should avoid 
%%        floating-point values because of possible problems in using them 
%%        consistently, particularly in a heterogeneous environment. 

%% Proposed Design that meets these goals 

%% 1) => Some setup at MPI_Init time to ensure all processes have 
%% consistent values.  Also requires hook for communicators that span 
%% multiple MPI_COMM_WORLDs. 
%% 2) => registration of names so that the data can be acquired scalably 
%% (e.g., read from file, environment variable on process with rank zero) 
%% 3) => a description string either in the call or in the structured 
%%     comment at the point of use. 
%% 4) also => an initialization step 
%% 5) => using the finalize callback.  To make them a component for the 
%% initialization step, there needs to be some mechanism to load modules 

%% Based on these, I propose the following: 

%% Initialization module: 
%% This routine is called from MPI_Init.  It details all parameters used 
%% by the collective routines, the associated environment variables, and 
%% the description strings.  It might look something like: 

%% int MPIR_Parm_register_collective( ) { 
%%      rc = MPIR_Parm_register_int( "MPICH:SCATTER_THRESHOLD", 
%%                                   &MPIR_Scatter_threshold, 
%%                                   2048, 
%%      "Maximum size of messages sent using \ 
%%      doubling in scatter algorithm with a default of %d" ); 
%%      ... 

%%      return MPI_SUCCESS; 
%% } 

%% The purpose of this routine is to communicate, at runtime, the names 
%% of the parameters that some part of the system may need. 

%% Once all of these routines are called (e.g., for any set of modules 
%% that choose to use this method to acquire parameters), the routine 

%% int MPIR_Parm_init( ) 

%% is called from within MPI_Init/MPI_Init_thread.  This is collective 
%% over COMM_WORLD and performs the 
%% necessary file reads, environment variable reads and broadcasts the 
%% results to all processes.  It handles all registered names, from all 
%% modules. 

%% In the code, all parameters are accessed through a macro that allows 
%% the values to be either compile-time or run-time constants 

%% MPIR_PARM_GET_INT( "MPICH:SCATTER_THRESHOLD", MPIR_Scatter_threshold, 2048 ) 

%% This expands into either: 

%% 2048 (compile-time only) 

%% MPIR_Scatter_threshold (run-time) 

%% or even (runtime, lazy evaluation): 

%% (MPIR_Scatter_threshhold_not_set ? \ 
%%      (MPIR_Scatter_threshhold_not_set=0,MPIR_Scatter_threshold=MPIR_ScatterMPIRParmGetInt("MPICH:SCATTER_THRESHOLD", 
%% 2048 )) : MPIR_Scatter_threshhold ) 

%% The name (the first argument) is provided to allow for consistency 
%% checking against the registered names.   And in practice, the "2048" 
%% would itself be a macro, set in an include file in the coll directory 
%% (not in src/include/mpiimpl.h). 

%% The parm init routine would use the name in the register routine (the 
%% first argument) in the following way: 

%% The : in the name separates the prefix from the rest of the name. 
%% The name after the : may be used in an init file 
%% The name with the : replaced by an _ may be used as an environment 
%% variable or in the init file. 

%% A more sophisticated approach would use code similar to (and shared 
%% with) the error message extraction to synthesize the 
%% MPIR_Parm_register_collective() routine. 

\end{document}
