\chapter{Automatic Parameter Marshalling}

Automatic Parameter Marshalling is a concise means of invoking
functions on remote processors.  The CPM module handles all the
details of packing, transmitting, translating, and unpacking the
arguments.  It also takes care of converting function pointers into
handler numbers.  With all these details out of the way, it is
possible to perform remote function invocation in a single line of
code.

\section{CPM Basics}

The heart of the CPM module is the CPM scanner.  The scanner reads a C
source file.  When it sees the keyword {\tt CpmInvokable} in front of
one of the user's function declarations, it generates a {\it launcher}
for that particular function.  The {\it launcher} is a function whose
name is {\tt Cpm\_} concatenated to the name of the user's function.
The launcher accepts the same arguments as the user's function, plus a
{\it destination} argument.  Calling the {\it launcher} transmits a
message to another processor determined by the {\it destination}
argument.  When the message arrives and is handled, the user's
function is called.

For example, if the CPM scanner sees the following function
declaration

\begin{alltt}
    CpmInvokable myfunc(int x, int y) \{ ... \}
\end{alltt}

The scanner will generate a launcher named {\tt Cpm\_myfunc}.
The launcher has this prototype:

\begin{alltt}
    void Cpm_myfunc(CpmDestination destination, int x, int y);
\end{alltt}

If one were to call {\tt Cpm\_myfunc} as follows:

\begin{alltt}
    Cpm_myfunc(CpmSend(3), 8, 9);
\end{alltt}

a message would be sent to processor 3 ordering it to call {\tt
myfunc(8,9)}.  Notice that the {\it destination} argument isn't just an
integer processor number.  The possible destinations for a message are
described later.

When the CPM scanner is applied to a C source file with a particular
name, it generates a certain amount of parameter packing and unpacking
code, and this code is placed in an include file named
similarly to the original C file: the {\tt .c} is replaced with {\tt
.cpm.h}.  The include file must be included in the original {\tt .c}
file, after the declarations of the types which are being packed and
unpacked, but before all uses of the CPM invocation mechanisms.

Note that the {\tt .cpm.h} include file is {\em not} for prototyping.
It contains the C code for the packing and unpacking mechanisms.
Therefore, it should only be included in the one source file from
which it was generated.  If the user wishes to prototype his code, he
must do so normally, by writing a header file of his own.

Each {\tt .cpm.h} file contains a function {\tt
CpmInitializeThisModule}, which initializes the code in {\it that}
{\tt .cpm.h} file.  The function is declared {\tt static}, so it is
possible to have one in each {\tt .cpm.h} file without conflicts.  It
is the responsibility of the CPM user to call each of these {\tt
CpmInitializeThisModule} functions before using any of the CPM
mechanisms.

We demonstrate the use of the CPM mechanisms using the following
short program {\tt myprog.c}:

\begin{alltt}
 1:    #include "myprog.cpm.h"
 2:   
 3:    CpmInvokable print_integer(int n)
 4:    \{
 5:      CmiPrintf("%d\verb+\+n", n);
 6:    \}
 7:    
 8:    user_main(int argc, char **argv)
 9:    \{
10:      int i;
11:      CpmModuleInit();
12:      CpmInitializeThisModule();
13:      if (CmiMyPe()==0)
14:        for (i=1; i<CmiNumPes(); i++)
15:          Cpm_print_integer(CpmSend(i), rand());
16:    \}
17:    
18:    main(int argc, char **argv)
19:    \{
20:      ConverseInit(argc, argv, user_main, 0, 0);
21:    \}
\end{alltt}

Lines 3-6 of this program contain a simple C function that prints an
integer.  The function is marked with the word {\tt CpmInvokable}.
When the CPM scanner sees this word, it adds the function {\tt
Cpm\_print\_integer} to the file {\tt myprog.cpm.h}.  The program
includes {\tt myprog.cpm.h} on line 1, and initializes the code in
there on line 12.  Each call to {\tt Cpm\_print\_integer} on line 15
builds a message that invokes {\tt print\_integer}.  The
destination-argument {\tt CpmSend(i)} causes the message to be sent to
the {\it i}'th processor.

The effect of this program is that the first processor orders each of
the other processors to print a random number.  Note that the example
is somewhat minimalist since it doesn't contain any code for
terminating itself.  Also note that it would have been more efficient
to use an explicit broadcast.  Broadcasts are described later.

All launchers accept a {\it CpmDestination} as their first argument.  A
{\it CpmDestination} is actually a pointer to a small C structure
containing routing and handling information.  The CPM module has many
built-in functions that return {\it CpmDestination}s.  Therefore, any
of these can be used as the first argument to a launcher:

\begin{itemize}

\item[]{{\bf CpmSend({\it pe})} - the message is transmitted to
processor {\it pe} with maximum priority}.

\item[]{{\bf CpmEnqueue({\it pe, queueing, priobits, prioptr})}
- The message is transmitted to processor {\it pe}, where it is
enqueued with the specified queueing strategy and priority.  The {\it
queueing}, {\it priobits}, and {\it prioptr} arguments are the same as
for {\bf CqsEnqueueGeneral}.}

\item[]{{\bf CpmEnqueueFIFO({\it pe})} - the message is transmitted to
processor {\it pe} and enqueued with the middle priority (zero), and
FIFO relative to messages with the same priority.}

\item[]{{\bf CpmEnqueueLIFO({\it pe})} - the message is transmitted to
processor {\it pe} and enqueued with the middle priority (zero), and
LIFO relative to messages with the same priority.}

\item[]{{\bf CpmEnqueueIFIFO({\it pe, prio})} - the message is
transmitted to processor {\it pe} and enqueued with the specified
integer-priority {\it prio}, and FIFO relative to messages with the
same priority.}

\item[]{{\bf CpmEnqueueILIFO({\it pe, prio})} - the message is
transmitted to processor {\it pe} and enqueued with the specified
integer-priority {\it prio}, and LIFO relative to messages with the
same priority.}

\item[]{{\bf CpmEnqueueBFIFO({\it pe, priobits, prioptr})} - the
message is transmitted to processor {\it pe} and enqueued with the
specified bitvector-priority, and FIFO relative to messages
with the same priority.}

\item[]{{\bf CpmEnqueueBLIFO({\it pe, priobits, prioptr})} - the
message is transmitted to processor {\it pe} and enqueued with the
specified bitvector-priority, and LIFO relative to messages with the
same priority.}

\item[]{{\bf CpmMakeThread({\it pe})} - The message is transmitted to
processor {\it pe} where a CthThread is created, and the thread
invokes the specified function.}

\end{itemize}

All the functions shown above accept processor numbers as arguments.
Instead of supplying a processor number, one can also supply the
special symbols CPM\_ALL or CPM\_OTHERS, causing a broadcast.  For
example,

\begin{alltt}
Cpm_print_integer(CpmMakeThread(CPM_ALL), 5);
\end{alltt}

would broadcast a message to all the processors causing each processor
to create a thread, which would in turn invoke {\tt print\_integer}
with the argument 5.

\section{CPM Packing and Unpacking}

Functions preceeded by the word {\bf CpmInvokable} must have simple
argument lists.  In particular, the argument list of a CpmInvokable
function can only contain cpm-single-arguments and cpm-array-arguments,
as defined by this grammar:

\begin{alltt}
    cpm-single-argument :== typeword varname
    cpm-array-argument  :== typeword '*' varname
\end{alltt}

When CPM sees the cpm-array-argument notation, CPM interprets it as
being a pointer to an array.  In this case, CPM attempts to pack an
entire array into the message, whereas it only attempts to pack a
single element in the case of the cpm-single-argument notation.

Each cpm-array-argument must be preceeded by a cpm-single-argument of
type {\tt CpmDim}.  {\tt CpmDim} is simply an alias for {\tt int}, but
when CPM sees an argument declared {\tt CpmDim}, it knows that the
next argument will be a cpm-array-argument, and it interprets the {\tt
CpmDim} argument to be the size of the array.  Given a pointer to the
array, its size, and its element-type, CPM handles the packing of
array values as automatically as it handles single values.

A second program, {\tt example2.c}, uses array arguments:

\begin{alltt}
 1:    #include "example2.cpm.h"
 2:   
 3:    CpmInvokable print_program_arguments(CpmDim argc, CpmStr *argv)
 4:    \{
 5:      int i;
 6:      CmiPrintf("The program's arguments are: ");
 7:      for (i=0; i<argc; i++) CmiPrintf("%s ", argv[i]);
 8:      CmiPrintf("\verb+\+n");
 9:    \}
10:
11:    user_main(int argc, char **argv)
12:    \{
13:      CpmModuleInit();
14:      CpmInitializeThisModule();
15:      if (CmiMyPe()==0)
16:        Cpm_print_program_arguments(CpmSend(1), argc, argv);
17:    \}
18:
19:    main(int argc, char **argv)
20:    \{
21:      ConverseInit(argc, argv, user_main, 0, 0);
22:    \}
\end{alltt}

The word {\tt CpmStr} is a CPM built-in type, it represents a
null-terminated string:

\begin{alltt}
	typedef char *CpmStr;
\end{alltt}

Therefore, the function {\tt print\_program\_arguments} takes exactly
the same arguments as {\tt user\_main}.  In this example, the main
program running on processor 0 transmits the arguments to processor 1,
which prints them out.

Thus far, we have only shown functions whose prototypes contain
builtin CPM types.  CPM has built-in knowledge of the following types:
char, short, int, long, float, double, CpmDim, and CpmStr (pointer to
a null-terminated string).  However, you may also also transmit
user-defined types in a CPM message.

For each (non-builtin) type the user wishes to pack, the user must
supply some pack and unpack routines.  The subroutines needed depend
upon whether the type is a pointer or a simple type.  Simple types are
defined to be those that contain no pointers at all.  Note that some
types are neither pointers, nor simple types.  CPM cannot currently
handle such types.

CPM knows which type is which only through the following declarations:

\begin{alltt}
    CpmDeclareSimple(typeword);
    CpmDeclarePointer(typeword);
\end{alltt}

The user must supply such declarations for each type that must be sent
via CPM.

When packing a value {\tt v} which is a simple type, CPM uses the
following strategy.  The generated code first converts {\tt v} to
network interchange format by calling {\tt CpmPack\_typename(\&v)},
which must perform the conversion in-place.  It then copies {\tt v}
byte-for-byte into the message and sends it.  When the data arrives,
it is extracted from the message and converted back using {\tt
CpmUnpack\_typename(\&v)}, again in-place.  The user must supply the
pack and unpack routines.

When packing a value {\tt v} which is a pointer, the generated code
determines how much space is needed in the message buffer by calling
{\tt CpmPtrSize\_typename(v)}.  It then transfers the data pointed to
by {\tt v} into the message using {\tt CpmPtrPack\_typename(p, v) },
where {\tt p} is a pointer to the allocated space in the message
buffer.  When the message arrives, the generated code extracts the
packed data from the message by calling {\tt
CpmPtrUnpack\_typename(p)}.  The unpack function must return a pointer
to the unpacked data, which is allowed to still contain pointers to
the message buffer (or simply be a pointer to the message buffer).
When the invocation is done, the function {\tt
CpmPtrFree\_typename(v)} is called to free any memory allocated by the
unpack routine. The user must supply the size, pack, unpack, and free
routines.

The following program fragment shows the declaration of two
user-defined types:

\begin{alltt}
 1:
 2:    typedef struct \{ double x,y; \} coordinate;    
 3:    CpmDeclareSimple(coordinate);
 4:    
 5:    void CpmPack_coordinate(coordinate *p)
 6:    \{
 7:      CpmPack_double(&(p->x));
 8:      CpmPack_double(&(p->y));
 9:    \}
10:
11:    void CpmPack_coordinate(coordinate *p)
12:    \{
13:      CpmUnpack_double(&(p->x));
14:      CpmUnpack_double(&(p->y));
15:    \}
16:
17:    typedef int *intptr;
18:    CpmDeclarePointer(intptr);
19:
20:    #define CpmPtrSize_intptr(p) sizeof(int)
21:    
22:    void CpmPtrPack_intptr(void *p, intptr v)
23:    \{
24:      *(int *)p = *v;
25:      CpmPack_int((int *)p);
26:    \}
27:
28:    intptr CpmPtrUnpack_intptr(void *p)
29:    \{
30:      CpmUnpack_int((int *)p);
31:      return (int *)p;
32:    \}
33:
34:    #define CpmPtrFree_intptr(p) (0)
35:
36:    #include "example3.cpm.h"
37:    ...
\end{alltt}

The first type declared in this file is the coordinate.  Line 2
contains the C type declaration, and line 3 notifies CPM that it is a
simple type, containing no pointers.  Lines 5-9 declare the pack
function, which receives a pointer to a coordinate, and must pack it
in place.  It makes use of the pack-function for doubles, which also
packs in place.  The unpack function is similar.

The second type declared in this file is the intptr, which we intend
to mean a pointer to a single integer.  On line 18 we notify CPM that
the type is a pointer, and that it should therefore use
CpmPtrSize\_intptr, CpmPtrPack\_intptr, CpmPtrUnpack\_intptr, and
CpmPtrFree\_intptr.  Line 20 shows the size function, a constant: we
always need just enough space to store one integer.  The pack function
copies the int into the message buffer, and packs it in place.  The
unpack function unpacks it in place, and returns an intptr, which
points right to the unpacked integer which is still in the message
buffer.  Since the int is still in the message buffer, and not in
dynamically allocated memory, the free function on line 34 doesn't
have to do anything.

Note that the inclusion of the {\tt .cpm.h} file comes after these
type and pack declarations: the {\tt .cpm.h} file will reference these
functions and macros, therefore, they must already be defined.

\section{Inventing New Types of CpmDestinations}

It is possible for the user to create new types of CpmDestinations,
and to write functions that return these new destinations.  In order
to do this, one must have a mental model of the steps performed when a
Cpm message is sent.  This knowledge is only necessary to those
wishing to invent new kinds of destinations.  Others can skip this
section.

The basic steps taken when sending a CPM message are:

\begin{itemize}

\item[]{{\bf 1. The destination-structure is created.}  The first
argument to the launcher is a CpmDestination.  Therefore, before the
launcher is invoked, one typically calls a function (like CpmSend)
to build the destination-structure.}

\item[]{{\bf 2. The launcher allocates a message-buffer}.  The buffer
contains space to hold a function-pointer and the function's arguments.
It also contains space for an ``envelope'', the size of which is
determined by a field in the destination-structure.}

\item[]{{\bf 3. The launcher stores the function-arguments in the message
buffer}.  In doing so, the launcher converts the arguments to a
contiguous sequence of bytes.}

\item[]{{\bf 4. The launcher sets the message's handler}.  For every
launcher, there is a matching function called an {\it invoker}.  The
launcher's job is to put the argument data in the message and send the
message.  The {\it invoker}'s job is to extract the argument data from
the message and call the user's function.  The launcher uses {\tt
CmiSetHandler} to tell \converse{} to handle the message by calling the
appropriate {\it invoker}.}

\item[]{{\bf 5. The message is sent, received, and handled}. 
The destination-structure contains a pointer to a {\it send-function}.
The {\it send-function} is responsible for choosing the message's
destination and making sure that it gets there and gets handled.  The
{\it send-function} has complete freedom to implement this in any
manner it wishes.  Eventually, though, the message should arrive at a
destination and its handler should be called.}

\item[]{{\bf 6. The user's function is invoked}.  The invoker
extracts the function arguments from the message buffer and calls
the user's function.}

\end{itemize}

The {\it send-function} varies because messages take different
routes to get to their final destinations.  Compare, for example,
CpmSend to CpmEnqueueFIFO.  When CpmSend is used, the message goes
straight to the target processor and gets handled.  When
CpmEnqueueFIFO is used, the message goes to the target processor, goes
into the queue, comes out of the queue, and {\it then} gets handled.
The {\it send-function} must implement not only the transmission of
the message, but also the possible ``detouring'' of the message
through queues or into threads.

We now show an example CPM command, and describe the steps that are
taken when the command is executed.  The command we will consider is
this one:

\begin{alltt}
Cpm_print_integer(CpmEnqueueFIFO(3), 12);
\end{alltt}

Which sends a message to processor 3, ordering it to call {\tt
print\_integer(12)}.

The first step is taken by CpmEnqueueFIFO, which builds the
CpmDestination.  The following is the code for CpmEnqueueFIFO:

\pagebreak

\begin{alltt}
typedef struct CpmDestinationSend
\{
  void *(*sendfn)();
  int envsize;
  int pe;
\}
*CpmDestinationSend;

CpmDestination CpmEnqueueFIFO(int pe)
\{
  static struct CpmDestinationSend ctrl;
  ctrl.envsize = sizeof(int);
  ctrl.sendfn  = CpmEnqueueFIFO1;
  ctrl.pe = pe;
  return (CpmDestination)&ctrl;
\}
\end{alltt}

Notice that the CpmDestination structure varies, depending upon which
kind of destination is being used.  In this case, the destination
structure contains a pointer to the send-function {\tt
CpmEnqueueFIFO1}, a field that controls the size of the envelope, and
the destination-processor.  In a CpmDestination, the {\tt sendfn} and
{\tt envsize} fields are required, additional fields are optional.

After CpmEnqueueFIFO builds the destination-structure, the launcher
Cpm\_print\_integer is invoked.  Cpm\_print\_integer performs all the
steps normally taken by a launcher:

\begin{itemize}

\item[]{{\bf 1. It allocates the message buffer.}  In this case, it sets aside
just enough room for one {\it int} as an envelope, as dictated by the
destination-structure's {\it envsize} field.}

\item[]{{\bf 2. It stores the function-arguments in the message-buffer.}  In
this case, the function-arguments are just the integer 12.}

\item[]{{\bf 3. It sets the message's handler.}  In this case, the message's
handler is set to a function that will extract the arguments and call
print\_integer.}

\item[]{{\bf 4. It calls the send-function to send the message.}}
\end{itemize}

The code for the send-function is here:

\begin{alltt}
void *CpmEnqueueFIFO1(CpmDestinationSend dest, int len, void *msg)
\{
  int *env = (int *)CpmEnv(msg);
  env[0] = CmiGetHandler(msg);
  CmiSetHandler(msg, CpvAccess(CpmEnqueueFIFO2_Index));
  CmiSyncSendAndFree(dest->pe,len,msg);
\}
\end{alltt}

The send-function CpmEnqueueFIFO1 starts by switching the handler.
The original handler is removed using using {\tt CmiGetHandler}.  It
is set aside in the message buffer in the ``envelope'' space described
earlier --- notice the use of {\tt CpmEnv} to obtain the envelope.
This is the purpose of the envelope in the message --- it is a place
where the send-function can store information.  The
destination-function must anticipate how much space the send-function
will need, and it must specify that amount of space in the
destination-structure field {\it envsize}.  In this case, the envelope
is used to store the original handler, and the message's handler is
set to an internal function called {\tt CpmEnqueueFIFO2}.

After switching the handler, {\tt CpmEnqueueFIFO1} sends the message.
Eventually, the message will be received by {\tt CsdScheduler}, and
its handler will be called.  The result will be that {\tt
CpmEnqueueFIFO2} will be called on the destination processor.  Here is
the code for {\tt CpmEnqueueFIFO2}:

\begin{alltt}
void CpmEnqueueFIFO2(void *msg)
\{
  int *env;
  CmiGrabBuffer(&msg);
  env = (int *)CpmEnv(msg);
  CmiSetHandler(msg, env[0]);
  CsdEnqueueFIFO(msg);
\}
\end{alltt}

This function takes ownership of the message-buffer from \converse{}
using {\tt CmiGrabBuffer}.  It extracts the original handler from the
envelope (the handler that calls {\tt print\_integer}), and restores it
using {\tt CmiSetHandler}.  Having done so, it enqueues the message
with the FIFO queueing policy.  Eventually, the scheduler picks the
message from the queue, and {\tt print\_integer} is invoked.

In summary, the procedure for implementing new kinds of destinations
is to write one send-function, one function returning a CpmDestination
(which contains a reference to the send-function), and one or more
\converse{} handlers to manipulate the message.

The destination-function must return a pointer to a
``destination-structure'', which can in fact be any structure matching
the following specifications:

\begin{itemize}
\item{The first field must be a pointer to a send-function,}
\item{The second field must the an integer, the envelope-size.}
\end{itemize}

This pointer must be coerced to type CpmDestination.

The send-function must have the following prototype:

\begin{alltt}
    void sendfunction(CpmDestination dest, int msglen, void *msgptr)
\end{alltt}

It can access the envelope of the message using CpmEnv:

\begin{alltt}
    int *CpmEnv(void *msg);
\end{alltt}

It can also access the data stored in the destination-structure
by the destination-function.

