\section{Messages}

\label{messages}
Although \charmpp{} supports automated parameter marshalling for entry methods,
you can also manually handle the process of packing and unpacking parameters by
using messages. 
%By using messages, you can potentially improve performance by
%avoiding unnecessary copying. 
A message encapsulates all the parameters sent to an
entry method.  Since the parameters are already encapsulated,
sending messages is often more efficient than parameter marshalling, and
can help to avoid unnecessary copying.
Moreover, assume that the receiver is unable to process the contents of the
message at the time that it receives it. For example, consider a 
tiled matrix multiplication program, wherein each chare receives an $A$-tile
and a $B$-tile before computing a partial result for $C = A \times B$. If we 
were using parameter marshalled entry methods, a chare would have to copy the first
tile it received, in order to save it for when it has both the tiles it needs. 
Then, upon receiving the second
tile, the chare would use the second tile and the first (saved) tile to 
compute a partial result. However, using messages, we would just save a {\em pointer} 
to the message encapsulating the tile received first, instead of the tile data itself.

\vspace{0.1in}
\noindent
{\bf Managing the memory buffer associated with a message.}
As suggested in the example above, the biggest difference between marshalled parameters and messages
is that an entry method invocation is assumed to {\em keep} the message that it
is passed. That is, the \charmpp{} runtime system assumes that code in the body of the invoked
entry method will explicitly manage the memory associated with the message that it is passed. Therefore,
in order to avoid leaking memory, the body of an entry method must either \kw{delete} the message that
it is receives, or save a pointer to it, and \kw{delete} it a later point in the execution of the code.
%is code written for the body of an 
%either store the passed message or explicitly {\em delete} it, or else the message
%will never be destroyed, wasting memory.

Moreover, in the \charm{} execution model, once you pass a message buffer to the runtime system (via 
an asynchronous entry method invocation), you should {\em not} reuse the buffer. That is, after you have
passed a message buffer into an asynchronous entry method invocation, you shouldn't 
access its fields, or pass that same buffer into a second entry method invocation. Note that this rule
doesn't preclude the {\em single reuse} of an input message -- consider an entry method invocation
$i_1$, which receives as input the message buffer $m_1$. Then, $m_1$ may be passed to an 
asynchronous entry method invocation $i_2$. However, once $i_2$ has been issued with $m_1$ as its input
parameter, $m_1$ cannot be used in any further entry method invocations.
%message buffer, that message buffer may in turn be passed to an entry method invocation that accepts a 
%message of the same type. However, 
 
%Thus each entry method must be passed a {\em new} message.

Several kinds of message are available.
Regular \charmpp{} messages are objects of
\textit{fixed size}. One can have messages that contain pointers or variable
length arrays (arrays with sizes specified at runtime) and still have these
pointers as valid when messages are sent across processors, with some
additional coding.  Also available is a mechanism for assigning
\textit{priorities} to a message regardless of its type.
A detailed discussion of priorities appears later in this section.

\subsection{Message Types}

\smallskip
\noindent {\bf Fixed-Size Messages.}
The simplest type of message is a {\em fixed-size} message. The size of each data member
of such a message should be known at compile time. Therefore, such a message may encapsulate
primitive data types, user-defined data types that {\em don't} maintain pointers to memory
locations, and {\em static} arrays of the aforementioned types. 

\smallskip
\noindent {\bf Variable-Size Messages.}
%An ordinary message in \charmpp\ is a fixed size message that is allocated
%internally with an envelope which encodes the size of the message. 
Very often,
the size of the data contained in a message is not known until runtime. 
%One can
%use packed\index{packed messages} messages to alleviate this problem.  However,
%it requires multiple memory allocations (one for the message, and another for
%the buffer.) 
For such scenarious, you can use variable-size (\emph{varsize}) messages.
A {\em varsize} message can encapsulate several arrays,
each of whose size is determined at run time. 
%In \emph{varsize} messages, 
The space required for these encapsulated, variable length arrays
is allocated with the entire message comprises a 
contiguous buffer of memory.
%message such that it is contiguous to the message.

\smallskip
\noindent {\bf Packed Messages.} A {\em packed} message is used to communicate non-linear
data structures via messages. However, we defer a more detailed description of their use
to \S~\ref{sec:messages/packed_msgs}.

\subsection{Using Messages In Your Program}

There are five steps to incorporating a (fixed or varsize) message type in your \charmpp{} program:
(1) Declare message type in \kw{.ci} file; (2) Define message type in \kw{.h} file;
(3) Allocate message; (4) Pass message to asynchronous entry method invocation and (5) Deallocate
message to free associated memory resources. 

\medskip
\noindent {\bf Declaring Your Message Type.}
Like all other entities involved in asynchronous entry method invocation, messages
must be declared in the {\tt .ci} file. 
This allows the \charmpp{} translator 
to generate support code for messages. 
Message declaration is straightforward for fixed-size messages. Given a 
message of type {\tt MyFixedSizeMsg}, simply include the following in the \kw{.ci} file:

\begin{alltt}
 message MyFixedSizeMsg;
\end{alltt}

For varsize messages, the \kw{.ci} declaration must also include the names and
types of the variable-length arrays that the message will encapsulate. The
following example illustrates this requirement. In it, a message of type {\tt
MyVarsizeMsg}, which encapsulates three variable-length arrays of different
types, is declared:

\begin{alltt}
 message MyVarsizeMsg \{
   int arr1[];
   double arr2[];
   MyPointerlessStruct arr3[];
 \};
\end{alltt}

\medskip
\noindent {\bf Defining Your Message Type.}
Once a message type has been declared to the \charmpp{} translator, its type definition must be provided.
Your message type must inherit from a specific generated base class. If the type of 
your message is {\tt T}, then {\tt class T} must inherit from {\tt CMessage\_T}.
This is true for both fixed and varsize messages.
As an example, for our fixed size message
type {\tt MyFixedSizeMsg} above, we might write the following in the \kw{.h} file:

\begin{alltt}
class MyFixedSizeMsg : public CMessage_MyFixedSizeMsg \{
  int var1;
  MyPointerlessStruct var2;
  double arr3[10];

  // Normal C++ methods, constructors, etc. go here
\};
\end{alltt}

In particular, note the inclusion of the static array of {\tt double}s, {\tt arr3}, whose size
is known at compile time to be that of ten {\tt double}s.
Similarly, for our example varsize message of type {\tt MyVarsizeMsg}, we would write something
like:

\begin{alltt}
class MyVarsizeMsg : public CMessage_MyVarsizeMsg \{
  // variable-length arrays
  int *arr1;
  double *arr2;
  MyPointerlessStruct *arr3;
  
  // members that are not variable-length arrays 
  int x,y;
  double z;

  // Normal C++ methods, constructors, etc. go here
\};
\end{alltt}

Note that the \kw{.h} definition of the class type must contain data members
whose names and types match those specified in the \kw{.ci} declaration.  In
addition, if any of data members are \kw{private} or \kw{protected}, it should
declare class \uw{CMessage\_MyVarsizeMsg} to be a \kw{friend} class.  Finally,
there are no limitations on the member methods of message classes, except that
the message class may not redefine operators \texttt{new} or \texttt{delete}.


Thus the \uw{mtype} class
declaration should be similar to:

\medskip
\noindent {\bf Creating a Message.}
With the \kw{.ci} declaration and \kw{.h} definition in place, messages can be allocated and 
used in the program.
\index{message}Messages are allocated using the \CC\ \kw{new} operator:

\begin{alltt}
 MessageType *msgptr =
  new [(int sz1, int sz2, ... , int priobits=0)] MessageType[(constructor arguments)];
\end{alltt}

The arguments enclosed within the square brackets are optional, and 
are used only when allocating messages
with variable length arrays or prioritized messages.
These arguments are not specified for fixed size messages. 
For instance, to allocate a message of our example message 
{\tt MyFixedSizeMsg}, we write:

\begin{alltt}
MyFixedSizeMsg *msg = new MyFixedSizeMsg(<constructor args>);
\end{alltt}

In order to allocate a varsize message, we must pass appropriate
values to the arguments of the overloaded \kw{new} operator presented previously. 
Arguments \uw{sz1, sz2, ...}
denote the size (in number of elements) of the memory blocks that need to be
allocated and assigned to the pointers (variable-length arrays) that the message contains. The
\uw{priobits} argument denotes the size of a bitvector (number of bits) that
will be used to store the message priority.   
So, if we wanted to create {\tt MyVarsizeMsg} whose 
{\tt arr1},  {\tt arr2} and {\tt arr3} arrays contain
10, 20 and 7 elements of their respective types, we would write:

\begin{alltt}
MyVarsizeMsg *msg = new (10, 20, 7) MyVarsizeMsg(<constructor args>);
\end{alltt}

%This allocates a \uw{VarsizeMessage}, in which \uw{firstArray} points to an
%array of 10 ints and \uw{secondArray} points to an array of 20 doubles.  This
%is explained in detail in later sections. 

Further, to add a 32-bit \index{priority}priority bitvector to this message, we would write:

\begin{alltt}
MyVarsizeMsg *msg = new (10, 20, 7, sizeof(uint32_t)*8) VarsizeMessage;
\end{alltt}

Notice the last argument to the overloaded \kw{new} operator, which specifies 
the number of bits used to store message priority.
The section on prioritized execution (\S~\ref{prioritized message passing}) describes how
priorities can be employed in your program.

Another version of the overloaded \kw{new} operator allows you to pass in
an array containing the size of each variable-length array, rather than specifying 
individual sizes as separate arguments. 
For example, we could create a message of type {\tt MyVarsizeMsg} in the following manner:

\begin{alltt}
int sizes[3];
sizes[0] = 10;               // arr1 will have 10 elements
sizes[1] = 20;               // arr2 will have 20 elements 
sizes[2] = 7;                // arr3 will have 7 elements 

MyVarsizeMsg *msg = new(sizes, 0) MyVarsizeMsg(<constructor args>); // 0 priority bits
\end{alltt}

%In Section~\ref{message packing} we explain how messages can contain arbitrary
%pointers, and how the validity of such pointers can be maintained across
%processors in a distributed memory machine.


\medskip
\noindent {\bf Sending a Message.}
Once we have a properly allocated message, 
we can set the various elements of the encapsulated arrays in the following manner:

\begin{alltt}
  msg->arr1[13] = 1;
  msg->arr2[5] = 32.82;
  msg->arr3[2] = MyPointerlessStruct();
  // etc.
\end{alltt}

And pass it to an asynchronous entry method invocation, thereby sending it to the 
corresponding chare:

\begin{alltt}
myChareArray[someIndex].foo(msg);
\end{alltt}

When a message \index{message} is {\em sent}, i.e.  passed to an asynchronous
entry method invocation, the programmer relinquishes control of it; the space
allocated for the message is freed by the runtime system.  However, when a
message is {\em received} at an entry point, it is {\em not} freed by the
runtime system.  As mentioned at the start of this section, received
messages may be reused or deleted by the programmer.  Finally, messages are
deleted using the standard \CC{} \kw{delete} operator.  

\zap{
\subsubsection{Messages with Variable Length Arrays}

\label{varsize messages}
\index{variable size messages}
\index{varsize message}

Such a message is declared as 

\begin{alltt}
 message mtype \{
   type1 var_name1[];
   type2 var_name2[];
   type3 var_name3[];
 \};
\end{alltt}

in \charmpp\ interface file. 
\begin{alltt}
class mtype : public CMessage_mtype \{
 private:
   type1 *var_name1;
   type2 *var_name2;
   type3 *var_name3;
   friend class CMessage_mtype;
\};
\end{alltt}

\small
\hrule

\noindent\textbf{An Example}

Suppose a \charmpp\ message contains two variable length arrays of types
\texttt{int} and \texttt{double}:

\begin{alltt} 
class VarsizeMessage: public CMessage_VarsizeMessage \{
  public:
    int lengthFirst;
    int lengthSecond;
    int* firstArray;
    double* secondArray;
    // other functions here
\};
\end{alltt}

Then in the \texttt{.ci} file, this has to be declared as: 

\begin{alltt}
message VarsizeMessage \{
  int firstArray[];
  double secondArray[];
\};
\end{alltt}

We specify the types and actual names of the fields that
contain variable length arrays. The dimensions of these arrays are NOT
specified in the interface file, since they will be specified in the
constructor of the message when message is created. In the {\tt .h} or {\tt .C}
file, this class is declared as:

\begin{alltt} 

class VarsizeMessage : public CMessage_VarsizeMessage \{ 
  public: 
    int lengthFirst;
    int lengthSecond;
    int* firstArray;
    double* secondArray;
    // other functions here
\};
\end{alltt}

The interface translator generates the \uw{CMessage\_VarsizeMessage} class,
which contains code to properly allocate, pack and unpack the
\uw{VarsizeMessage}.


One can allocate messages of the \uw{VarsizeMessage} class as follows:

\begin{alltt}
// firstArray will have 4 elements
// secondArray will have 5 elements 
VarsizeMessage* p = new(4, 5, 0) VarsizeMessage;
p->firstArray[2] = 13;     // the arrays have already been allocated 
p->secondArray[4] = 6.7; 
\end{alltt}

\hrule
\normalsize
No special handling is needed for deleting varsize messages.
} % end zap

\subsection{Message Packing}

\label{message packing}
\index{message packing}

The \charmpp{} interface translator generates implementation for three static
methods for the message class \uw{CMessage\_mtype}. These methods have the
prototypes:

\begin{alltt}
    static void* alloc(int msgnum, size_t size, int* array, int priobits);
    static void* pack(mtype*);
    static mtype* unpack(void*);
\end{alltt}

One may choose not to use the translator-generated methods and may override
these implementations with their own \uw{alloc}, \uw{pack} and \uw{unpack}
static methods of the \uw{mtype} class.  The \kw{alloc} method will be called
when the message is allocated using the \CC\ \kw{new} operator. The programmer
never needs to explicitly call it.  Note that all elements of the message are
allocated when the message is created with \kw{new}. There is no need to call
\kw{new} to allocate any of the fields of the message. This differs from a
packed message where each field requires individual allocation. The \kw{alloc}
method should actually allocate the message using \kw{CkAllocMsg}, whose
signature is given below:

\begin{alltt}
void *CkAllocMsg(int msgnum, int size, int priobits); 
\end{alltt}  


For varsize messages, these static methods \texttt{alloc}, \texttt{pack}, and 
\texttt{unpack} are
generated by the interface translator.  For example, these
methods for the \kw{VarsizeMessage} class above would be similar to:

\begin{alltt}
// allocate memory for varmessage so charm can keep track of memory
static void* alloc(int msgnum, size_t size, int* array, int priobits)
\{
  int totalsize, first_start, second_start;
  // array is passed in when the message is allocated using new (see below).
  // size is the amount of space needed for the part of the message known
  // about at compile time.  Depending on their values, sometimes a segfault
  // will occur if memory addressing is not on 8-byte boundary, so altered
  // with ALIGN8
  first_start = ALIGN8(size);  // 8-byte align with this macro
  second_start = ALIGN8(first_start + array[0]*sizeof(int));
  totalsize = second_start + array[1]*sizeof(double);
  VarsizeMessage* newMsg = 
    (VarsizeMessage*) CkAllocMsg(msgnum, totalsize, priobits);
  // make firstArray point to end of newMsg in memory
  newMsg->firstArray = (int*) ((char*)newMsg + first_start);
  // make secondArray point to after end of firstArray in memory
  newMsg->secondArray = (double*) ((char*)newMsg + second_start);

  return (void*) newMsg;
\}

// returns pointer to memory containing packed message
static void* pack(VarsizeMessage* in)
\{
  // set firstArray an offset from the start of in
  in->firstArray = (int*) ((char*)in->firstArray - (char*)in);
  // set secondArray to the appropriate offset
  in->secondArray = (double*) ((char*)in->secondArray - (char*)in);
  return in;
\}

// returns new message from raw memory
static VarsizeMessage* VarsizeMessage::unpack(void* inbuf)
\{
  VarsizeMessage* me = (VarsizeMessage*)inbuf;
  // return first array to absolute address in memory
  me->firstArray = (int*) ((size_t)me->firstArray + (char*)me);
  // likewise for secondArray
  me->secondArray = (double*) ((size_t)me->secondArray + (char*)me);
  return me;
\}
\end{alltt}

The pointers in a varsize message can exist in two states.  At creation, they
are valid \CC\ pointers to the start of the arrays.  After packing, they become
offsets from the address of the pointer variable to the start of the pointed-to
data.  Unpacking restores them to pointers. 

\subsubsection{Custom Packed Messages}
\label{sec:messages/packed_msgs}

\index{packed messages}

In many cases, a message must store a {\em non-linear} data structure using
pointers.  Examples of these are binary trees, hash tables etc. Thus, the
message itself contains only a pointer to the actual data. When the message is
sent to the same processor, these pointers point to the original locations,
which are within the address space of the same processor. However, when such a
message is sent to other processors, these pointers will point to invalid
locations.

Thus, the programmer needs a way to ``serialize'' these messages
\index{serialized messages}\index{message serialization}{\em only if} the
message crosses the address-space boundary.  \charmpp{} provides a way to do
this serialization by allowing the developer to override the default
serialization methods generated by the \charmpp{} interface translator.
Note that this low-level serialization has nothing to do with parameter
marshalling or the PUP framework described later.

Packed messages are declared in the {\tt .ci} file the same way as ordinary
messages:

\begin{alltt}
message PMessage;
\end{alltt}

Like all messages, the class \uw{PMessage} needs to inherit from
\uw{CMessage\_PMessage} and should provide two {\em static} methods: \kw{pack}
and \kw{unpack}. These methods are called by the \charmpp\ runtime system, when
the message is determined to be crossing address-space boundary. The prototypes
for these methods are as follows:

\begin{alltt}
static void *PMessage::pack(PMessage *in);
static PMessage *PMessage::unpack(void *in);
\end{alltt}

Typically, the following tasks are done in \kw{pack} method:

\begin{itemize}

\item Determine size of the buffer needed to serialize message data.

\item Allocate buffer using the \kw{CkAllocBuffer} function. This function
takes in two parameters: input message, and size of the buffer needed, and
returns the buffer.

\item Serialize message data into buffer (alongwith any control information
needed to de-serialize it on the receiving side.

\item Free resources occupied by message (including message itself.)  

\end{itemize}

On the receiving processor, the \kw{unpack} method is called. Typically, the
following tasks are done in the \kw{unpack} method:

\begin{itemize}

\item Allocate message using \kw{CkAllocBuffer} function. {\em Do not
use \kw{new} to allocate message here. If the message constructor has
to be called, it can be done using the in-place \kw{new} operator.}

\item De-serialize message data from input buffer into the allocated message.

\item Free the input buffer using \kw{CkFreeMsg}.

\end{itemize}

Here is an example of a packed-message implementation:

\begin{alltt}
// File: pgm.ci
mainmodule PackExample \{
  ...
  message PackedMessage;
  ...
\};

// File: pgm.h
...
class PackedMessage : public CMessage_PackedMessage
\{
  public:
    BinaryTree<char> btree; // A non-linear data structure 
    static void* pack(PackedMessage*);
    static PackedMessage* unpack(void*);
    ...
\};
...

// File: pgm.C
...
void*
PackedMessage::pack(PackedMessage* inmsg)
\{
  int treesize = inmsg->btree.getFlattenedSize();
  int totalsize = treesize + sizeof(int);
  char *buf = (char*)CkAllocBuffer(inmsg, totalsize);
  // buf is now just raw memory to store the data structure
  int num_nodes = inmsg->btree.getNumNodes();
  memcpy(buf, &num_nodes, sizeof(int));  // copy numnodes into buffer
  buf = buf + sizeof(int);               // don't overwrite numnodes
  // copies into buffer, give size of buffer minus header
  inmsg->btree.Flatten((void*)buf, treesize);    
  buf = buf - sizeof(int);              // don't lose numnodes
  delete inmsg;
  return (void*) buf;
\}

PackedMessage*
PackedMessage::unpack(void* inbuf)
\{
  // inbuf is the raw memory allocated and assigned in pack
  char* buf = (char*) inbuf;
  int num_nodes;
  memcpy(&num_nodes, buf, sizeof(int));
  buf = buf + sizeof(int);
  // allocate the message through Charm RTS
  PackedMessage* pmsg = 
    (PackedMessage*)CkAllocBuffer(inbuf, sizeof(PackedMessage));
  // call "inplace" constructor of PackedMessage that calls constructor
  // of PackedMessage using the memory allocated by CkAllocBuffer,
  // takes a raw buffer inbuf, the number of nodes, and constructs the btree
  pmsg = new ((void*)pmsg) PackedMessage(buf, num_nodes);  
  CkFreeMsg(inbuf);
  return pmsg;
\}
... 
PackedMessage* pm = new PackedMessage();  // just like always 
pm->btree.Insert('A');
...
\end{alltt}


While serializing an arbitrary data structure into a flat buffer, one must be
very wary of any possible alignment problems.  Thus, if possible, the buffer
itself should be declared to be a flat struct.  This will allow the \CC\
compiler to ensure proper alignment of all its member fields.



\subsubsection{Immediate Messages}

Immediate messages are special messages that skip the Charm scheduler, they
can be executed in an ``immediate'' fashion even in the middle of 
a normal running entry method. 
They are supported only in nodegroup.



