\subsection{Communication Optimizations Framework}

The communication framework in Charm++/Converse is aimed at optimizing certain
communication patterns. Currently the programmer has to specify the
communication pattern it intends to optimize, together with the strategy to be
used. The communications library uses the delegation framework
(\ref{delegation}) in order to enable easy and transparent access to the
framework by the programmer.

For \ampi{} programs, the communication optimization is done by the \ampi{}
layer, so that the user does not need to worry about that. In Charm++, however,
the user must create the strategies in the program explicitly. Charm++ programs
are normally based on communicating arrays of chares, that compute and then
invoke entry methods on local or remote chares by sending them messages. These
array elements send messages to each other through proxies. The messages are
passed to the Charm++ runtime which calls lower level network APIs to
communicate. To optimize communication in Charm++, the user can redirect a
communication {\em call} to go through an instance of a strategy.

To access the communication framework, the user first creates and initializes a
communication library strategy. He then needs to make a copy of the array proxy
and associate it with that strategy. In order to use the framework, the
receiving entry methods need to receive messages (see \ref{messages}), and not
marshalled parameters. The user can create several instances of the same or
different strategies, to optimize different communication calls in the
application. In order to access the class signatures, the file ``comlib.h''
should be included.

Each communication operation is associated with a proxy, through which the
message is sent. These proxies can be associated in the mainchare constructor
(useful for all-to-all strategies), or later in the single chare
array elements (useful for section multicasts). In both cases, some
information has to be kept, either the CProxy or the ComlibInstanceHandle, and
this can be done in readonly variables, or as internal variables of the objects.

An example on how to use commlib can be found in the charm distribution, under
\examplerefdir{commlib/multicast}, where the proxies are associated in the
chare arrays.


\subsubsection{Using commlib}

One thing typically useful is having the the proxy associated with the strategy,
or an instance of the strategy (to be used for future associations) to be
declared as readonly variable, although this in not necessary. This is done by
declaring them readonly (see \ref{readonly} for more information).

\begin{alltt}
  readonly CProxy\_MyArray aproxy;
  readonly CProxy\_MyArray comlibproxy;
  readonly ComlibInstanceHandle cinst;
\end{alltt}

The creation of all the strategies needed, and their registration must be done
in the constructor of the mainchare (for more on array creation see
\ref{advanced array create}):

\begin{alltt}
  // Create the array
  aproxy = CProxy\_MyArray::ckNew();

  // Create the strategy (description of constructors later)
  CharmStrategy *strategy = new EachToManyMulticastStrategy(USE_MESH, srcarray, destarray);
  //or
  CharmStrategy *strategy = new StreamingStrategy(10,10);

  // Either associate the strategy with the proxy we declared, or register the
  // strategy to commlib for future association, or both.
  comlibproxy = aproxy;
  ComlibAssociateProxy(strategy, comlibproxy);

  cinst = ComlibRegister(strategy);
\end{alltt}

In this example, after aproxy has been associated with comlibproxy it can only be
used with commlib, and cannot send anymore regular messages. For this, if
regular messages without commlib are desired, a copy of the original proxy
should be made (like here).

In the chare array element, if {\texttt{cinst}} has been defined, other proxies
can be created and associated, like here a CProxySection\_MyArray, which allows
to send multicasts (see \ref{array section} for more on section proxies).

\begin{alltt}
  CProxySection\_MyArray mysection = .....
  ComlibAssociateProxy(\&cinst, mysection);  // mysection will always use commlib
\end{alltt}

After a proxy has been associated in some way to commlib, it can be used to send
messages with commlib:

\begin{alltt}
  comlibproxy.receive(msg); // send with commlib
  mysection.compute(msg);   // send to a section with commlib

  aproxy[0].single(msg);    // send to a single element {\textbf{without}} commlib
\end{alltt}

In case a bracketed strategy is used, two additional function calls have to
be added before starting to send the messages and after finishing. These are
discussed later in \ref{bracketed strategies}.

The signatures of the functions used here are the following:

\begin{alltt}
ComlibRegister (CharmStrategy *strat);
ComlibAssociateProxy (CharmStrategy *strat, CProxy &proxy);
ComlibAssociateProxy (ComlibInstanceHandle *cinst, CProxy &proxy);
\end{alltt}

%% \ subsubsection{Proxy interface}

%% A proxy containing the delegation should be kept, and reused every time the
%% associated stragety wants to be used.

%% \ begin{enumerate}
%% \ item main.C global
%% \ begin{alltt}
%% // Include the appropriate header file
%% #include <EachToManyMulticastStrategy.h>
%% #include <StreamingStrategy.h>

%% // Declare the global variable
%% CProxy_MyArray aproxy;
%% CProxy_MyArray dproxy;
%% \ end{alltt}

%% \ item main.C:main()
%% \ begin{alltt}
%% // Create the array
%% aproxy = CProxy_Hello::ckNew();

%% // Create the strategy (description of constructors later)
%% CharmStrategy *strategy = new EachToManyMulticastStrategy(USE_MESH, srcarray, destarray);
%% //or
%% CharmStrategy *strategy = new StreamingStrategy(10,10);

%% // Register the strategy to a new proxy, so that aproxy is without commlib,
%% // while dproxy uses it
%% dproxy = aproxy;
%% ComlibAssociateProxy(strategy, dproxy);
%% \ end{alltt}

%% \ item In the array element
%% \ begin{alltt}
%% // First proxy should be delegated
%% ComlibBeginIteration(dproxy);   // Only for bracketed strategies
%% dproxy[index].entry(message);   // Sending a message
%% .....     //sending more messages
%% .....
%% aproxy[index].entry2{message2); // Send a message without commlib
%% ComlibEndIteration(dproxy);     // Only for bracketed strategies
%% \ end{alltt}
%% \ end{enumerate}

%% The above example shows the usage of EachToManyStrategy. Notice the
%% ComlibBeginIteration and ComlibEndIteration calls, needed for bracketed
%% strategies. The construction of the strategies has been done in tha main::main,
%% from where they are broadcasted and initialized in every processor before being
%% used.

%% \ subsubsection{Instance interface}

%% In this interface, the chares need to keep information about the commlib instance.

%% \ begin{enumerate}
%% \ item main.C global
%% \ begin{alltt}
%% // Include the appropriate header file
%% #include <EachToManyMulticastStrategy.h>
%% #include <StreamingStrategy.h>

%% // Declare the global variable
%% CProxy_MyArray aproxy;
%% ComlibInstanceHandle cinst;
%% \ end{alltt}

%% \ item main.C:main()
%% \ begin{alltt}
%% // Create the array
%% aproxy = CProxy_Hello::ckNew();

%% // Create your strategy
%% Strategy *strategy = new EachToManyStrategy(USE_MESH, srcarray, destarray);
%% //or
%% Strategy *strategy = new StreamingStrategy(10,10);

%% // Create a Communication Library Instance
%% cinst = CkGetComlibInstance();

%% // Register the strategy
%% cinst.setStrategy(strategy);
%% \ end{alltt}

%% \ item In the array element 
%% \ begin{alltt}
%% // Before calling an entry method whose message should go thorough the
%% // library the proxy has to be delegated. Create a new copy of the
%% // proxy and delegate it before using it.
%% CProxy_Hello dproxy = aproxy;
%% ComlibDelegateProxy(&dproxy); //Now all calls to dproxy will go through the library.

%% cinst.beginIteration();       // Only for bracketed strategies
%% dproxy[index].entry();        // Send a message
%% .....
%% .....
%% aproxy[index].entry2();       // Send a message without commlib
%% cinst.endIteration();         // Only for bracketed strategies
%% \ end{alltt}
%% \ end{enumerate}

%% \ subsubsection{Sections}

%% In order to multicast only to a part of the array instead of the entire array,
%% it is necessary to create a {\ textrm{CProxySection\ _class}}
%% (\ ref{array_section_multicast}) of the desired portion of the destination array,
%% delegate it with {\ textrm{ComlibAssociateProxy()}}, and send a broadcast to it.
%% This broadcast on the section will result in the desired multicast on the global
%% array. Only multicast strategies can be used for this.


\subsubsection{Loadbalancing and Migration support}

The Communication optimization framework supports both loadbalancing and array
migration. It enables migration through message forwarding. Messages sent by a
migrated array are forwarded to the processor where it is mapped to, and from
here they get accounted. Messages sent to migrated arrays are forwarded from the
processor where they are mapped to their current destination.

This mapping of array elements to processors can be updated by the user by
calling {\textrm{ComlibResetProxy}} for array proxies, and
{\textrm{ComlibResetSectionProxy}} for section proxies. This should be done
especially during load-balancing, where most of the migrations happen. As shown
in the following example, these calls should be made inside the
{\textrm{resumeFromSync}} method.

\begin{alltt}
  void arrayelement::resumeFromSync() {
      .......
      .......
      ComlibResetProxy(comlibproxy);
      ComlibResetSectionProxy(mysection);
  }
\end{alltt}

A migrating array element containing {\em associated proxies} or {\em
instances} should pup them all at the source and destination.

\begin{alltt}
  void arrayelement::pup(PUP::er &p){
      ..........
      ..........
      p | mysection;
      p | cinst;
  }
\end{alltt}

\subsubsection{Compiling User Code}

All user programs that use the communication library should use the
linker option {\textrm{-module comlib}. For example,
\begin{alltt}
charmc -o pgm pgm.o -module comlib
\end{alltt}


\subsubsection{Supported Operations and Strategies}

The communication framework now supports four different communication
operations:
\begin{enumerate}
\item all-to-all/many-to-many communication,
\item array and group broadcast,
\item section multicast,
\item streaming.
\end{enumerate}
Table~\ref{tbl:com_operation} shows the different strategies that optimize these
communication operations. Some of these are converse strategies while others are
charm strategies. In the following paragraphs, we present in detail the
strategies optimizing the above mentioned operations.

\begin{table}[h]
\begin{center}
\begin{small}
\begin{tabular}{|c|c|c|}
\hline
{\bf Operation} & {\bf Object Strategy} & {\bf Processor Strategy} \\
\hline
\begin{tabular}{c}
All-to-All/Many-to-many \\
personalized and multicast
\end{tabular}
 & EachToManyMulticastStrategy & Mesh, Grid, Hypercube, Direct \\
%Many-to-many  multicast    & EachToManyMulticastStrategy & Mesh, Grid, Hypercube, Direct \\
Broadcast  & BroadcastStrategy, PipeBroadcastStrategy & Binomial tree, Binary tree\\
Section Multicast &
\begin{tabular}{c}
DirectMulticastStrategy, RingMulticastStrategy,\\
MultiRingMulticast
\end{tabular} & \\
Streaming  & Streaming, MeshStreaming, PrioStreaming & \\
\hline
\end{tabular}
\end{small}
\end{center}
\caption{Communication Operations supported in the Framework}
\label{tbl:com_operation}
\end{table}

There are two types of strategies in the communication framework:

\begin{itemize}

\item \label{bracketed strategies}
Bracketed Strategies: In bracketed strategies each source chare (which could be
an array element or a group) deposits its entries and then the strategy performs
the communication optimization. For example the EachToManyMulticastStrategy is a
bracketed strategy. For bracketed strategies a beginIteration and an
endIteration must be called before and after making the deposits respectively.

The usage of the strategy becomes:

\begin{alltt}
ComlibBegin(dproxy);

dproxy[index].entry(message);   // Sending a message
.....     //sending more messages
.....

ComlibEnd(dproxy);
\end{alltt}

\item Non-Bracketed Strategies: Non-bracketed strategies perform communication
optimizations without needing calls to beginIteration and endIteration to start
processing. Non-bracketed strategies either immediately process messages or
after a timeout, and, in both cases, it's not triggered from the application.

\end{itemize}


\subsubsection{Many-to-many Strategies}

The class {\em EachToManyMulticastStrategy} optimizes both all-to-all
personalized and all-to-all multicast communication using several virtual
topologies like 2-D Mesh, 3-D Mesh and Hypercube. Personalized communication
happens when a chare sends different messages to the other chares, multicast
communication happens when a chare sends the same message to all other chares.
EachToManyMulticastStrategy also optimizes the special cases of many-to-many multicast where
not all the chares in an array are involved in the collective operation.

The charm level strategy collects all the messages from the chares and delivers
them to the destination, while the low level (processor-to-processor)
communication is performed through converse level {\em routers} and
implements the various virtual topologies.

%For example, with the mesh router, the strategy on each processor
%first sends messages to its row neighbors. After having received its row
%messages each processor sends the column messages. After having received the
%column messages an iteration of the strategy finishes. All local messages are
%delivered as soon they are received.

%{\em EachToManyMulticastStrategy} is also used to optimize all-to-all multicast
%communication, where a processor sends the same message to all others, using the
%same virtual topologies at the lower level.

EachToManyMulticastStrategy requires that all local messages be deposited
before they can be packed into single messages. Hence, it needs to be a {\em
bracketed} strategy. This strategy can also be used to optimize all-to-all collectives between charm
groups.

%Bracketed strategies require each of the participating objects to deposit their
%intended messages within brackets. Calls to {\em ComlibBeginIteration} and {\em
%ComlibEndIteration} create a bracket. The call ComlibBeginIteration sets up the
%delegation framework to forward user messages to the correct strategy instance.
%User messages then get passed to the insertMessage entry function of the
%strategy. When all local objects have called ComlibEndIteration, doneInserting
%is invoked on the strategy.

%Bracketed strategies are typically needed when the communication
%optimization requires local source objects to reach a barrier. At this
%local barrier the communication framework invokes doneInserting on
%that strategy, which the calls the converse level strategy.

%Non-bracketed strategies have no such restriction. They process
%messages as soon as they arrive. so, non-bracketed strategies should
%not expect a doneInserting to be invoked on them. They must all
%process messages in the insertMessage call itself.

As for the constructors to be used in the main chare, the two prototypes follow.
The first one is for groups, the second for arrays. The optional parameters
allow to specify the many-to-many behaviour, passing the lists of source and
destination elements participating in the operation. If they are left to the
default value, the collective is an all-to-all.

\begin{alltt}
EachToManyMulticastStrategy(int substrategy, int nsrcpes=0, int *srcpelist=0,
                            int ndestpes=0, int *destpelist=0);

EachToManyMulticastStrategy(int substrategy, CkArrayID src, CkArrayID dest,
                            int nsrc=0, CkArrayIndex *srcelements=0,
                            int ndest=0, CkArrayIndex *destelements=0);
\end{alltt}

Both have as first parameter the virtual topology that the strategy will use for
the low level optimization. The possible values are:

\begin{description}
\item[USE\_DIRECT] to send messages directly;
\item[USE\_MESH] to send messages across a 2D Mesh;
\item[USE\_GRID] to send messages across a 3D Grid;
\item[USE\_HYPERCUBE] to send messages across a Hypercube.
\end{description}

USE\_HYPERCUBE will do best for very small messages and small number of
processors, 3d has better performance for slightly higher message sizes and then
Mesh starts performing best. The programmer is encouraged to try out all the
topologies.


\subsubsection{Broadcast Strategies}

There are two strategies of this type: {\em BroadcastStrategy} and {\em
PipeBroadcastStrategy}. The first works only for group broadcast, while the
second works for both groups and arrays.

BroadcastStrategy performs a broadcast through a hypercube (default) or a tree,
and the constructor is:

\begin{alltt}
BroadcastStrategy(int topology=USE_HYPERCUBE);
\end{alltt}

PipeBroadcastStrategy performs a broadcast through a ring or a hypercube
(default). The characteristic of this strategy is that it fragments the message
into small chunks that fit a predetermined size (passed as argument to the
constructor), and it reassembles them before delivery. The constructor
prototypes for groups and arrays respectively are:

\begin{alltt}
PipeBroadcastStrategy(int topology, CkArrayID aid, int pipeSize=DEFAULT_PIPE);
PipeBroadcastStrategy(CkGroupID gid, int topology=USE_HYPERCUBE, int pipeSize=DEFAULT_PIPE);
\end{alltt}


\subsubsection{Section Multicast Strategies}

The subclasses of MulticastStrategy can multicast a message to the entire array
or a section of array elements (MulticastStrategy itself is abstract). The
multicast strategies are non-bracketed, and the message is processed when the
application deposits it. These strategies do not combine messages, but they may
sequence the destinations of the multicast to minimize contention on a network.

In order to use these strategies, the message sent must inherit from class
{\textrm{CkMcastBaseMsg}}. (For an example see
\examplerefdir{commlib/multicast}).

These are the subclass strategies that are available:

\begin{description}
\item[DirectMulticastStrategy] sends the messages directly to all recipients;
\item[RingMulticastStrategy] sends the messages along ring resulting in good throughput as the ring permutation is contention free on many communication topologies;
\item[MultiRingMulticast] sends the message along two rings (the ordered list of recipients is split in half).
\end{description}

For these, the constructors are of the form:

\begin{alltt}
DirectMulticastStrategy(CkArrayID aid, int flag=0);
RingMulticastStrategy(CkArrayID dest_id, int flag=0);
MultiRingMulticast(CkArrayID dest_id, int flag=0);
\end{alltt}

For section multicast, the user must create a section proxy and delegate it to
the communication library. Invocations on section proxies are passed on to the
section multicast strategy.


\subsubsection{Streaming Strategies}

This strategy optimizes the scenario where chares send several small messages to
other chares. The StreamingStrategy collects messages destined to the same
physical processor and, after a timeout or when a certain number of messages
have been collected, it sends them as a single message. This results in sending
fewer messages of larger size. The timeout is a floating-point parameter to the
StreamingStrategy. It needs to be specified in milliseconds, with a default
value of 1ms. Micro-second timeouts can also be specified by passing values less
than 1. For example, $0.1$ represents $100\mu s$.

The Streaming Strategy is a non-bracketed strategy. Since messages can be
delayed due to the timeout present, it is possible to call
{\textrm{ComlibEnd()}} to flush all the messages to be sent immediately.

The prototype of the constructor is:

\begin{alltt}
StreamingStrategy(float period\_in\_ms, int nmsgs);
\end{alltt}

There are two variants of this strategy:

\begin{description}
\item[MeshStreamingStrategy] which sends the messages along a mesh instead of a linear array as the basic one;
\item[PrioStreaming] which looks at the priority of the messages, and sends those with a priority above a certain threshold directly, without delay. This strategy accepts a third parameter in the constructor for the threshold priority.
\end{description}


\subsubsection{Communication Optimization Development}

Optimization algorithms are implemented as Strategies in the communication
library. Strategies can be implemented at the Object (\charmpp) level or the
processor (\converse) level. Code reuse is possible by having a few object
managers perform object level optimizations and then call several other
processor level optimization schemes. For example, to optimize all-to-all
communication the processor level strategies could use the different virtual
topologies.

All processor (\converse) level strategies inherit from the {\em class~Strategy}
defined below and override its virtual methods.

\begin{alltt}
// Converse or Processor level strategy
class Strategy : public PUP::able{
public:
    // Called for each message
    virtual void insertMessage(MessageHolder *msg);
    // Called after all chares and groups have finished depositing their
    // messages on that processor.

    virtual void doneInserting();
    virtual void beginProcessing(int nelements);
};
\end{alltt}

The class method {\em insertMessage} is called to deposit messages with the
strategy. MessageHolder is a wrapper for converse messages. When a processor has
sent all its messages {\em doneInserting} is invoked on the strategy.

At the \charmpp{} level, all strategies inherit from the {\em
class~CharmStrategy} reported here.

\begin{alltt}
// Charm++ or Object level strategy
class CharmStrategy : public Strategy{
 protected:
    int isArray;
    int isGroup;
    int isStrategyBracketed;
    ............   
    ............   
public:
    // Called for each message
    virtual void insertMessage(CharmMessageHolder *msg);
    // Called after all chares and groups have finished depositing their
    // messages on that processor.
    virtual void doneInserting();
    virtual void beginProcessing(int nelements);
};
\end{alltt}

\charmpp{} level strategies also have to implement the insertMessage and
doneInserting methods. Here insertMessage takes a CharmMessageHolder which is a
\charmpp{} message wrapper. The call to beginProcessing initializes the strategies
on each processor. This additional call is needed because the constructor of the
strategy is called by user code in main::main on processor 0, while the strategy
needs to be constructed everywhere. Along with initializing its data,
beginProcessing can also register message handlers, as the communication library
strategies use Converse handlers to communicate between processors. The flags
{\em isArray} and {\em isGroup} store the type of objects that call the strategy
and the flag {\em isStrategyBracketed} specifies if the CharmStrategy is
bracketed or not. Bracketed strategies require that the application deposits
messages in brackets demarcated by the calls ComlibBegin and
ComlibEnd.
