With the latest 6.5.0 release, \charmpp has been augmented with support for 
partitioning. The key idea is to
divide the allocated set of nodes into subsets that run independent \charmpp
instances. These \charmpp instances (called partitions from now on) have a
unique identifier, can be programmed to do different tasks, and can interact
with each other. Addition of the partitioning scheme does not affect the
existing code base or codes that do not want to use partitioning. Some of the use
cases of partitioning are replicated NAMD, replica based fault tolerance,
studying mapping performance etc. In some aspects, partitioning is similar to 
disjoint communicator creation in MPI. 

\section{Overview}
\charmpp stack has three components - Charm++, Converse and machine layer. In
general, machine layer handles the exchange of messages among nodes, and
interacts with the next layer in the stack - Converse. Converse is responsible
for scheduling of tasks (including user code) and is used by Charm++ to execute
the user application. Charm++ is the top-most level in which the applications
are written. During partitioning, Charm++ and machine layers are unaware of
the partitioning. Charm++ assumes its partition to be the entire world, whereas
machine layer considers the whole set of allocated nodes as one partition.
During start up, converse divides the allocated set of nodes into partitions, in
each of which Charm++ instances are run. It performs the necessary translations
as interactions happen with Charm++ and the machine layer.  The partitions can 
communicate with each other using the Converse API described later.

\section{Ranking}
\charmpp stack assigns a rank to every processing element (PE). In the non-partitioned
version, a rank assigned to a PE is same at all three layers of \charmpp
stack. This rank also (generally) coincides with the rank provided to processors/cores 
by the underlying job scheduler. The importance of these ranks derive from the 
fact that they are used for multiple purposes. Partitioning leads to segregation of the 
notion of ranks at different levels of Charm++ stack. What used to be the PE is now a 
local rank within a partition running a Charm++ instance. Existing methods such as {\tt CkMyPe()},
{\tt CkMyNode()}, { \tt CmiMyPe()}, etc continue to provide these local ranks. Hence, existing 
codes do not require any change as long as inter-partition interaction is not required. 

On the other hand, machine layer is provided with the target ranks that
are globally unique. These ranks can be obtained using functions with \emph{Global}
suffix such as {\tt CmiNumNodesGlobal()}, {\tt CmiMyNodeGlobal()}, {\tt CmiMyPeGlobal()} etc.

Converse, which operates at a layer between Charm++ and machine layer,
performs the required transitions. It maintains relevant information for any
conversion. Information related to partitions can be obtained using Converse
level functions such as {\tt CmiMyPartition()}, {\tt CmiNumPartitions()}, 
etc. If required, one can also obtain the mapping of a local rank to a global
rank using functions such as {\tt CmiGetPeGlobal(int perank, int partition)} and 
{\tt CmiGetNodeGlobal(int noderank, int partition)}. These functions
take two arguments - the local rank and the partition number. For example, 
CmiGetNodeGlobal(5, 2) will return the global rank of the node that belongs to 
partition 2 and has a local rank of 5 in partition 2. The inverse 
translation, from global rank to local rank, is not supported.


\section{Startup and Partitioning}

A number of compile time and runtime parameters are available for users who want 
to run multiple partitions in one single job.

\begin{itemize}
\item Runtime parameter: {\tt +partitions <part\_number>} or {\tt +replicas
<replica\_number>} - number of partitions to be created. If no further options are
provided, allocated cores/nodes are divided equally among partitions. Only this option 
is supported from the 6.5.0 release; remaining options are supported starting 6.6.0.

\item Runtime parameter: {\tt +master\_partition} - assign one core/node as the master 
partition (partition 0), and divide the remaining cores/nodes equally among remaining
partitions.

\item Runtime parameter: {\tt +partition\_sizes L[-U[:S[.R]]]\#W[,...]} - defines the size of 
partitions.  A single number identifies a particular partition. Two numbers separated by
a dash identify an inclusive range (\emph{lower bound} and \emph{upper bound}). If they 
are followed by a colon and another number (a \emph{stride}), that range will be stepped 
through in increments of the additional number. Within each stride, a dot followed by a
\emph{run} will indicate how many partitions to use from that starting point.  Finally, 
a compulsory number sign (\#) followed by a \emph{width} defines the size of each of the
partitions identified so far. For example, the sequence {\tt 0-4:2\#10,1\#5,3\#15} states that 
partitions 0, 2, 4 should be of size 10, partition 1 of size 5 and partition 3 of size 15. 
In SMP mode, these sizes are in terms of nodes. All workers threads associated with a node are
assigned to the partition of the node. This option conflicts with {\tt +assign\_master}.

\item Runtime parameter: {\tt +partition\_topology} - use a default topology aware scheme
to partition the allocated nodes. 

\item Runtime parameter: {\tt +partition\_topology\_scheme <scheme>} - use the given scheme
to partition the allocated nodes. Currently, two generalized schemes are supported that 
should be useful on torus networks. If scheme is set to 1, allocated nodes are traversed 
plane by plane during partitioning. A hilbert curve based traversal is used with scheme 2.

\item Compilation parameter: {\tt -custom-part}, runtime parameter: {\tt +use\_custom\_partition} -
enables use of user defined partitioning. In order to implement a new partitioning scheme, 
a user must link an object exporting a C function with following prototype: \\ 

extern ``C'' void createCustomPartitions(int numparts, int *partitionSize, int *nodeMap);\\
{\tt numparts} (input) - number of partitions to be created. \\
{\tt partitionSize} (input) - an array that contains size of each partition. \\
{\tt nodeMap} (output, preallocated) - a preallocated array of length {\tt CmiNumNodesGlobal()}.
Entry \emph{i} in this array specifies the new global node rank of a node with default node rank \emph{i}. 
The entries in this array are block wise divided to create partitions, i.e entries 0 to 
partitionSize[0]-1 belong to partition 1, partitionSize[0] to
partitionSize[0]+partitionSize[1]-1 to partition 2 and so on.\\

When this function is invoked to create partitions, TopoManager is configured to 
view all the allocated node as one partition. Partition based API is yet to be 
initialized, and should not be used. A link time parameter {\tt -custom-part} 
is required to be passed to {\tt charmc} for successful compilation.
\end{itemize}

\section{Redirecting output from individual partitions}
Output to standard output (stdout) from various partitions can be directed 
to separate files by passing the target path as a command line option. The run
time parameter {\tt +stdout <path>} is to be used for this purpose. The
{\tt <path>} may contain the C format specifier \emph{\%d}, which will be replaced by the
partition number. In case, \emph{\%d} is specified multiple times, only the first
three instances from the left will be replaced by the partition number (other or additional format specifiers will result in undefined behavior). If a format specifier is not specified, 
the partition number will be appended as a suffix to the specified path. Example usage:

\begin{itemize}
\item {\tt +stdout out/\%d/log} will write to \emph{out/0/log, out/1/log,
out/2/log,} $\cdots$.
\item {\tt +stdout log} will write to \emph{log.0, log.1, log.2,} $\cdots$.
\item {\tt +stdout out/\%d/log\%d} will write to \emph{out/0/log0, out/1/log1,
out/2/log2,} $\cdots$.
\end{itemize}

\section{Inter-partition Communication}

A new API was added to Converse to enable sending messages from one replica to
another. Currently, following functions are available for the same
\begin{itemize}
\item CmiInterSyncSend(local\_rank, partition, size, message)        
\item CmiInterSyncSendAndFree(local\_rank, partition, size, message)
\item CmiInterSyncNodeSend(local\_node, partition, size, message)        
\item CmiInterSyncNodeSendAndFree(local\_node, partition, size, message)
\end{itemize}

Users who have coded in Converse will find these functions to be very similar
to basic Converse functions for send – CmiSyncSend and CmiSyncSendAndFree.
Given the local rank of a PE and the partition it belongs to, these two
functions will pass the message to the machine layer. CmiInterSyncSend does
not return till “message” is ready for reuse. CmiInterSyncSendAndFree passes
the ownership of “message” to Charm++ RTS, which will free the message when
the send is complete. Each converse message contains a message header, which
makes those messages active – they contain information about their handlers.
These handlers can be registered using existing API in Charm++ -
CmiRegisterHandler.  CmiInterNodeSend and CmiInterNodeSendAndFree are
counterparts to these functions that allow sending of a message to a node (in
    SMP mode).


