
\charm is a C++-based parallel programming system, founded on the
migratable-objects programming model, and supported by a novel and
powerful adaptive runtime system. It supports both irregular as well
as regular applications, and can be used to specify task-parallelism
as well as data parallelism in a single application. It automates
dynamic load balancing for task-parallel as well as data-parallel
applications, via separate suites of load-balancing strategies. Via
its message-driven execution model, it supports automatic latency
tolerance, modularity and parallel composition. \charm also supports
automatic checkpoint/restart, as well as fault tolerance based on
distributed checkpoints.
% {\sc Converse} interoperable runtime system for parallel
% programming.

\charm is a production-quality parallel programming system used by
multiple applications in science and engineering on supercomputers as
well as smaller clusters around the world.  Currently the parallel
platforms supported by \charm are the BlueGene/L, BlueGene/P,
BlueGene/Q, Cray XT, XE and XK series (including XK6 and XE6),
% XT3/4, Cray X1, Cray T3E,
a single workstation or a network of workstations (including x86
(running Linux, Windows, MacOS)), etc.  The communication protocols
and infrastructures supported by
\charm are UDP, TCP, Myrinet, Infiniband, MPI, uGNI, and PAMI. 
\charm programs can run without changing the source
on all these platforms. If built on MPI, \charm programs can 
also interoperate with pure MPI programs (\S\ref{sec:interop}).
  Please see the \charm/\converse{}
Installation and Usage
\htmladdnormallink{Manual}{http://charm.cs.illinois.edu/manuals/html/install/manual.html}
for details about installing, compiling and running
\charm programs.


\section{Programming Model}
The key feature of the migratable-objects programming model is {\em
over-decomposition}: The programmer decomposes the program into a
large number of work units and data units, and specifies the
computation in terms of creation of and interactions between these
units, without any direct reference to the processor on which any unit
resides. This empowers the runtime system to assign units to
processors, and to change the assignment at runtime as
necessary. \charm is the main (and early) exemplar of this
programming model. AMPI is another example within the \charm family
of the same model.


\section{Execution Model}

% A \charm program consists of a number of \charm objects
% distributed across the available number of processors. Thus, 
A basic
unit of parallel computation in \charm programs is a {\em
chare}\index{chare}. 
% a \charm object that can be created on any
% available processor and can be accessed from remote processors.
A \index{chare}chare is similar to a process, an actor, an ADA task,
etc. At its most basic level, it is just a C++ object.
%  with some of its methods
% that can be invoked from remote objects. 
A \charm computation consists of a large number of chares
distributed on available processors of the system, and interacting
with each other via asynchronous method invocations.
Asynchronously invoking a method on a remote object can also be
thought of as
sending a ``message'' to it. So, these method invocations are
sometimes referred to as messages. (besides, in the implementation,
the method invocations are packaged as messages anyway).
\index{chare}Chares can be
created dynamically.
% , and many chares may be active simultaneously.
% Chares send \index{message}{\em messages} to one another to invoke
% methods asynchronously.  

Conceptually, the system maintains a
``work-pool'' consisting of seeds for new \index{chare}chares, and
\index{message}messages for existing chares. The \charm runtime system ({\em
Charm RTS}) may pick multiple items, non-deterministically, from this
pool and execute them, with the proviso that two different methods
cannot be simultaneously executing on the same chare object (say, on
different processors). Although one can define a reasonable
theoretical operational semantics of \charm in this fashion, a more
practical description of execution is useful to understand \charm: On
each PE (``PE'' stands for a ``Processing Element''. PEs are akin to
processor cores; see section \ref{sec:machine} for a precise
description), there is a scheduler operating with its own private pool
of messages. Each instantiated chare has one PE which is where it
currently resides. The pool on each PE includes messages meant for
Chares residing on that PE, and seeds for new Chares that are
tentatively meant to be instantiated on that PE. The scheduler picks a
message, creates a new chare if the message is a seed (i.e. a
constructor invocation) for a new Chare, and invokes the method
specified by the message. When the method returns control back to the
scheduler, it repeats the cycle. I.e. there is no pre-emptive
scheduling of other invocations.

When a chare method executes, it may create  method invocations for other
chares. The Charm Runtime System (RTS, sometimes referred to as the
Chare Kernel in the manual) locates the PE where the targeted chare
resides, and delivers the invocation to the scheduler on that PE. 

Methods of a \index{chare}chare that can be remotely invoked are called
\index{entry method}{\em entry} methods.  Entry methods may take serializable
parameters, or a pointer to a message object.  Since \index{chare}
chares can be created on remote processors, obviously some constructor
of a chare needs to be an entry method.  Ordinary entry
methods\footnote{``Threaded'' or ``synchronous'' methods are
different. But even they do not lead to pre-emption; only to
cooperative multi-threading} are completely non-preemptive--
\charm will not interrupt an executing method to start any other work,
and all calls made are asynchronous.

\charm provides dynamic seed-based load balancing. Thus location (processor
number) need not be specified while creating a
remote \index{chare}chare. The Charm RTS will then place the remote
chare on a suitable processor. Thus one can imagine chare creation
as generating only a seed for the new chare, which may {\em take root}
on some specific processor at a later time. 

% Charm RTS identifies a \index{chare}chare by a {\em ChareID}.  

% Since user code does not
% need to name a chares' processor, chares can potentially migrate from
% one processor to another.  (This behavior is used by the dynamic
% load-balancing framework for chare containers, such as arrays.)

Chares can be grouped into collections. The types of collections of
chares supported in \charm are: {\em chare-arrays}, \index{group}{\em
chare-groups}, and \index{nodegroup}{\em chare-nodegroups}, referred
to as {\em arrays}, {\em groups}, and {\em nodegroups} throughout this
manual for brevity. A Chare-array is a collection of an arbitrary number
of migratable chares, indexed by some index type, and mapped to
processors according to a user-defined map group. A group (nodegroup)
is a collection of chares, with exactly one member element on each PE
(``node'').

\charm does not allow global variables, except readonly variables
(see \ref{readonly}). A chare can normally only access its own data directly.
However, each chare is accessible by a globally valid name. So, one
can think of \charm as supporting a {\em global object space}.



Every \charm program must have at least one \kw{mainchare}.  Each
\kw{mainchare} is created by the system on processor 0 when the \charm
program starts up.  Execution of a \charm program begins with the
Charm Kernel constructing all the designated \kw{mainchare}s.  For
a \kw{mainchare} named X, execution starts at constructor X() or
X(CkArgMsg *) which are equivalent.  Typically, the
\kw{mainchare} constructor starts the computation by creating arrays, other
chares, and groups.  It can also be used to initialize shared \kw{readonly}
objects.

\charm program execution is terminated by the \kw{CkExit} call.  Like the
\kw{exit} system call, \kw{CkExit} never returns. The Charm RTS ensures
that no more messages are processed and no entry methods are called after a
\kw{CkExit}. \kw{CkExit} need not be called on all processors; it is enough
to call it from just one processor at the end of the computation.

\zap{
The only method of communication between processors in \charm is
asynchronous \index{entry method} entry method invocation on remote chares.
For this purpose, Charm RTS needs to know the types of
\index{chare}chares in the user program, the methods that can be invoked on
these chares from remote processors, the arguments these methods take as
input etc. Therefore, when the program starts up, these user-defined
entities need to be registered with Charm RTS, which assigns a unique
identifier to each of them. While invoking a method on a remote object,
these identifiers need to be specified to Charm RTS. Registration of
user-defined entities, and maintaining these identifiers can be cumbersome.
Fortunately, it is done automatically by the \charm interface translator.
The \charm interface translator generates definitions for {\em proxy}
objects. A proxy object acts as a {\em handle} to a remote chare. One
invokes methods on a proxy object, which in turn carries out remote method
invocation on the chare.}

As described so far, the execution of individual Chares is
``reactive'': When method A is invoked the chare executes this code,
and so on. But very often, chares have specific life-cycles, and the
sequence of entry methods they execute can be specified in a
structured manner, while allowing for some localized non-determinism
(e.g. a pair of methods may execute in any order, but when they both
finish, the execution continues in a pre-determined manner, say
executing a 3rd entry method). To simplify expression of such control
structures, \charm provides two methods: the structured dagger
notation (Sec \ref{sec:sdag}), which is the main notation we recommend
you use.  Alternatively, you may use threaded entry methods, in
combination with {\em futures} and {\em sync} methods
(See \ref{threaded}). The threaded methods run in light-weight
user-level threads, and can block waiting for data in a variety of
ways. Again, only the particular thread of a particular chare is
blocked, while the PE continues executing other chares.

The normal entry methods, being asynchronus, are not allowed to return
any value, and are declared with a void return type. However, the {\em
sync} methods are an exception to this. They must be called from a
threaded method, and so are allowed to return (certain types of)
values.  

\section{Proxies and the charm interface file}
\label{proxies}

To support asynchronous method invocation and global object space, the
RTS needs to be able to serialize (``marshall'') the parameters, and
be able to generate global ``names'' for chares. For this purprpose,
programmers have to declare the chare classes and the signature of
their entry methods in a special ``\verb#.ci#'' file, called an
interface file. Other than the interface file, the rest of a \charm
program consists of just normal C++ code. The system generates several
classes based on the declarations in the interface file, including
``Proxy'' classes for each chare class.
Those familiar with various component models (such as CORBA) in the
distributed computing world will recognize ``proxy'' to be a dummy, standin
entity that refers to an actual entity.  For each chare type, a ``proxy''
class exists.
% \footnote{The proxy class is generated by the ``interface
% translator'' based on a description of the entry methods}  
The methods of
this ``proxy'' class correspond to the remote methods of the actual class, and
act as ``forwarders''. That is, when one invokes a method on a proxy to a
remote object, the proxy marshalls the parameters into a message, puts
adequate information about the target chare on the envelope of the
message, and forwards it to the
remote object. 
Individual chares, chare array, groups, node-groups, as well as the
individual elements of these collections have a such a
proxy. Multiple methods for obtaining such proxies are described in
the manual.
Proxies for each type of entity in \charm
have some differences among the features they support, but the basic
syntax and semantics remain the same -- that of invoking methods on
the remote object by invoking methods on proxies.

% You can have several proxies that all refer to the same object.

\zap{
Historically, handles (which are basically globally unique
identifiers) were used to uniquely identify \charm objects.  Unlike
pointers, they are valid on all processors and so could be sent as
parameters in messages.  They are still available, but now proxies
also have the same feature.

Handles (like CkChareID, CkArrayID, etc.) are still used internally, but should only be considered relevant to expert level usage.}
\zap{
Proxies (like
CProxy\_foo) are just bytes and can be sent in messages, pup'd, and
parameter marshalled.  This is now true of almost all objects in
\charm: the only exceptions being entire Chares (Array Elements,
etc.) and, paradoxically, messages themselves.
}

The following sections provide detailed information about various features of the
\charm programming system. Part I, ``Basic Usage'', is sufficient
for writing full-fledged applications. Note that only the last two
chapters of this part involve the notion of physical processors
(cores, nodes, ..), with the exception of simple query-type utilities
(Sec \ref{basic utility fns}). We strongly suggest that all
application developers, beginners and experts alike, try to stick to
the basic language to the extent possible, and use features from the
advanced sections only when you are convinced they are
essential. (They are are useful in specific situations; but a common
mistake we see when we examine programs written by beginners is the
inclusion of complex features that are not necessary for their
purpose. Hence the caution). The advanced concepts in the Part II of
the manual support optimizations, convenience features, and more
complex or sophisticated features. \footnote{For a description of the underlying design
philosophy please refer to the following papers :\\
    L. V. Kale and Sanjeev Krishnan,
    {\em ``\charm : Parallel Programming with Message-Driven Objects''},
    in ``Parallel Programming Using \CC'',
    MIT Press, 1995. \\
    L. V. Kale and Sanjeev Krishnan,
    {\em ``\charm : A Portable Concurrent Object Oriented System
    Based On \CC''},
    Proceedings of the Conference on Object Oriented Programming,
    Systems, Languages and Applications (OOPSLA), September 1993.
}
