\zap{
\subsection{Basic Constructs and Program Structure}

\charmpp\ is an object-based parallel programming paradigm.
%What sets \charmpp\ apart
%from traditional programming models such as message passing and shared variable
%programming is that the execution model of \charmpp\ is message-driven.
Computations in \charmpp\ are triggered based on arrival of
associated messages. These computations in turn can fire off more messages to
other (possibly remote) processors that trigger more computations on those
processors.

At the heart of any \charmpp\ program is a scheduler that repetitively chooses
a message from the available pool of messages, and executes the computations
associated with that message.

The programmer-visible entities in a \charmpp\ program are:

\begin{itemize}
\item Concurrent Objects : called {\em chares}\footnote{
      Chare (pronounced {\bf ch\"ar}, \"a as in c{\bf a}rt) is Old 
      English for chore.
      }
\item Communication Objects : Messages
\item Readonly data
\end{itemize}

\charmpp\ starts a program by creating a single \index{chare} instance of each
{\em mainchare} on processor 0, and invokes constructor methods of these
chares.  Typically, these chares then creates a number of other \index{chare}
chares, possibly on other processors, which can simultaneously work to solve
the problem at hand.

Each \index{chare}chare contains a number of \index{entry method}{\em entry
methods}, which are methods that can be invoked from remote processors. The
\charmpp\ runtime system needs to be explicitly told about these methods, via
an {\em interface} in a separate file.  The syntax of this interface
specification file is described in the later sections.

\charmpp\ provides system calls to asynchronously create remote \index{chare}
chares and to asynchronously invoke entry methods on remote chares by sending
\index{message} messages to those chares. This asynchronous
\index{message}message passing is the basic interprocess communication
mechanism in \charmpp. However, \charmpp\ also permits wide variations on this
mechanism to make it easy for the programmer to write programs that adapt to
the dynamic runtime environment.  These possible variations include
prioritization (associating priorities with method invocations), conditional
\index{message packing}message packing and unpacking (for reducing messaging
overhead), \index{quiescence}quiescence detection (for detecting completion of
some phase of the program), and dynamic load balancing (during remote object
creation). In addition, several libraries are built on top of \charmpp\ that
can simplify otherwise arduous parallel programming tasks.

We think that \charmpp\ is easy to use if you are familiar with object-based
programming. (But of course that is our opinion, if your opinion differs,
you are encouraged to let us know the reasons, and features that you would
like to see in \charmpp.) Object-based programming is built around the
concept of ``encapsulation'' of data. As implemented in \CC, data
encapsulation is achieved by grouping together data and methods (also known
as functions, subroutines, or procedures) inside of an object.

A class is a blueprint for an object.  The encapsulated data is said to be
``private'' to the object, and only the methods of that class can manipulate
that data. A method that has the same name as the class is a ``blessed''
method, called a ``Constructor'' for that class.  A constructor method is
typically responsible for initializing the encapsulated data of an object.
Each method, including the constructor can optionally be supplied data in
the form of parameters (or arguments). In \CC, one can create objects with
the {\tt new} operator that returns a pointer to the object. This pointer
can be used to refer to the object, and call methods on that object.

\charmpp{} is built on top of \CC, and also based on ``encapsulation''.
Similar to \CC, \charmpp\ entities can contain private data, and public
methods. The major difference is that these methods can be invoked from
remote processors asynchronously.  Asynchronous method invocation means that
the caller does not wait for the method to be actually executed and does not
wait for the method's return value. Therefore, \charmpp\ methods (called
entry methods) do not have a return value\footnote{Asynchronous remote
method invocation is the core of \charmpp. However, to simplify programming,
\charmpp\ makes use of the interoperable nature of its runtime system, and
combines seamlessly with user-level threads to also support synchronous
method execution, albeit with a slight overhead of thread creation and
scheduling.}. Since the actual \charmpp\ object on which the method is being
invoked may be on a remote processor\footnote{With its own, different address
space}, the \CC\ way of referring to an object, via a pointer, is not valid
in \charmpp.  Instead, we refer to a remote chare via a ``proxy'',
as explained below.


\subsection{Entities in \charmpp\ programs}

This section describes various entities in a typical \charmpp\ program.

\subsubsection{Sequential Objects}

A \charmpp\ program typically consists mostly of ordinary sequential \CC
code and objects. Such entities are only accessible locally, are not known
to the \charmpp\ runtime system, and thus need not be mentioned in the
module interface files. 

\charmpp\ does not affect the syntax or semantics of such \CC\ entities,
except that changes to global variables (or static data members of a class)
on one node will not be visible on other nodes.  Global data changes
must be explicitly sent between processors.  For processor- and
thread-private storage, refer to the ``Global Variables'' section
of the Converse manual.


\subsubsection{Messages}

Messages supply data arguments to the asynchronous remote method invocation.
These objects are treated differently from other objects in \charmpp\ by the
runtime system, and therefore they must be specified in the interface file
of the module.  With parameter marshalling, the system creates and handles
the message completely internally. Other messages are instances of \CC\
classes that are subclassed from a special class that is generated by the
\charmpp\ interface translator.  Another variation of communication objects
is conditionally packed and unpacked. This variation should be used when one
wants to send messages that contain pointers to the data rather than the
actual data to other processors. This type of communication objects contains
two static methods: \kw{pack}, and \kw{unpack}. The third variation of
communication objects is called {\em varsize} messages. Varsize messages is
an effective optimization on conditionally packed messages, and can be
declared with special syntax in the interface file.

\subsubsection{Chares}

Chares are the most important entities in a \charmpp\ program. These concurrent
objects are different from sequential \CC\ objects in many ways. Syntactically,
Chares are instances of \CC\  classes that are derived from a system-provided
class called \kw{Chare}. Also, in addition to the usual \CC\ private and public
data and method members, they contain some public methods called {\em entry
methods}. These entry methods do not return anything (they are {\tt void}
methods), and take at most one argument, which is a pointer to a message.
Chares are {\em accessed} using a proxy (an object of a specialized class
generated by the \charmpp\ interface translator) or using a handle (a \kw{
CkChareID} structure defined in \charmpp), rather than a pointer as in \CC.
Semantically, they are different from \CC\ objects because they can be created
asynchronously from remote processors, and their entry methods also could be
invoked asynchronously from the remote processors. Since the constructor method
is invoked from remote processor (while creating a chare), every chare should
have its constructors as entry methods (with at most one message pointer
parameter). These chares and their entry methods have to be specified in the
interface file.

\subsubsection{Chare Arrays}

Chare arrays are collections of chares. However, unlike chare groups or
nodegroups, arrays are not constrained by characteristics of the underlying
parallel machine such as number of processors or nodes. Thus, chare arrays
can have any number of {\em elements}. The array elements themselves are
chares, and methods can be invoked on individual array elements as usual.  
Each element of an array has a globally unique index, and messages are
addressed to that index.

Unlike other entities in \charmpp\, the dynamic load balancing framework (LB
Framework) treats array elements as objects that can be migrated across
processors. Thus, the runtime system keeps track of computational load
across the system, and also the time spent in execution of entry methods on
array elements, and then employs one of several strategies to redistribute
array elements across the available processors.

\subsubsection{Chare Groups}

Chare Groups\footnote{ These were called Branch Office Chares (BOC) in earlier
versions of Charm.} are a special type of concurrent objects.  Each chare group
is a collection of chares, with one representative (group member) on each
processor. All the members of a chare group share a globally unique name
(handle, defined by Charm RTS to be of type \kw{CkGroupID}). An entire chare
group could be addressed using this global handle, and an individual member of
a chare group can be addressed using the global handle, and a processor number.
Chare groups are instances of \CC\ classes subclassed from a system-provided
class called \kw{Group}. The Charm RTS has to be notified that these chares
are semantically different, and therefore chare groups have a different
declaration in the interface specification file.

\subsubsection{Chare Nodegroups}

Chare nodegroups are very similar to chare groups except that instead of having
one group member on each processor, the nodegroup has one member on each shared
memory multiprocessor node. Note that \charmpp\ (and its underlying runtime
system Converse) distinguish between processors and nodes. A node consists of
one or more processors that share an address space. The last few years have
seen emergence of fast SMP systems of small (2-4 processors) to large (32-64
processors) number of processors per node. A network of such SMP nodes is the
most general model of parallel computers, making pure distributed and pure
shared memory systems mere special cases. \charmpp\ is built on top of this
machine abstraction, and Chare nodegroups embody this abstraction in a higher
level language construct. Semantically, methods invoked on a nodegroup member
could be executed on any processor within that node. This fact can be utilized
for supporting load balance across processors within a node. However, this also
means that different processors within a node could be executing methods of the
same nodegroup member simultaneously, thus leading to common problems
associated with shared address space programming. However, \charmpp\ eases such
problems by allowing the programmer to specify an entry method of a nodegroup
to be {\em exclusive}, thus guaranteeing that no other {\em exclusive} method
of that nodegroup member can execute simultaneously within the node.

\subsubsection{Entry Methods}
In \charmpp, \index{chare}chares, \index{group}groups and \index{nodegroup}
nodegroups communicate using remote method invocation.  These ``remote entry'' methods may either take marshalled parameters, described in the next section; or special objects called messages.

\charm programs are open to the full gamut of program design techniques that are possible in
\CC. One may view the role of the programming model as providing a platform for
addressing and interacting with remote objects.
However, \charmpp does not provide a full global address space. Each process in
the parallel execution has its own address space and \charmpp does not
implicitly share or synchronize global or static variables.

}
