
The orchestration code in the .or file can be divided into two part. The header
part contains information about the program, included external files, defines,
and declaration of parallel constructs used in the code. The orchestration
section is made up of statements that forms a global control flow of the
parallel program. In the orchestration code, Charisma employs a macro dataflow
approach; the statements produce and consume values, from which the control flows
can be organized, and messages and method invocations generated. 

\subsubsection{Header Section}

The very first line should give the name of the Charisma program with the {\tt
program} keyword.

\begin{SaveVerbatim}{foodecl}
    program jacobi 
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

The {\tt program} keyword can be replaced with {\tt module}, which means that
the output program is going to be a library module instead of a stand-alone
program. Please refer to Section~\ref{sec:module} for more details.

Next, the programmer can include external code files in the generated code with
keyword {\tt include} with the filename without extension. For example, the
following statement tells the Charisma compiler to look for header file
``particles.h'' to be included in the generated header file ``jacobi.h'' and to
look for C/C++ code file ``particles.[C|cc|cpp|cxx|c]'' to be included in the
generated C++ code file ``jacobi.C''.

\begin{SaveVerbatim}{foodecl}
    include particles;
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

It is useful when there are source code that must precede the generated
parallel code, such as basic data structure declaration. 

After the {\tt include} section is the {\tt define} section, where environmental
variables can be defined for Charisma. For example, to tell Charisma to generate
additional code to enable the load balancing module, the programmer needs to 
define ``ldb'' in the orchestration code. Please refer to Section~\ref{sec:ldb}
for details.

\subsubsection{Declaration Section}

Next comes the declaration section, where classes, objects and parameters are
declared. A Charisma program is composed of multiple sets of parallel objects
which are organized by the orchestration code. Different sets of objects can be
instantiated from different class types. Therefore, we have to specify the class
types and object instantiation. Also we need to specify the parameters (See
Section~\ref{sec:orchsec}) to use in the orchestration statements. 

A Charisma program or module has one ``MainChare'' class, and it does not
require explicit instantiation since it is a singleton. The statement to declare
MainChare looks like this:

\begin{SaveVerbatim}{foodecl}
    class JacobiMain : MainChare;
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

For object arrays, we first need to declare the class types inherited from 1D
object array, 2D object array, etc, and then instantiate from the class types. 
The dimensionality information of the object array is given in a pair of 
brackets with each dimension size separated by a comma.

\begin{SaveVerbatim}{foodecl}
    class JacobiWorker : ChareArray1D;
    obj workers : JacobiWorker[N];

    class Cell : ChareArray3D;
    obj cells : Cell[M,M,M];
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

Note that key word ``class'' is for class type derivation, and ``obj'' is for
parallel object or object array instantiation. The above code segment declares a
new class type JacobiWorker which is a 1D object array, (and the programmer is
supposed to supply sequential code for it in files ``JacobiWorker.h'' and
``JacobiWorker.C'' (See Section~\ref{sec:sequential} for more details on
sequential code). Object array ``workers'' is instantiated from ``JacobiWorker''
and has 16 elements.

The last part is orchestration parameter declaration. These parameters are used
only in the orchestration code to connect input and output of orchestration
statements, and their data type and size is declared here. More explanation of
these parameters can be found in Section~\ref{sec:orchsec}. 

\begin{SaveVerbatim}{foodecl}
    param lb : double[N];
    param rb : double[N];
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

With this, ``lb'' and ``rb'' are declared as parameters of that can be 
``connected'' with local variables of double array with size of 512. 

\subsubsection{Orchestration Section}
\label{sec:orchsec}

In the main body of orchestration code, the programmer describes the behavior
and interaction of the elements of the object arrays using orchestration
statements.

$\bullet$ {\bf Foreach Statement}

The most common kind of parallelism is the invocation of a method
across all elements in an object array. Charisma provides a {\em foreach}
statement for specifying such parallelism. The keywords \code{foreach} and
\code{end-foreach} forms an enclosure within which the parallel invocation is
performed. The following code segment invokes the entry method \code{compute} on
all the elements of array \code{myWorkers}. 

\begin{SaveVerbatim}{foodecl}
  foreach i in workers
    workers[i].compute();
  end-foreach
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

$\bullet$ {\bf Publish Statement and Produced/Consumed Parameters}

In the orchestration code, an object method invocation can have input and output
(consumed and produced) parameters. Here is an orchestration statement that
exemplifies the input and output of this object methods
\code{workers.produceBorders} and \code{workers.compute}. 

\begin{SaveVerbatim}{foodecl}
  foreach i in workers
    (lb[i], rb[i]) <- workers[i].produceBorders();
    workers[i].compute(lb[i+1], rb[i-1]);
    
    (+error) <- workers[i].reduceData();
  end-foreach
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

Here, the entry method \code{workers[i].produceBorders} produces (called {\em
published} in Charisma) values of \code{lb[i], rb[i]}, enclosed in a pair of
parentheses before the publishing sign ``\code{<-}''. In the second
statement, function \code{workers[i].compute} consumes values of \code{lb[i+1],
rb[i-1]}, just like normal function parameters. If a reduction operation is
needed, the reduced parameter is marked with a ``\code{+}'' before it, like the
\code{error} in the third statement. 

A entry method can have arbitrary number of published (produced and reduced)
values and consumed values. In addition to basic data types, each of these
values can also be an object of arbitrary type. The values published by
\code{A[i]} must have the index \code{i}, whereas values consumed can have the
index \code{e(i)}, which is an index expression in the form of \code{i}$\pm c$
where $c$ is a constant. Although we have used different symbols (\code{p} and
\code{q}) for the input and the output variables, they are allowed to overlap. 

The parameters are produced and consumed in the program order. Namely, a
parameter produced in an early statement will be consumed by the next consuming
statement, but will no longer be visible to any consuming statement after a 
subsequent statement producing the same parameter in program order.
Special rules involving loops are discussed later with loop statement.

$\bullet$ {\bf Overlap Statement}

Complicated parallel programs usually have concurrent flows of control. To
explicitly express this, Charisma provides a \code{overlap} keyword, whereby the
programmer can fire multiple overlapping control flows. These flows may contain
different number of steps or statements, and their execution should be
independent of one another so that their progress can interleave with arbitrary
order and always return correct results. 

\begin{SaveVerbatim}{foodecl}
  overlap
  {
    foreach i in workers1
      (lb[i], rb[i]) <- workers1[i].produceBorders();
    end-foreach
    foreach i in workers1
      workers1[i].compute(lb[i+1], rb[i-1]);
    end-foreach
  }
  {
    foreach i in workers2
      (lb[i], rb[i]) <- workers2[i].compute(lb[i+1], rb[i-1]);
    end-foreach
  }
  end-overlap
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

This example shows an \code{overlap} statement where two blocks in curly
brackets are executed in parallel. Their execution join back to one at the end
mark of \code{end-overlap}. 

$\bullet$ {\bf Loop Statement}

Loops are supported with \code{for} statement and \code{while} statement. Here
are two examples.
\begin{SaveVerbatim}{foodecl}
  for iter = 0 to MAX_ITER
     workers.doWork();
  end-for
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}
  
\begin{SaveVerbatim}{foodecl}
  while (err > epsilon)
     (+err) <- workers.doWork();
     MainChare.updateError(err);
  end-while
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

The loop condition in \code{for} statement is independent from the main program;
It simply tells the program to repeat the block for so many times. The loop
condition in \code{while} statement is actually updated in the MainChare. In the
above example, \code{err} and \code{epsilon} are both member variables of class
\code{MainChare}, and can be updated as the example shows. The programmer can
active the ``autoScalar'' feature by including a ``\code{define autoScalar;}''
statement in the orchestration code. When autoScalar is enabled,  Charisma will
find all the scalars in the \code{.or} file, and create a local copy in the
\code{MainChare}. Then every time the scalar is published by a statement, an
update statement will automatically be inserted after that statement. The only
thing that the programmer needs to do is to initialize the local scalar with
a proper value. 

Rules of connecting produced and consumed parameters concerning loops are
natural. The first consuming statement will look for values produced by the last
producing statement before the loop, for the first iteration. The last
producing statement within the loop body, for the following iterations. At the
last iteration, the last produced values will be disseminated to the code
segment following the loop body. Within the loop body, program order holds. 

\begin{SaveVerbatim}{foodecl}
  for iter = 1 to MAX_ITER
    foreach i in workers
      (lb[i], rb[i]) <- workers[i].compute(lb[i+1], rb[i-1]);
    end-foreach
  end-for
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

One special case is when one statement's produced parameter and consumed
parameter overlaps. It must be noted that there is no dependency within the same
\code{foreach} statement. In the above code segment, the values consumed
\code{lb[i], rb[i]} by \code{worker[i]} will not come from  
its neighbors in this iteration. The rule is that the consumed values always
originate from previous \code{foreach} statements or \code{foreach} statements
from a previous loop iteration, and the published values are visible only to
following \code{foreach} statements or \code{foreach} statements in following
loop iterations. 

$\bullet$ {\bf Scatter and Gather Operation}

A collection of values produced by one object may be split and consumed by
multiple object array elements for a scatter operation. Conversely, a collection
of values from different objects can be gathered to be consumed by one object.

\begin{SaveVerbatim}{foodecl}
  foreach i in A
    (points[i,*]) <- A[i].f(...);
  end-foreach
  foreach k,j in B
    (...) <- B[k,j].g(points[k,j]);
  end-foreach
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

A wildcard dimension ``*'' in \code{A[i].f()}'s output \code{points} specifies
that it will publish multiple data items. At the consuming side, each
\code{B[k,j]} consumes only one point in the data, and therefore a scatter
communication will be generated from \code{A} to \code{B}. For instance,
\code{A[1]} will publish data \code{points[1,0..N-1]} to be consumed by multiple
array objects \code{B[1,0..N-1]}.  

\begin{SaveVerbatim}{foodecl}
  foreach i,j in A
    (points[i,j]) <- A[i,j].f(...);
  end-foreach
  foreach k in B
    (...) <- B[k].g(points[*,k]);
  end-foreach
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}


Similar to the scatter example, if a wildcard dimension ``*'' is in the
consumed parameter and the corresponding published parameter does not have a
wildcard dimension, there is a gather operation generated from the publishing
statement to the consuming statement. In the following code segment, each 
\code{A[i,j]} publishes a data point, then data points from \code{A[0..N-1,j]} are
combined together to for the data to be consumed by \code{B[j]}.  

Many communication patterns can be expressed with combination of orchestration
statements. For more details, please refer to PPL technical report 06-18,
``Charisma: Orchestrating Migratable Parallel Objects''.

Last but not least, all the orchestration statements in the \code{.or} file
together form the dependency graph. According to this dependency graph, the
messages are created and the parallel program progresses. Therefore, the user is
advised to put only parallel constructs that are driven by the data dependency
into the orchestration code. Other elements such as local dependency should be
coded in the sequential code. 

