\charmpp\ is based on the message-driven parallel programming paradigm. In
contrast to many other approaches, \charmpp\ programmers encode entry points to
their parallel objects, but do not explicitly wait (i.e. block) on the runtime
to indicate completion of posted `receive' requests. Thus, a \charmpp\ object's
overall flow of control can end up fragmented across a number of separate
methods, obscuring the sequence in which code is expected to
execute. Furthermore, there are often constraints on when different pieces of
code should execute relative to one another, related to data and
synchronization dependencies.

Consider one way of expressing these constraints using flags, buffers, and
counters, as in the following example:
%
%\begin{figure}[ht]
\begin{center}
\begin{alltt}
// in .ci file
chare ComputeObject \{
  entry void ComputeObject();
  entry void startStep();
  entry void firstInput(Input i);
  entry void secondInput(Input j);
\};

// in C++ file
class ComputeObject : public CBase_ComputeObject \{
  int   expectedMessageCount;
  Input first, second;

public:
  ComputeObject() \{
    startStep();
  \}
  void startStep() \{
    expectedMessageCount = 2;
  \}

  void firstInput(Input i) \{
    first = i;
    if (\verb|--expectedMessageCount == 0|)
      computeInteractions(first, second);
    \}
  void recv_second(Input j) \{
    second = j;
    if (\verb|--expectedMessageCount == 0|)
      computeInteractions(first, second);
  \}

  void computeInteractions(Input a, Input b) \{
    // do computations using a and b
    . . .
    // send off results
    . . .
    // reset for next step
    startStep();
  \}
\};
\end{alltt}
\end{center}
%\caption{Compute Object in a Molecular Dynamics Application}
%\label{figchareexample}
%\end{figure}
In each step, this object expects pairs of messages, and waits to process the
incoming data until it has both of them. This sequencing is encoded across 4
different functions, which in real code could be much larger and more numerous,
resulting in a spaghetti-code mess.

Instead, it would be preferable to express this flow of control using
structured constructs, such as loops. \charmpp\ provides such constructs for
structured control flow across an object's entry methods in a notation called
Structured Dagger. The basic constructs of Structured Dagger (SDAG) provide for
\emph{program-order execution} of the entry methods and code blocks that they
define. These definitions appear in the {\tt .ci} file definition of the
enclosing chare class as a `body' of an entry method following its signature.

The most basic construct in SDAG is the {\tt serial} (aka the {\tt atomic}) block.
Serial blocks
contain sequential \CC code.  They're also called atomic because the code within
them executes without returning control to the \charmpp\ runtime scheduler, and
thus avoiding interruption from incoming messages. The keywords atomic and serial
are synonymous, and you can find example programs that use atomic. However, we
recommend the use of serial and are considering the deprecation of the atomic keyword.
Typically serial blocks hold
the code that actually deals with incoming messages in a {\tt when} statement,
or to do local operations before a message is sent or after it's received. The
earlier example can be adapted to use serial blocks as follows:
\begin{center}
\begin{alltt}
// in .ci file
chare ComputeObject \{
  entry void ComputeObject();
  entry void startStep();
  entry void firstInput(Input i) \{
    serial \{
      first = i;
      if (\verb|--expectedMessageCount == 0|)
        computeInteractions(first, second);
    \}
  \};
  entry void secondInput(Input j) \{
    serial \{
      second = j;
      if (\verb|--expectedMessageCount == 0|)
        computeInteractions(first, second);
    \}
  \};
\};

// in C++ file
class ComputeObject : public CBase\_ComputeObject \{
  ComputeObject\_SDAG\_CODE
  int   expectedMessageCount;
  Input first, second;

public:
  ComputeObject() \{
    startStep();
  \}
  void startStep() \{
    expectedMessageCount = 2;
  \}

  void computeInteractions(Input a, Input b) \{
    // do computations using a and b
    . . .
    // send off results
    . . .
    // reset for next step
    startStep();
  \}
\};
\end{alltt}
\end{center}
Note that chare classes containing SDAG code must include a few additional declarations
in addition to inheriting from their {\tt CBase\_Foo} class, by incorporating the
{\tt Foo\_SDAG\_CODE} generated-code macro in the class.

Serial blocks can also specify a textual `label' that will appear in traces, as
follows:
\begin{center}
\begin{alltt}
  entry void firstInput(Input i) \{
    serial "process first" \{
      first = i;
      if (\verb|--expectedMessageCount == 0|)
        computeInteractions(first, second);
    \}
  \};
\end{alltt}
\end{center}

In order to control the sequence in which entry methods are processed, SDAG
provides the {\tt when} construct. These statements, also called triggers,
indicate that we expect an incoming message of a particular type, and provide
code to handle that message when it arrives. From the perspective of a chare
object reaching a {\tt when} statement, it is effectively a `blocking
receive.'

Entry methods defined by a {\tt when} are
not executed immediately when a message targeting them is delivered, but
instead are held until control flow in the chare reaches a corresponding {\tt
  when} clause. Conversely, when control flow reaches a {\tt when} clause, the
generated code checks whether a corresponding message has arrived: if one has
arrived, it is processed; otherwise, control is returned to the
\charmpp\ scheduler. 

The use of {\tt when} substantially simplifies the example from above:
\begin{center}
\begin{alltt}
// in .ci file
chare ComputeObject \{
  entry void ComputeObject();
  entry void startStep() \{
    when firstInput(Input first)
      when secondInput(Input second)
        serial \{
          computeInteractions(first, second);
        \}
  \};
  entry void firstInput(Input i);
  entry void secondInput(Input j);
\};

// in C++ file
class ComputeObject : public CBase_ComputeObject \{
  ComputeObject_SDAG_CODE

public:
  ComputeObject() \{
    startStep();
  \}

  void computeInteractions(Input a, Input b) \{
    // do computations using a and b
    . . .
    // send off results
    . . .
    // reset for next step
    startStep();
  \}
\};
\end{alltt}
\end{center}
Like an {\tt if} or {\tt while} in C code, each {\tt when} clause has a body
made up of the statement or block following it. The variables declared as
arguments to the entry method triggering the when are available in the scope of
the body. By using the sequenced execution of SDAG code and the availability of
parameters to when-defined entry methods in their bodies, the counter {\tt
  expectedMessageCount} and the intermediate copies of the received input are
eliminated. Note that the entry methods {\tt firstInput} and {\tt secondInput}
are still declared in the {\tt .ci} file, but their definition is in the SDAG
code. The interface translator generates code to handle buffering and
triggering them appropriately.

For simplicity, {\tt when} constructs can also specify multiple expected entry
methods that all feed into a single body, by separating their prototypes with
commas:
\begin{center}
\begin{alltt}
entry void startStep() \{
  when firstInput(Input first),
       secondInput(Input second)
    serial \{
      computeInteractions(first, second);
    \}
\};
\end{alltt}
\end{center}

A single entry method is allowed to appear in more than one {\tt when} statement.
If only one of those {\tt when} statements has been triggered when the runtime
delivers a message to that entry method, that {\tt when} statement is guaranteed
to process it. If there is no trigger waiting for that entry method, then the
next corresponding {\tt when} to be reached will receive that message. If there is
more than one {\tt when} waiting on that method, which one will receive it is not
specified, and should not be relied upon. For an example of multiple {\tt when}
statements handling the same entry method without reaching the unspecified case,
see the CharmLU benchmark.

To more finely control the correspondence between incoming messages and {\tt when}
clauses associated with the target entry method, SDAG supports \emph{matching} on
reference numbers. Matching is typically used to denote an iteration of a program
that executes asynchronously (without any sort of barrier or other synchronization
between steps) or a particular piece of the problem being solved.
Matching is requested by placing an expression denoting the desired reference
number in square brackets between the entry method name and its parameter list.
For parameter marshalled entry methods, the reference number expression will be
compared for equality with the entry method's first argument. For entry methods
that accept an explicit message (\S~\ref{messages}), the reference number on
the message can be set by calling the function
\verb|CkSetRefNum(void *msg, CMK_REFNUM_TYPE ref)|.
Matching is used in the loop example below, and in
\examplereffile{jacobi2d-sdag/jacobi2d.ci}. Multiple {\tt when} triggers for
an entry method with different matching reference numbers will not conflict - each
will receive only corresponding messages.

SDAG supports the {\tt for} and {\tt while} loop constructs mostly as if they
appeared in plain C or C++ code. In the running example, {\tt
  computeInteractions()} calls {\tt startStep()} when it is finished to start
the next step. Instead of this arrangement, the loop structure can be made
explicit:
\begin{center}
\begin{alltt}
// in .ci file
chare ComputeObject \{
  entry void ComputeObject();
  entry void runForever() \{
    while(true) \{
      when firstInput(Input first),
           secondInput(Input second) serial \{
          computeInteractions(first, second);
      \}
    \}
  \};
  entry void firstInput(Input i);
  entry void secondInput(Input j);
\};

// in C++ file
class ComputeObject : public CBase_ComputeObject \{
  ComputeObject_SDAG_CODE

public:
  ComputeObject() \{
    runForever();
  \}

  void computeInteractions(Input a, Input b) \{
    // do computations using a and b
    . . .
    // send off results
    . . .
  \}
\};
\end{alltt}
\end{center}
If this code should instead run for a fixed number of iterations, we can
instead use a for loop:
\begin{center}
\begin{alltt}
// in .ci file
chare ComputeObject \{
  entry void ComputeObject();
  entry void runForever() \{
    for(iter = 0; iter < n; ++iter) \{
      // Match to only accept inputs for the current iteration
      when firstInput[iter](int a, Input first),
           secondInput[iter](int b, Input second) serial \{
        computeInteractions(first, second);
      \}
    \}
  \};
  entry void firstInput(int a, Input i);
  entry void secondInput(int b, Input j);
\};

// in C++ file
class ComputeObject : public CBase_ComputeObject \{
  ComputeObject_SDAG_CODE
  int n, iter;

public:
  ComputeObject() \{
    n = 10;
    runForever();
  \}

  void computeInteractions(Input a, Input b) \{
    // do computations using a and b
    . . .
    // send off results
    . . .
  \}
\};
\end{alltt}
\end{center}
Note that {\tt int iter;} is declared in the chare's class definition and not
in the {\tt .ci} file. This is necessary because the \charmpp\ interface
translator does not fully parse the declarations in the {\tt for} loop header,
because of the inherent complexities of C++.

SDAG also supports conditional execution of statements and blocks with {\tt if}
statements. The syntax of SDAG {\tt if} statements matches that of C and
C++. However, if one encounters a syntax error on correct-looking code in a
loop or conditional statement, try assigning the condition expression to a
boolean variable in a serial block preceding the statement and then testing that
boolean's value. This can be necessary because of the complexity of parsing C++
code.

In cases where multiple tasks must be processed before execution continues, but
with no dependencies or interactions among them, SDAG provides the {\tt
  overlap} construct. Overlap blocks contain a series of SDAG statements within
them which can occur in any order. Commonly these blocks are used to hold a
series of {\tt when} triggers which can be received and processed in any
order. Flow of control doesn't leave the overlap block until all the statements
within it have been processed.

In the running example, suppose each input needs to be preprocessed independently
before the call to {\tt computeInteractions}. Since we don't care which order
they get processed in, and want it to happen as soon as possible, we can apply
{\tt overlap}:
\begin{center}
\begin{alltt}
// in .ci file
chare ComputeObject \{
  entry void ComputeObject();
  entry void startStep() \{
    overlap \{
      when firstInput(Input i)
        serial \{ first = preprocess(i); \}
      when secondInput(Input j)
        serial \{ second = preprocess(j); \}
     \}
     serial \{
       computeInteractions(first, second);
     \}
  \};
  entry void firstInput(Input i);
  entry void secondInput(Input j);
\};

// in C++ file
class ComputeObject : public CBase_ComputeObject \{
  ComputeObject_SDAG_CODE

public:
  ComputeObject() \{
    startStep();
  \}

  void computeInteractions(Input a, Input b) \{
    // do computations using a and b
    . . .
    // send off results
    . . .
    // reset for next step
    startStep();
  \}
\};
\end{alltt}
\end{center}

Another construct offered by SDAG is the {\tt forall} loop. These loops are
used when the iterations of a loop can be performed independently and in any
order. This is in contrast to a regular {\tt for} loop, in which each iteration
is executed sequentially. The {\tt forall} loop can be seen as an overlap with
an indexed set of otherwise identical statements in the body. Its syntax is 
\begin{center}
\begin{alltt}
forall [IDENT] (MIN:MAX,STRIDE) BODY
\end{alltt}
\end{center}
The range from MIN to MAX is inclusive. Its use is demonstrated through
distributed parallel matrix-matrix multiply shown in
\examplereffile{matmul/matmul.ci}

\subsection{The \texttt{case} Statement}

The \texttt{case} statement in SDAG expresses a disjunction over a set of
\texttt{when} clauses. In other words, if it is known that one dependency out
of a set will be satisfied, but which one is not known, this statement allows
the set to be specified and will execute the corresponding block based on which
dependency ends up being fulfilled.

The following is a basic example of the \texttt{case} statement. Note that the
trigger \texttt{b(), d()} will only be fulfilled if both \texttt{b()} and
\texttt{d()} arrive. If only one arrives, then it will partially match, and the
runtime will not ``commit'' to this branch until the second arrives. If another
dependency fully matches, the partial match will be ignored and can be used to
trigger another \texttt{when} later in the execution.

\begin{verbatim}
case {
  when a() { }
  when b(), d() { }
  when c() { }
}
\end{verbatim}

A full example of the \texttt{case} statement can be found
\testreffile{sdag/case/caseTest.ci}.

\section{Usage Notes}

If you've added \sdag\ code to your class, you must link in the code by:
\begin{itemize}
  \item Adding ``{\it className}\_SDAG\_CODE'' inside the class declaration
     in the .h file.  This macro defines the entry points and support
     code used by \sdag{}.  Forgetting this results in a compile error
     (undefined sdag entry methods referenced from the .def.h file).
  \item Adding a call to the pup routine ``\_\_sdag\_pup(p);'' from your pup routine.
     Forgetting this results in failure after migration.
\end{itemize}
For example, an array named ``Foo'' that uses sdag code might contain:
\begin{center}
\begin{alltt}
class Foo : public CBase_Foo \{
public:
    Foo_SDAG_CODE
    Foo(...) \{
       ...
    \}
    Foo(CkMigrateMessage *m) \{ \}
    
    void pup(PUP::er &p) \{
       CBase_Foo::pup(p);
       __sdag_pup(p);
    \}
    . . .
\};
\end{alltt}
\end{center}

\zap{
\section{Relationship to Threads}

Threads are typically used to perform the abovementioned sequencing.
Lets us code our previous example using threads.

%\begin{figure}[ht]
\begin{center}
\begin{alltt}
void compute_thread(int first_index, int second_index)
\{
    getPatch(first_index);
    getPatch(second_index);
    threadId[0] = createThread(recvFirst);
    threadId[1] = createThread(recvSecond);
    threadJoin(2, threadId);
    computeInteractions(first, second);
  \}
  void recvFirst(void)
  \{
    recv(first, sizeof(Patch), ANY_PE, FIRST_TAG);
    filter(first);
  \}
  void recvSecond(void)
  \{
    recv(second, sizeof(Patch), ANY_PE, SECOND_TAG);
    filter(second);
  \}
\end{alltt}
\end{center}
%\caption{Compute Thread in a Molecular Dynamics Application}
%\label{figthreadexample}
%\end{figure}

Contrast the compute chare-object example in figure~\ref{figchareexample} with
a thread-based implementation of the same scheme in
figure~\ref{figthreadexample}. Functions \uw{getFirst}, and \uw{getSecond} send
messages asynchronously to the PatchManager, requesting that the specified
patches be sent to them, and return immediately. Since these messages with
patches could arrive in any order, two threads, \uw{recvFirst} and
\uw{recvSecond}, are created. These threads block, waiting for messages to
arrive. After each message arrives, each thread performs the filtering
operation. The main thread waits for these two threads to complete, and then
computes the pairwise interactions. Though the programming complexity of
buffering the messages and maintaining the counters has been eliminated in this
implementation, considerable overhead in the form of thread creation, and
synchronization in the form of {\em join} has been added. Let us now code the
same example in \sdag. It reduces the parallel programming complexity without
adding any significant overhead.

%\begin{figure}[ht]
\begin{center}
\begin{alltt}
  array[1D] compute_object \{
    entry void recv_first(Patch *first);
    entry void recv_second(Patch *first);
    entry void compute_object(MSG *msg)\{
      serial \{
         PatchManager->Get(msg->first_index,\dots);
         PatchManager->Get(msg->second_index,\dots);
      \}
      overlap \{
        when recv_first(Patch *first) serial \{ filter(first); \}
        when recv_second(Patch *second) serial \{ filter(second); \}
      \}
      serial \{ computeInteractions(first, second); \}
    \}
  \}
\end{alltt}
\end{center}
%\caption{\sdag\ Implementation of the Compute Object}
%\label{figsdagexample}
%\end{figure}
}


\zap{
\section{Grammar}

\paragraph{Tokens}

\begin{alltt}
  <ident> = Valid \CC{} identifier 
  <int-expr> = Valid \CC{} integer expression 
  <\CC{}-code> = Valid \CC{} code 
\end{alltt}

\paragraph{Grammar in EBNF Form}

\begin{alltt}
<sdag> := <class-decl> <sdagentry>+ 

<class-decl> := "class" <ident> 

<sdagentry> := "sdagentry" <ident> "(" <ident> "*" <ident> ")" <body> 

<body> := <stmt> 
        | "\{" <stmt>+ "\}" 

<stmt> := <overlap-stmt> 
        | <when-stmt> 
        | <atomic-stmt> 
        | <if-stmt> 
        | <while-stmt> 
        | <for-stmt> 
        | <forall-stmt> 

<overlap-stmt> := "overlap" <body> 

<atomic-stmt> := "serial" "\{" <\CC-code> "\}" 

<if-stmt> := "if" "(" <int-expr> ")" <body> [<else-stmt>] 

<else-stmt> := "else" <body> 

<while-stmt> := "while" "(" <int-expr> ")" <body> 

<for-stmt> := "for" "(" <c++-code> ";" <int-expr> ";" <c++-code> ")" <body> 

<forall-stmt> := "forall" "[" <ident> "]" "(" <range-stride> ")" <body> 

<range-stride> := <int-expr> ":" <int-expr> "," <int-expr> 

<when-stmt> := "when" <entry-list>  <body> 

<entry-list> := <entry> 
              | <entry> [ "," <entry-list> ] 

<entry> := <ident> [ "[" <int-expr> "]" ] "(" <ident> "*" <ident> ")" 
  
\end{alltt}
}

\zap{
\sdag\ is a coordination language built on top of \charmpp\ that supports the
sequencing mentioned above, while overcoming limitations of thread-based
languages, and facilitating a clear expression of flow of control within the
object without losing the performance benefits of adaptive message-driven
execution.  In other words, \sdag\ is a structured notation for specifying
intra-process control dependences in message-driven programs. It combines the
efficiency of message-driven execution with the explicitness of control
specification. \sdag\ allows easy expression of dependences among messages and
computations and also among computations within the same object using
when-blocks and various structured constructs.  \sdag\ is adequate for
expressing control-dependencies that form a series-parallel control-flow graph.
\sdag\ has been developed on top of \charmpp\. \sdag\ allows \charmpp\ entry
methods (in chares, groups or arrays) to specify code (a when-block body) to be
executed upon occurrence of certain events.  These events (or guards of a
when-block) are entry methods of the object that can be invoked remotely. While
writing a \sdag\ program, one has to declare these entries in \charmpp\
interface file. The implementation of the entry methods that contain the
when-block is written using the \sdag\ language. Grammar of \sdag\ is given in
the EBNF form below.

\subsubsection{Usage}

You can use SDAG to implement entry methods for any chare, chare array, group,
or nodegroup. Any entry method implemented using SDAG must be implemented in the
interface (.ci) file for its class. An SDAG entry method consists of a series of
SDAG constructs of the following kinds:

\begin{itemize}
    \item {\tt forall} loops: 
    \item {\tt if}, {\tt for}, and {\tt while} statements: these statements have
        the same meaning as the normal {\tt if}, {\tt for}, and {\tt while}
        loops in sequential \CC programs. This allows the programmer to use
        common control flow constructs outside the context of serial blocks.
\end{itemize}

\sdag{} code can be inserted into the .ci file for any array, group, or chare's entry methods.

For more details regarding \sdag{}, look at \examplerefdir{hello/sdag}
}

