\subsection{Coding}

To activate load balancing module and prepare objects for migration, there are 3
things that needs to be added in Charisma code. 

First, the programmer needs to inform Charisma about the load balancing with a
"\code{define ldb;}" statement in the header section of the orchestration code.
This will make Charisma generates extra Charm++ code to do load balancing such
as \code{PUP} methods.

Second, the user has to provide a \code{PUP} function for each class with
sequential data that needs to be moved when the object migrates. When choosing
which data items to \code{pup}, the user has the flexibility to leave the dead
data behind to save on communication overhead in migration. The syntax for the
sequential \code{PUP} is similar to that in a Charm++ program. Please refer to
the load balancing section in Charm++ manual for more information on \code{PUP}
functions. A typical example would look like this in user's sequential \code{.C}
file: 

\begin{SaveVerbatim}{foodecl}
  void JacobiWorker::sequentialPup(PUP::er& p){
    p|myLeft; p|myRight; p|myUpper; p|myLower;
    p|myIter;
    PUParray(p,(double *)localData,1000);
  }
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

Thirdly, the user will make the call to invoke load balancing session in the
orchestration code. The call is \code{AtSync();} and it is invoked on all
elements in an object array. The following example shows how to invoke load
balancing session every 4th iteration in a for-loop. 

\begin{SaveVerbatim}{foodecl}
  for iter = 1 to 100
    // work work
    if(iter % 4 == 0) then
      foreach i in workers
        workers[i].AtSync();
      end-foreach
    end-if
  end-for
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}

If a while-loop is used instead of for-loop, then the test-condition in the
\code{if} statement is a local variable in the program's MainChare. In the
sequential code, the user can maintain a local variable called \code{iter} in
MainChare and increment it every iteration. 


\subsection{Compiling and Running}
Unless linked with load balancer modules, a Charisma program will not perform
actual load balancing. The way to link in a load balancer module is adding
\code{-module EveryLB} as a link-time option. 

At run-time, the load balancer is specified in command line after the
\code{+balancer} option. If the balancer name is incorrect, the job launcher will
automatically print out all available load balancers. For instance, the following
command uses \code{RotateLB}. 

\begin{SaveVerbatim}{foodecl}
    > ./charmrun ./pgm +p16 +balancer RotateLB
\end{SaveVerbatim}
\smallfbox{\BUseVerbatim{foodecl}}



