\experimental{}
\charmpp{} currently includes an experimental automatic tuning
framework that can dynamically adapt a program at runtime to improve
its performance. The program provides a set of tunable knobs that are
adjusted automatically by the tuning framework. The user program also
provides information about the control points so that intelligent
tuning choices can be made. This information will be used to steer the
program instead of requiring the tuning framework to blindly search
the possible program configurations.

\textbf{Warning: this is still an experimental feature not meant for production applications}

\section{Exposing Control Points in a Charm++ Program}
The program should include a header file before any of its \texttt{*.decl.h} files:

\begin{alltt} 
    #include <controlPoints.h> 
\end{alltt} 

The control point framework initializes itself, so no changes need to be made at startup in the program.

The program will request the values for each control point on PE 0. Control
point values are non-negative integers:

\begin{alltt} 
    my_var = controlPoint("any_name", 5, 10);
    my_var2 = controlPoint("another_name", 100,101);
\end{alltt} 

To specify information about the effects of each control point, make calls such as these once on PE 0 before accessing any control point values:

\begin{alltt} 
    ControlPoint::EffectDecrease::Granularity("num_chare_rows");
    ControlPoint::EffectDecrease::Granularity("num_chare_cols");
    ControlPoint::EffectIncrease::NumMessages("num_chare_rows");
    ControlPoint::EffectIncrease::NumMessages("num_chare_cols");
    ControlPoint::EffectDecrease::MessageSizes("num_chare_rows");
    ControlPoint::EffectDecrease::MessageSizes("num_chare_cols");
    ControlPoint::EffectIncrease::Concurrency("num_chare_rows");
    ControlPoint::EffectIncrease::Concurrency("num_chare_cols");
    ControlPoint::EffectIncrease::NumComputeObjects("num_chare_rows");
    ControlPoint::EffectIncrease::NumComputeObjects("num_chare_cols");
\end{alltt} 

For a complete list of these functions, see \texttt{cp\_effects.h} in \texttt{charm/include}.


The program, of course, has to adapt its behavior to use these new control point values. There are two ways for a the control point values to change over time. The program can request that a new phase (with its own control point values) be used whenever it wants, or the control point framework can automatically advance to a new phase periodically. The structure of the program will be slightly different in these to cases. Sections \ref{frameworkAdvancesPhases} and \ref{programAdvancesPhases} describe the additional changes to the program that should be made for each case.

\subsection{Control Point Framework Advances Phases}
\label{frameworkAdvancesPhases}

The program provides a callback to the control point framework in a manner such as this:

\begin{alltt} 
    // Once early on in program, create a callback, and register it 
    CkCallback cb(CkIndex_Main::granularityChange(NULL),thisProxy); 
    registerCPChangeCallback(cb, true);
\end{alltt} 

In the callback or after the callback has executed, the program should request the new control point values on PE 0, and adapt its behavior appropriately.

Alternatively, the program can specify that it wants to call \texttt{gotoNextPhase();} itself when it is ready. Perhaps the program wishes to delay its adaptation for a while. To do this, it specifies \texttt{false} as the final parameter to \texttt{registerCPChangeCallback} as follows:

\begin{alltt} 
   registerCPChangeCallback(cb, false);
\end{alltt} 


\subsection{Program Advances Phases}
\label{programAdvancesPhases}

\begin{alltt} 
     registerControlPointTiming(duration); // called after each program iteration on PE 0
     gotoNextPhase(); // called after some number of iterations on PE 0
    // Then request new control point values
\end{alltt} 



\section{Linking With The Control Point Framework}

The control point tuning framework is now an integral part of the Charm++ runtime system. It does not need to be linked in to an application in any special way. It contains the framework code responsible for recording information about the running program as well as adjust the control point values. The trace module will enable measurements to be gathered including information about utilization, idle time, and memory usage. 

\section{Runtime Command Line Arguments}

Various following command line arguments will affect the behavior of the program when running with the control point framework. As this is an experimental framework, these are subject to change.

The scheme used for tuning can be selected at runtime by the use of one of the following options:
\begin{alltt} 
     +CPSchemeRandom            Randomly Select Control Point Values
 +CPExhaustiveSearch            Exhaustive Search of Control Point Values
      +CPSimulAnneal            Simulated Annealing Search of Control Point Values
 +CPCriticalPathPrio            Use Critical Path to adapt Control Point Values
        +CPBestKnown            Use BestKnown Timing for Control Point Values
         +CPSteering            Use Steering to adjust Control Point Values
      +CPMemoryAware            Adjust control points to approach available memory
\end{alltt} 

To intelligently tune or steer an application's performance, performance measurements ought to be used. Some of the schemes above require that memory footprint statistics and utilization statistics be gathered. All measurements are performed by a tracing module that has some overheads, so is not enabled by default. To use any type of measurement based steering scheme, it is necessary to add a runtime command line argument to the user program to enable the tracing module:

\begin{alltt} 
    +CPEnableMeasurements
\end{alltt}

The following flags enable the gathering of the different types of available measurements.
\begin{alltt} 
        +CPGatherAll            Gather all types of measurements for each phase
+CPGatherMemoryUsage            Gather memory usage after each phase
+CPGatherUtilization            Gather utilization \& Idle time after each phase
\end{alltt} 


The control point framework will periodically adapt the control point values. The following command line flag determines the frequency at which the control point framework will attempt to adjust things.
\begin{alltt} 
     +CPSamplePeriod     number The time between Control Point Framework samples (in seconds)
\end{alltt} 

The data from one run of the program can be saved and used in subsequent runs. The following command line arguments specify that a file named \texttt{controlPointData.txt} should be created or loaded. This file contains measurements for each phase as well as the control point values used in each phase. 
\begin{alltt} 
         +CPSaveData            Save Control Point timings \& configurations at completion
         +CPLoadData            Load Control Point timings \& configurations at startup
     +CPDataFilename            Specify the data filename 
\end{alltt} 

It might be useful for example, to run once with \texttt{+CPSimulAnneal +CPSaveData} to try to find a good configuration for the program, and then use  \texttt{+CPBestKnown +CPLoadData} for all subsequent production runs.

