\documentclass[10pt]{article}

\usepackage{../pplmanual}
\input{../pplmanual}

\makeindex

\title{\charmpp\\ Debugger Manual}
\version{0.2}
\credits{
By Filippo Gioachin, Rashmi Jyothi, Orion Lawlor, Eric Bohm
}
%%Rashmi didn't actually write the manual, but she wrote the debugger and I'm cutting and pasting much of this document from her thesis and the _Debugging Support for Charm++_ paper written by Rashmi and Orion. - Eric
%% After starting working on the debugger, and implemented the memory views, I revised the entire manual to be conform to the new implementation of CharmDebug. - Filippo
\begin{document}

\maketitle

\section{Introduction}
\label{sec:intro}
\input{intro}


\section{Building the \charmpp{} Debug Tool}

To get the Charm Debug tool, check out the source code from the repository.
This will create a directory named ccs\_tools. Move to this directory and 
build Charm Debug.

\begin{verbatim}
 git clone git://charm.cs.uiuc.edu/ccs_tools.git 
 cd ccs_tools
 ant
\end{verbatim}

This will create the executable {\tt bin/charmdebug}, which should work.

You can also download the binaries from the \charmpp{} downloads website and use it
directly without building.
(NOTE: Binaries may not be properly working in some platforms, so building from the
source code is recommended.)


\section{Preparing the \charmpp Application for Debugging}

No instrumentation is required to use the \charmpp{} debugger.  Being
CCS based, you can use it to set and step through entry point
breakpoints and examine \charmpp{} structures on any \charmpp{}
application.

Nevertheless, for some features to be present some additional options might
be required at either compile or link time:

\begin{itemize}

\item In order to provide a symbol conversion of the assembly code executed
by the application, the \textit{-g} option is needed at compile time. This
conversion is needed to provide function names as well as source file names
and line numbers wherever useful. This is useful also to fully utilize gdb
(or any other serial debugger) on one or more processes.

\item Optimization options, with their nature of transforming the source code,
can produce a mismatch between the function displayed in the debugger (for
example in a stack trace) and the functions present in the source code. To
produce information coherent with source code, optimization is discouraged.

\item The link time option \textit{-memory charmdebug} is needed in
order to gather information regarding the memory of the application. This
includes both the Memory view (see \ref{sec:memory}) and the Inspector
framework (see \ref{sec:inspector}).

\end{itemize}

\subsection{Record Replay}

The \textit{Record Replay} feature is independant of the charmdebug
application.  It is a mechanism used to detect bugs that happen only
once in a while depending on the order in which messages are
processed. The program in consideration is first run in a record mode
which produces a trace. When the program is run in replay mode it uses
a previous trace gotten from a record run to ensure that messages are
processed in the same order as the recorded run. The idea is to make
use of a message-sequence number and a theorem says that the serial
numbers will be the same if the messages are processed in the same
order. \cite{rashmithesis}

\textit{Record Replay} tracing is automatically enabled for \charmpp{}
programs and requires nothing special to be done during compilation
(linking with the option ``\textit{-tracemode recordreplay}'' used to
be necessary).  At run time, the ``\textit{+record}'' option is used,
which records messages in order in a file for each processor.  The
same execution order can be replayed using the ``\textit{+replay}''
runtime option, which can be used at the same time as the other
debugging tools in \charmpp{}.

\emph{ Note! } If your \charmpp{} is built with {\tt CMK\_OPTIMIZE} on,
all tracing will be disabled. So, use an unoptimized \charmpp{} to do your
debugging.

\section{Running the Debugger}

\subsection{Charmdebug command line parameters}

\begin{description}

\item[{\tt -pes}] Number of PEs 

\item[{\tt +p}] Number of PEs 

\item[{\tt -host}] hostname of CCS server for application

\item[{\tt -user}] the username to use to connect to the hostname selected

\item[{\tt -port}] portnumber of CCS server for application

\item[{\tt -sshtunnel}] force the communication between client and server
(in particular the one for CCS) to be tunnelled through ssh. This allow the
bypass of firewalls.

\item[{\tt -display}] X Display

\end{description}

\subsection{Basic usage}

To run an application locally via the debugger on 4 pes with command line options for your pgm (e.g. opt1 opt2 ):

\begin{verbatim}
  charmdebug pgm +p4 4 opt1 opt2
\end{verbatim}

If the application should be run in a remote cluster behind a firewall, the previous command line will become:

\begin{verbatim}
  charmdebug -host cluster.inst.edu -user myname -sshtunnel pgm +p4 4 opt1 opt2
\end{verbatim}

Charmdebug can also be executed without any parameters.  The user can
then choose the application to launch and its command line parameters
from within the \texttt{File} menu as shown in Figure~\ref{menu}.

\begin{figure}[]
\includegraphics[scale=0.5, height=3in, width=3in]{figs/menu}
\caption{Using the menu to set parameters for the \charmpp{} program}
\label{menu}
\end{figure}


\emph{Note: charmdebug command line launching only works on net-*
builds of \charmpp{} }

To replay a previously recorded session:

\begin{verbatim}
  charmdebug pgm +p4 opt1 opt2  +replay
\end{verbatim}

%%uncomment this when we fix the debugger so it works remotely

%% \subsection{Connecting to a running application}

%% To connect the debugger to a previously launched application.  The
%% application needs to have been launched with the ccs server feature
%% enabled.

%% \begin{verbatim}
%%   charmrun pgm ++server ++server-port 29999
%%   charmdebug pgm -port 29999
%% \end{verbatim}


%% \subsection{Connecting to a remote application}
%% \begin{verbatim}
%%   charmdebug pgm +p4 opt1 opt2   -host remotehost -port 29999
%% \end{verbatim}

%% To have your application freeze upon start so you can initiate flow
%% control via the debugger.
%% \begin{verbatim}
%%   charmdebug pgm +p4 opt1 opt2   -host remotehost -port 29999 +cpd
%% \end{verbatim}

\subsection{Charm Debugging Related Options}

When using the charm debugger to launch your application, it will automatically
set these to defaults appropriate for most situations.

\begin{description}

\item[{\tt +cpd}] Triggers application freeze at startup for debugger.

\item[{\tt ++charmdebug}] Triggers charmrun to provide some information about the
executable, as well as provide an interface to gdb for querying.

\item[{\tt +debugger}] Which debuggers to use.

\item[{\tt ++debug}] Run each node under gdb in an xterm window, prompting
the user to begin execution.

\item[{\tt ++debug-no-pause}] Run each node under gdb in an xterm window
immediately (i.e. without prompting the user to begin execution).

 \emph{ Note: } If you're using the charm debugger it will probably be
 best to control the sequential (i.e. gdb) debuggers from within its
 GUI interface.
 
\item[{\tt ++DebugDisplay}]          X Display for xterm

\item[{\tt ++server-port}]      Port to listen for CCS requests

\item[{\tt ++server}]           Enable client-server (CCS) mode

\item[{\tt +record}]           Use the recordreplay tracemode to record the exact event/message sequence for later use.  

\item[{\tt +replay}]           Force the use of recorded log of events/messages  to exactly reproduce a previous run.

  The preceding pair of commands {\tt +record +replay} are used to
  produce the ``instant replay'' feature.  This feature is valuable
  for catching errors which only occur sporadically.  Such bugs which
  arise from the nondeterminacy of parallel execution can be
  fiendishly difficult to replicate in a debugging environment.
  Typical usage is to keep running the application with +record until
  the bug occurs.  Then run the application under the debugger with
  the +replay option.

\end{description}

\subsection{Charmdebug limitations}

\subsubsection{Clusters}

Charmdebug is currently limited to applications started directly by the debugger
due to implementation peculiarities. It will be extended to support connection
to remote running applications in the near future. 
%%Behavior on a cluster can be
%%recorded and then replayed in a local run.

Due to the current implementation, the debugging tool is limited to net-*
versions. Other builds of \charmpp{} might have unexpected behaviour. In the near
future this will be extended at least to the mpi-* versions.

\subsubsection{Record Replay}
The \texttt{record replay} feature does not work well with spontaneous
events.  Load balancing is the most common form of spontaneous event
in that it occurs periodically with no other causal event. As per

\begin{figure}[]
\includegraphics[scale=0.5,height=4in, width=3in]{figs/snapshot3}
\caption{Parallel debugger when a break point is reached}
\label{snapshot3}
\end{figure}

As per Rashmi's thesis: \textit{ There are some unique issues for
replay in the context of Charm because it provides high-level support
for dynamic load balancing, quiescence detection and information
sharing. Many of the load balancing strategies in Charm have a
spontaneous component. The strategy periodically checks the sizes of
the queues on the local processor. A replay load balancing strategy
implements the known load redistribution. The behavior of the old
balancing strategy is therefore not replayed only its effect is. Since
minimal tracing is used by the replay mechanism the amount of
perturbation due to tracing is reduced. The replay mechanism is
proposed as a debugging support to replay asynchronous message arrival
orders.}

Moreover, if your application crashes without a clean shutdown, the log may be
lost with the application.

\subsection{Using the Debugger}
\label{sec:using}
\input{using}


\section{Debugger Implementation Details}
\label{sec:implementation}
\input{implementation}

\end{document}
