
\section{BigSim Emulator}
\label{bgemulator}

The BigSim emulator environment is designed with the following
objectives:

\begin{enumerate}
\item To support a realistic BigSim API on existing parallel machines

\item To obtain first-order performance estimates of algorithms

\item To facilitate implementations of alternate programming models for
      Blue Gene
\end{enumerate}

The machine supported by the emulator consists of three-dimensional grid of
1-chip nodes.  The user may specify the size of the machine along each
dimension (e.g. 34x34x36).  The chip supports $k$ threads (e.g. 200), each with
its own integer unit.  The proximity of the integer unit with individual memory
modules within a chip is not currently modeled.

The API supported by the emulator can be broken down into several
components:

\begin{enumerate}
\item Low-level API for chip-to-chip communication
\item Mid-level API that supports local micro-tasking with a chip level
scheduler with features such as: read-only variables, reductions, broadcasts,
distributed tables, get/put operations
\item Migratable objects with automatic load balancing support
\end{enumerate}

Of these, the first two have been implemented.  The simple time stamping
algorithm, without error correction, has been implemented.  More
sophisticated timing algorithms, specifically aimed at error correction,
and more sophisticated features (2, 3, and others), as well as libraries
of commonly needed parallel operations are part of the proposed work for
future.

The following sections define the appropriate parts of the API, with
example programs and instructions for executing them.

\subsection{BigSim Programming Environment}

The basic philosophy of the BigSim Emulator is to hide intricate details of the
simulated machine from the application developer. Thus, the application
developer needs to provide initialization details and handler functions only and
gets the result as though running on a real machine.  Communication, Thread
creation, Time Stamping, etc are done by the emulator.

\subsubsection{BigSim API: Level 0}

\function{void addBgNodeInbuffer(bgMsg *msgPtr, int nodeID)}
\desc{
        low-level primitive invoked by Blue Gene emulator to put the 
        message to the inbuffer queue of a node.

        msgPtr - pointer to the message to be sent to target node; 

        nodeID - node ID of the target node, it is the serial number of a 
                 bluegene node in the emulator's physical node.
}

\function{void addBgThreadMessage(bgMsg *msgPtr, int threadID)}
\desc{
        add a message to a thread's affinity queue, these messages can be 
 	only executed by a specific thread indicated by threadID.
}

\function{void addBgNodeMessage(bgMsg *msgPtr)}
\desc{
	add a message to a node's non-affinity queue, these messages can be 
	executed by any thread in the node.
}

\function{boolean checkReady()}
\desc{
        invoked by communication thread to see if there is any unattended
        message in inBuffer.
}

\function{bgMsg * getFullBuffer()}
\desc{
	invoked by communication thread to retrieve the unattended message 
	in inBuffer.
}

\function{CmiHandler msgHandlerFunc(char *msg)}
\desc{
	Handler function type that user can register to handle the message.
}

\function{void sendPacket(int x, int y, int z, int msgSize,bgMsg *msg)}
\desc{
	chip-to-chip communication function. It send a message to Node[x][y][z].
        
	bgMsg is the message type with message envelope used internally.
}

\subsubsection{Initialization API: Level 1a}

All the functions defined in API Level 0 are used internally for the 
implementation of bluegene node communication and worker threads.

From this level, the functions defined are exposed to users to write bluegene
programs on the emulator.

Considering that the emulator machine will emulate several Bluegene nodes on
each physical node, the emulator program defines this function 
\function{BgEmulatorInit(int argc, char **argv)} to initialize each emulator
node. In this function, user program can define the Bluegene machine size,
number of communication/worker threads, and check the command line arguments.

The size of the simulated machine being emulated and the number of thread per
node is determined either by the command line arguments or calling following
functions:

\function{void BgSetSize(int sx, int sy, int sz)}
\desc{
	set Blue Gene Machine size;
}

\function{void BgSetNumWorkThread(int num)}
\desc{
	set number of worker threads per node;
}

\function{void BgSetNumCommThread(int num)}
\desc{
	set number of communication threads per node;
}

\function{int BgRegisterHandler(BgHandler h)}
\desc{
	register user message handler functions; 
}

For each simulated node, the execution starts at 
\function{BgNodeStart(int argc, char **argv)} called by the emulator,
where application handlers can be registered and computation 
is triggered by creating a task at required nodes.

Similar to pthread's thread specific data, each bluegene node has its
own node specific data associated with it. To do this, the user needs to define its 
own node-specific variables encapsulated in a struct definition and register
 the pointer to the data with the emulator by following function:

\function{void BgSetNodeData(char *data)}

To retrieve the node specific data, call:

\function{char *BgGetNodeData();}

After completion of execution, user program invokes a function:

\function{void BgShutdown()}

to terminate the emulator.

\subsubsection{Handler Function API: Level 1a}

The following functions can be called in user's application program to retrieve
the simulated machine information, get thread execution time, and perform the
communication.

\function{void BgGetSize(int *sx, int *sy, int *sz);}

\function{int BgGetNumWorkThread();}

\function{int BgGetNumCommThread();}

\function{int BgGetThreadID();}

\function{double BgGetTime();}

\function{void BgSendPacket(int x, int y, int z, int threadID, int handlerID, WorkType type, int numbytes, char* data);}
\desc{
This sends a trunk of data to Node[x, y, z] and also specifies the
handler function to be used for this message i.e. the handlerID;
threadID specifies the desired thread to handle the message, ANYTHREAD means
no preference.

To specify the thread category:
\begin{description}
\item[1:] a small piece of work that can be done by
communication thread itself, so NO scheduling overhead.
\item[0:] a large piece of work, so communication thread
schedules it for a worker thread
\end{description}
}


\subsection{Writing a BigSim Application}

\subsubsection{Application Skeleton}

\begin{alltt}
Handler function prototypes;
Node specific data type declarations;

void  BgEmulatorInit(int argc, char **argv)  function
  Configure bluegene machine parameters including size, number of threads, etc.
  You also need to register handlers here.

void *BgNodeStart(int argc, char **argv) function
  The usual practice in this function is to send an initial message to trigger 
  the execution.
  You can also register node specific data in this function.

Handler Function 1, void handlerName(char *info)
Handler Function 2, void handlerName(char *info)
..
Handler Function N, void handlerName(char *info)

\end{alltt}

\subsubsection{Sample Application 1}

\begin{verbatim}
/* Application: 
 *   Each node starting at [0,0,0] sends a packet to next node in
 *   the ring order.
 *   After node [0,0,0] gets message from last node
 *   in the ring, the application ends.
 */


#include "blue.h"

#define MAXITER 2

int iter = 0;
int passRingHandler;

void passRing(char *msg);

void nextxyz(int x, int y, int z, int *nx, int *ny, int *nz)
{
  int numX, numY, numZ;

  BgGetSize(&numX, &numY, &numZ);
  *nz = z+1; *ny = y; *nx = x;
  if (*nz == numZ) {
    *nz = 0; (*ny) ++;
    if (*ny == numY) {
      *ny = 0; (*nx) ++;
      if (*nx == numX) *nx = 0;
    }
  }
}

void BgEmulatorInit(int argc, char **argv)
{
  passRingHandler = BgRegisterHandler(passRing);
}

/* user defined functions for bgnode start entry */
void BgNodeStart(int argc, char **argv)
{
  int x,y,z;
  int nx, ny, nz;
  int data, id;

  BgGetXYZ(&x, &y, &z);
  nextxyz(x, y, z, &nx, &ny, &nz);
  id = BgGetThreadID();
  data = 888;
  if (x == 0 && y==0 && z==0) {
    BgSendPacket(nx, ny, nz, -1,passRingHandler, LARGE_WORK, 
				sizeof(int), (char *)&data);
  }
}

/* user write code */
void passRing(char *msg)
{
  int x, y, z;
  int nx, ny, nz;
  int id;
  int data = *(int *)msg;

  BgGetXYZ(&x, &y, &z);
  nextxyz(x, y, z, &nx, &ny, &nz);
  if (x==0 && y==0 && z==0) {
    if (++iter == MAXITER) BgShutdown();
  }
  id = BgGetThreadID();
  BgSendPacket(nx, ny, nz, -1, passRingHandler, LARGE_WORK, 
				sizeof(int), (char *)&data);
}

\end{verbatim}


\subsubsection{Sample Application 2}

\begin{verbatim}

/* Application: 
 *   Find the maximum element.
 *   Each node computes maximum of it's elements and
 *   the max values it received from other nodes
 *   and sends the result to next node in the reduction sequence.
 * Reduction Sequence: Reduce max data to X-Y Plane
 *   Reduce max data to Y Axis
 *   Reduce max data to origin.
 */


#include <stdlib.h>
#include "blue.h"

#define A_SIZE 4

#define X_DIM 3
#define Y_DIM 3
#define Z_DIM 3

int REDUCE_HANDLER_ID;
int COMPUTATION_ID;

extern "C" void reduceHandler(char *);
extern "C" void computeMax(char *);

class ReductionMsg {
public:
  int max;
};

class ComputeMsg {
public:
  int dummy;
};

void BgEmulatorInit(int argc, char **argv)
{
  if (argc < 2) { 
    CmiAbort("Usage: <program> <numCommTh> <numWorkTh>\n"); 
  }

  /* set machine configuration */
  BgSetSize(X_DIM, Y_DIM, Z_DIM);
  BgSetNumCommThread(atoi(argv[1]));
  BgSetNumWorkThread(atoi(argv[2]));

  REDUCE_HANDLER_ID = BgRegisterHandler(reduceHandler);
  COMPUTATION_ID = BgRegisterHandler(computeMax);

}

void BgNodeStart(int argc, char **argv) {
  int x, y, z;
  BgGetXYZ(&x, &y, &z);

  ComputeMsg *msg = new ComputeMsg;
  BgSendLocalPacket(ANYTHREAD, COMPUTATION_ID, LARGE_WORK, 
			sizeof(ComputeMsg), (char *)msg);
}

void reduceHandler(char *info) {
  // assumption: THey are initialized to zero?
  static int max[X_DIM][Y_DIM][Z_DIM];
  static int num_msg[X_DIM][Y_DIM][Z_DIM];

  int i,j,k;
  int external_max;

  BgGetXYZ(&i,&j,&k);
  external_max = ((ReductionMsg *)info)->max;
  num_msg[i][j][k]++;

  if ((i == 0) && (j == 0) && (k == 0)) {
    // master node expects 4 messages:
    // 1 from itself;
    // 1 from the i dimension;
    // 1 from the j dimension; and
    // 1 from the k dimension
    if (num_msg[i][j][k] < 4) {
      // not ready yet, so just find the max
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
    } else {
      // done. Can report max data after making last comparison
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
      CmiPrintf("The maximal value is %d \n", max[i][j][k]);
      BgShutdown();
      return;
    }
  } else if ((i == 0) && (j == 0) && (k != Z_DIM - 1)) {
    // nodes along the k-axis other than the last one expects 4 messages:
    // 1 from itself;
    // 1 from the i dimension;
    // 1 from the j dimension; and
    // 1 from the k dimension
    if (num_msg[i][j][k] < 4) {
      // not ready yet, so just find the max
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
    } else {
      // done. Forwards max data to node i,j,k-1 after making last comparison
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
      ReductionMsg *msg = new ReductionMsg;
      msg->max = max[i][j][k];
      BgSendPacket(i,j,k-1,ANYTHREAD,REDUCE_HANDLER_ID,LARGE_WORK, 
				sizeof(ReductionMsg), (char *)msg);
    }
  } else if ((i == 0) && (j == 0) && (k == Z_DIM - 1)) {
    // the last node along the k-axis expects 3 messages:
    // 1 from itself;
    // 1 from the i dimension; and
    // 1 from the j dimension
    if (num_msg[i][j][k] < 3) {
      // not ready yet, so just find the max
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
    } else {
      // done. Forwards max data to node i,j,k-1 after making last comparison
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
      ReductionMsg *msg = new ReductionMsg;
      msg->max = max[i][j][k];
      BgSendPacket(i,j,k-1,ANYTHREAD,REDUCE_HANDLER_ID,LARGE_WORK, 
				sizeof(ReductionMsg), (char *)msg);
    }
  } else if ((i == 0) && (j != Y_DIM - 1)) {
    // for nodes along the j-k plane except for the last and first row of j,
    // we expect 3 messages:
    // 1 from itself;
    // 1 from the i dimension; and
    // 1 from the j dimension
    if (num_msg[i][j][k] < 3) {
      // not ready yet, so just find the max
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
    } else {
      // done. Forwards max data to node i,j-1,k after making last comparison
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
      ReductionMsg *msg = new ReductionMsg;
      msg->max = max[i][j][k];
      BgSendPacket(i,j-1,k,ANYTHREAD,REDUCE_HANDLER_ID,LARGE_WORK, 
				sizeof(ReductionMsg), (char *)msg);
    }
  } else if ((i == 0) && (j == Y_DIM - 1)) {
    // for nodes along the last row of j on the j-k plane,
    // we expect 2 messages:
    // 1 from itself;
    // 1 from the i dimension;
    if (num_msg[i][j][k] < 2) {
      // not ready yet, so just find the max
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
    } else {
      // done. Forwards max data to node i,j-1,k after making last comparison
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
      ReductionMsg *msg = new ReductionMsg;
      msg->max = max[i][j][k];
      BgSendPacket(i,j-1,k,ANYTHREAD,REDUCE_HANDLER_ID,LARGE_WORK, 
				sizeof(ReductionMsg), (char *)msg);
    }
  } else if (i != X_DIM - 1) {
    // for nodes anywhere the last row of i,
    // we expect 2 messages:
    // 1 from itself;
    // 1 from the i dimension;
    if (num_msg[i][j][k] < 2) {
      // not ready yet, so just find the max
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
    } else {
      // done. Forwards max data to node i-1,j,k after making last comparison
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
      ReductionMsg *msg = new ReductionMsg;
      msg->max = max[i][j][k];
      BgSendPacket(i-1,j,k,ANYTHREAD,REDUCE_HANDLER_ID,LARGE_WORK, 
				sizeof(ReductionMsg), (char *)msg);
    }
  } else if (i == X_DIM - 1) {
    // last row of i, we expect 1 message:
    // 1 from itself;
    if (num_msg[i][j][k] < 1) {
      // not ready yet, so just find the max
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
    } else {
      // done. Forwards max data to node i-1,j,k after making last comparison
      if (max[i][j][k] < external_max) {
	max[i][j][k] = external_max;
      }
      ReductionMsg *msg = new ReductionMsg;
      msg->max = max[i][j][k];
      BgSendPacket(i-1,j,k,-1,REDUCE_HANDLER_ID,LARGE_WORK, 
				sizeof(ReductionMsg), (char *)msg);
    }
  }
}

void computeMax(char *info) {
  int A[A_SIZE][A_SIZE];
  int i, j;
  int max = 0;

  int x,y,z; // test variables
  BgGetXYZ(&x,&y,&z);

  // Initialize
  for (i=0;i<A_SIZE;i++) {
    for (j=0;j<A_SIZE;j++) {
      A[i][j] = i*j;
    }
  }

//  CmiPrintf("Finished Initializing %d %d %d!\n",  x , y , z);

  // Find Max
  for (i=0;i<A_SIZE;i++) {
    for (j=0;j<A_SIZE;j++) {
      if (max < A[i][j]) {
	max = A[i][j];
      }
    }
  }

  // prepare to reduce
  ReductionMsg *msg = new ReductionMsg;
  msg->max = max;
  BgSendLocalPacket(ANYTHREAD, REDUCE_HANDLER_ID, LARGE_WORK, 
				sizeof(ReductionMsg), (char *)msg);

//  CmiPrintf("Sent reduce message to myself with max value %d\n", max);
}


\end{verbatim}

