\chapter{Parallel Arrays of Threads}

This module is CPath: \converse{} Parallel Array of Threads.  It makes it
simple to create arrays of threads, where the threads are distributed
across the processors.  It provides simple operations like sending a
message to a thread, as well as group operations like multicasting to
a row of threads, or reducing over an array of threads.

\section{Creating Arrays of Threads}

This module defines a data type CPath, also known as an ``array
descriptor''.  Arrays are created by the function CPathMakeArray,
and individual threads are created using CPathMakeThread:

\function{void CPathMakeArray(CPath *path, int threadfn, int mapfn, ...)}
\desc{This function initiates the creation of an array of threads.  It
fills in the array descriptor \param{*path}.  Each thread in the 
array starts executing the function represented by \param{threadfn}.
The function \param{mapfn} represents a mapping function, controlling
the layout of the array.  This parameter must be followed by the
dimensions of the array, and then a zero.}

\function{void CPathMakeThread(CPath *path, int startfn, int pe)}
\desc{This function makes a zero-dimensional array of threads, in
other words, just one thread.}

\section{Mapping Functions for Arrays of Threads}

One of the parameters to CPathMakeArray is a ``mapping function'',
which maps array elements to processors.  Mapping functions must be
registered.  The integer index returned by the registration process is
the number which is passed to CPathMakeArray.  Mapping functions
receive the array descriptor as a parameter, and may use it to determine
the dimensions of the array.

\function{unsigned int MapFn(CPath *path, int *indices)}
\desc{This is a prototype map function, all mapping functions must have this
parameter list.  It accepts an array descriptor and a set of indices.
It returns the processor number of the specified element.}

\function{int CPathRegisterMapper(void *mapfn)}
\desc{Accepts a pointer to a mapping function, and returns an integer
index for the function.  This number can be used as a parameter to
CPathMakeArray.}

\function{int CPathArrayDimensions(CPath *path)}
\desc{Returns the number of dimensions in the specified array.}

\function{int CPathArrayDimension(CPath *path, int n)}
\desc{Returns the nth dimension of the specified array.}

\section{Thread Functions for Arrays of Threads}

Thread functions (the functions that the threads execute) must have
the following prototype, and must be registered using the following
registration function.  The integer index returned by the registration
process is the number which is passed to CPathMakeArray.

\function{void ThreadFn(CPath *self, int *indices)}
\desc{This is a prototype thread function.  All thread-functions must
have these parameters.  When an array of threads is created, each
thread starts executing the specified thread function.  The function
receives a pointer to a copy of the array's descriptor, and the array
element's indices.}

\function{int CPathRegisterThreadFn(void *mapfn)}
\desc{Accepts a pointer to a thread function, and returns an integer
index for the function.  This number can be used as a parameter to
CPathMakeArray.}

\section{Sending Messages to Threads}

Threads may send messages to each other using CPathSend, which takes a
complicated set of paramters.  The parameters are most easily
described by a context-free grammar:

\function{void CPathSend(dest-clause, tag-clause, data-clause, end-clause)}
Where:

\begin{alltt}
    dest-clause :== CPATH_DEST ',' pathptr ',' index ',' index ',' ...
    tag-clause  :== CPATH_TAG ',' tag
    tag-clause  :== CPATH_TAGS ',' tag ',' tag ',' ... ',' 0
    tag-clause  :== CPATH_TAGVEC ',' numtags ',' tagvector
    data-clause :== CPATH_BYTES ',' numbytes ',' bufptr
    end-clause  :== CPATH_END
\end{alltt}

The symbols {\tt CPATH\_DEST}, {\tt CPATH\_TAG}, {\tt CPATH\_TAGS},
{\tt CPATH\_TAGVEC}, {\tt CPATH\_BYTES}, {\tt CPATH\_END}, and the
comma are terminal symbols.  The symbols descriptor, index, tag,
numtags, tagvector, numbytes, and bufptr all represent C expressions.

The dest-clause specifies which array and which indices the message is
to go to.  One must provide a pointer to an array descriptor and a set
of indices.  Any index may be either a normal index, or the wildcard
{\tt CPATH\_ALL}.  Using the wildcard causes a multicast.  The
tag-clause provides several notations, all of which specify an array
of one or more integer tags to be sent with the message.  These tags
can be used at the receiving end for pattern matching.  The
data-clause specifies the data to go in the message, as a sequence of
bytes.  The end-clause represents the end of the parameter list.

Messages sent with CPathSend can be received using CPathRecv, analyzed
using CPathMsgDecodeBytes, and finally discarded with CPathMsgFree:

\function{void *CPathRecv(tag-clause, end-clause)}
\desc{The tag-clause and end-clause match the grammar for CPathSend.
The function will wait until a message with the same tags shows up (it
waits using the thread-blocking primitives, see \converse{} threads).  If
any position in the CPathRecv tag-vector is {\tt CPATH\_WILD}, then
that one position is ignored.  CPathRecv returns an ``opaque CPath
message''.  The message contains the data somewhere inside it.  The
data can be located using CPathMsgDecodeBytes, below.  The opaque
CPath message can be freed using CPathMsgFree below.}

\function{void CPathMsgDecodeBytes(void *msg, int *len, void *bytes)}
\desc{Given an opaque CPath message (as sent by CPathSend and returned
by CPathRecv), this function will locate the data inside it.  The
parameter \param{*len} is filled in with the data length, and
\param{*bytes} is filled in with a pointer to the data bytes.  Bear in
mind that once you free the opaque CPath message, this pointer is no
longer valid.}

\function{void CPathMsgFree(void *msg)}
\desc{Frees an opaque CPath message.}

\section{Performing Reductions over Array Elements}

An set of threads may participate in a reduction.  All the threads
wishing to participate must call CPathReduce.  The parameters to
CPathReduce are most easily described by a context-free grammar:

\function{void CPathReduce(over-clause, tag-clause, red-clause, data-clause, dest-clause, end-clause)}
Where:

\begin{alltt}
    over-clause :== CPATH_OVER ',' pathptr ',' index ',' index ',' ...
    dest-clause :== CPATH_DEST ',' pathptr ',' index ',' index ',' ...
    tag-clause  :== CPATH_TAG ',' tag
    tag-clause  :== CPATH_TAGS ',' tag ',' tag ',' ... ',' 0
    tag-clause  :== CPATH_TAGVEC ',' numtags ',' tagvector
    data-clause :== CPATH_BYTES ',' vecsize ',' eltsize ',' data
    red-clause  :== CPATH_REDUCER ',' redfn
    end-clause  :== CPATH_END
\end{alltt}

The over-clause specifies the set of threads participating in the
reduction.  One or more of the indices should be {\tt CPATH\_ALL}, the
wildcard value.  All array elements matching the pattern are
participating in the reduction.  All participants must supply the same
over-clause.  The tags-clause specifies a vector of integer tags. All
participants must supply the same tags.  The reducer represents the
function used to combine data pairwise.  All participants must supply
the same reducer.  The data-clause specifies the input-data, which is
an array of arbitrary-sized values.  All participants must agree on
the vecsize and eltsize.  The dest-clause specifies the recipient of
the reduced data (which may contain {\tt CPATH\_ALL} again).  The data
is sent to the recipient.  The results can be received with CPathRecv
using the same tags specified in the CPathReduce.  The results may be
analyzed with CPathMsgDecodeReduction, and freed with CPathMsgFree.

\function{void CPathMsgDecodeReduction(void *msg,int *vecsize,int *eltsize,void *bytes)}
\desc{This function accepts an opaque CPath message which was created by
a reduction.  It locates the data within the message, and determines
the vecsize and eltsize.}

The function that combines elements pairwise must match this
prototype, and be registered with the following registration function.
It is the number returned by the registration function which must be
passed to CPathReduce:

\function{void ReduceFn(int vecsize, void *data1, void *data2)}
\desc{The reduce function accepts two equally-sized arrays of input
data.  It combines the two arrays pairwise, storing the results in
array 1.}

\function{int CPathRegisterReducer(void *fn)}
\desc{Accepts a pointer to a reduction function, and returns an integer
index for the function.  This number can be used as a parameter to
CPathReduce.}

