%% One sided communication in Converse
%% Author: Nilesh Choudhury

\chapter{\converse{} One Sided Communication Interface}

This chapter deals with one sided communication support in converse.
It is imperative to provide a one-sided communication interface to take
advantage of the hardware RDMA facilities provided by a lot of NIC cards.
Drivers for these hardware provide or promise to soon provide capabilities
to use this feature. 

\converse{} provides an implementation which wraps the functionality provided
by different hardware and presents them as a uniform interface to the
programmer. For machines which do not have a one-sided hardware at their
disposal, these operations are emulated through converse messages.

\converse{} provides the following types of operations to support one-sided
communication.


\section{Registering / Unregistering Memory for RDMA}
The interface provides functions to register(pin) and unregister(unpin) memory
on the NIC hardware. The emulated version of these operations do not do anything.

\function{int CmiRegisterMemory(void *addr, unsigned int size);}
\index{CmiRegister}
\desc{This function takes an allocated memory at starting address {\it addr} 
of length {\it size} and registers it with the hardware NIC, 
thus making this memory DMAable. This is also called pinning memory on the
NIC hardware, making remote DMA operations on this memory possible.
This directly calls the hardware driver function for registering the memory
region and is usually an expensive operation, so should be used sparingly.}

\function{int CmiUnRegisterMemory(void *addr, unsigned int size);}
\index{CmiUnregister}
\desc{This function unregisters the memory at starting address {\it addr} 
of length {\it size}, making it no longer DMAable. This operation corresponds
to unpinning memory from the NIC hardware. This is also an expensive operation
and should be sparingly used.}

For certain machine layers which support a DMA, we support the function
\function{void *CmiDMAAlloc(int size);}
\index{CmiDMAAlloc}
\desc{This operation allocates a memory region of length {\it size} from the 
DMAable region on the NIC hardware. The memory region returned is pinned to the
NIC hardware. This is an alternative to {\it CmiRegisterMemory} and is implemented
only for hardwares that support this.}


\section{RDMA operations (Get / Put)}
This section presents functions that provide the actual RDMA operations.
For hardware architectures that support these operations these functions 
provide a standard interface to the operations, while for NIC architectures that
do not support RDMA operations, we provide an emulated implementation.
There are three types of NIC architectures based on how much support they
provide for RDMA operations:
\begin{itemize}
\item Hardware support for both {\it Get} and {\it Put} operations.
\item Hardware support for one of the two operations, mostly for {\it Put}. For these
the other RDMA operation is emulated by using the operation that is implemented
in hardware and extra messages.
\item No hardware support for any RDMA operation. For these, both the RDMA operations
are emulated through messages.
\end{itemize}

There are two different sets of RDMA operations
\begin{itemize}
\item The first set of RDMA operations return an opaque handle to the programmer, 
which can only be used to verify if the operation is complete. This suits
AMPI better and closely follows the idea of separating communication
from synchronization. So, the user program needs to keep track of 
synchronization.
\item The second set of RDMA operations do not return anything, instead they
provide a callback when the operation completes. This suits nicely the charm++
framework of sending asynchronous messages. The handler(callback) will be 
automatically invoked when the operation completes.
\end{itemize}

{\bf For machine layer developer:} Internally, every machine layer is free
to create a suitable data structure for this purpose. This is the reason this
has been kept opaque from the programmer.

\function{void *CmiPut(unsinged int sourceId, unsigned int targetId, void *Saddr, void *Taadr, unsigned int size);}
\index{CmiPut}
\desc{This function is pretty self explanatory. It puts the memory location 
at {\it Saddr} on the machine specified by {\it sourceId} to {\it Taddr} on
the machine specified by {\it targetId}. The memory region being RDMA'ed is
of length {\it size} bytes.}

\function{void *CmiGet(unsinged int sourceId, unsigned int targetId, void *Saddr, void *Taadr, unsigned int size);}
\index{CmiGet}
\desc{Similar to {\it CmiPut} except the direction of the data transfer
is opposite; from target to source.}

\function{void CmiPutCb(unsigned int sourceId, unsigned int targetId, void *Saddr, void *Taddr, unsigned int size, CmiRdmaCallbackFn fn, void *param);}
\index{CmiGet}
\desc{Similar to {\it CmiPut} except a callback is called when the operation 
completes.}

\function{void CmiGetCb(unsigned int sourceId, unsigned int targetId, void *Saddr, void *Taddr, unsigned int size, CmiRdmaCallbackFn fn, void *param);}
\index{CmiGet}
\desc{Similar to {\it CmiGet} except a callback is called when the operation 
completes.}


\section{Completion of RDMA operation}
This section presents functions that are used to check for completion
of an RDMA operation. The one sided communication operations are
asynchronous, thus there needs to be a mechanism to verify for completion.
One mechanism is for the programmer to check for completion. The other 
mechanism is through callback functions registered during the RDMA operations.

\function{int CmiWaitTest(void *obj);}
\index{CmiWaitTest}
\desc{This function takes this RDMA handle and verifies if the operation
corresponding to this handle has completed.}

A typical usage of this function would be in AMPI when there is a call to
AMPIWait. The implementation should call the CmiWaitTest for all 
pending RDMA operations in that window.
