
\section{Introduction and Motivation}
The 3D FFT library provides an interface to do paralle FFT computation
in a scalable fashion. 

The parallelization is achieved by splitting the 3D transform into
multiple phases. There are two possibilities for doing the splitting:
One is dividing the data space (over which the fft is to be performed)
into a set of slabs (figure 1). Each slab is essentially a collection
of planes). First, 2D FFTs are done over the planes in the slab. Then
a distributed 'transform' will send the data to destination so that
fft in the third direction is performed. This approach takes two
computation phases and one 'transform' phase. The second way for
splitting is dividing the data into collections of pencils. First, 1D
FFTs are computed over the pencils; then a 'transform' is performed
and 1D FFTs are done over second dimention; again a 'transform' is
performed and FFTs are computed over the last dimension. So this
approach takes three computation phases and two 'transform' phases. In
first approach, the parallelism is limited by the number of
planes.While in second approach, it's limited by the number of
pencils. So the second approach provides finer grained parallelism and
it's possible to perform better when the number of processing units is
larger than the number of planes.

\section{Compilation and Execution}

To install the FFT library, you will need to have charm++ installed in
you system. You can follow the Charm++ manual to do that. Also you
will need to have FFTW (version 2.1.5) installed. FFTW can be
downloaded from http://www.fftw.org.

The FFT library source is at
your-charm-dir/src/libs/ck-libs/fftlib. Before installation of the
library, make sure that the path for FFTW library is consistent with
your FFTW installation. Then cd to your-charm-dir/tmp, and do 'make
fftlib'. To compile a program using the fftlib, pass the '-lfftlib
-L(your-fftwlib-dir) -lfftw' flag to charmc.

\section{Library Initialization and Data Format} 
To initialize the library, user will need to construct a data struct and pass it to the library. 

For plane-based version, the struct is called: NormalFFTinfo . And the
constructor of 'NormalFFTinfo' is defined as:

\begin{alltt}
        NormalFFTinfo(CProxy_NormalSlabArray &srcProxy, CProxy_NormalSlabArray &destProxy,
                      int srcDim[2], int destDim[2], int isSrcArray, complex *dataptr, 
                      int srcPlanesPerSlab=1, int destPlanesPerSlab=1)

        Where: 
        CProxy_NormalSlabArray &srcProxy : proxy for source charm array 
        CProxy_NormalSlabArray &destProxy : proxy for destination charm array 
        int srcDim[2] : FFT plane data dimension at source array (*)
        int destDim[2] : FFT plane data dimension at destination array ( srcDim[1] must equal to destDim[0].) (*) 
        int isSrcArray : whether this array is source (1) or destination (0)
        complex *dataptr : pointer to FFT data 
        int srcPlanesPerSlab : number of planes in each slab at source array, default value is 1 (**) 
        int destPlanesPerSlab : number of planes in each slab at destination array, default value is 1 (**)

          * Data layout : The multi-dimensional FFT data are supposed
            to be stored in a contiguous one-dimensional array in
            row-major order. For example, in source array, data is
            {\it srcPlanesPerSlab} planes, each plane is {\it
            srcDim[0]} rows of size {\it srcDim[1]} numbers. Similar
            rules apply to destination side.


          ** Currently, srcPlanesPerSlab/destPlanesPerSlab has to be
             the same across all array elements.

          *** Total data size can be calculated by:
              srcPlanesPerSlab*srcDim[0]*srcDim[1] at source array, and
              destPlanesPerSlab*destDim[0]*destDim[1] at destination array
 
\end{alltt}

%{\it put a figure here with reference of above parameters.}
%\caption doing a {\it nz*ny*nz} size FFT, if data layout at source side is x then y then z, and at destination is z then x then y,
%srcDim[] = {ny, nx}, destDim={nx, nz}


For pencil-based version, the struct is called: LineFFTinfo.

\begin{alltt} 
        LineFFTinfo(CProxy_NormalLineArray &xProxy, 
                    CProxy_NormalLineArray &yProxy, 
                    CProxy_NormalLineArray &zProxy, 
                    int size[3], int isSrcArray, complex *dataptr, 
                    int srcPencilsPerSlab=1, int destPencilsPerSlab=1) 

        where: 
        CProxy_NormalSlabArray &xProxy : proxy for first charm array 
        CProxy_NormalSlabArray &yProxy : proxy for second charm array 
        CProxy_NormalSlabArray &zProxy : proxy for third charm array 
        int size[3] : FFT plane data dimension (*)
        int isSrcArray : whether this array is source (1) or intermediate (2) or  destination (0)
        complex *dataptr : pointer to FFT data 
        int srcPencilsPerSlab : number of pencils in each slab at source array, default value is 1
        int destPencilsPerSlab : number of pencils in each slab at destination array, default value is 1

          *data layout : pencils in the three arrays are of size {\it
           size[0]/size[1]/size[2]}. And if there is more than one
           pencil per slab, the other dimension is the dimension for
           pencils in the next array.

\end{alltt}

%{\it put a figure here with reference of above parameters.}
%\caption doing a {\it nz*ny*nz} size FFT,


In both cases, data is deposited by passing in a pointer to the data
field, and the pointer will be stored in 'complex *dataptr' in the
struct. Memory allocation and initializtion of data field needs to be
done by user before pointer is passed in. The library doesn't allocate
any memory for data field. Also note that FFT's done internally in the
library are in-place FFTs, which means that data field will be
overwriten with results.

\section{Library Interfaces}
There are two types of interfaces provided by the library: Charm++
based and AMPI based. 

\subsection{Charm++ interface}

The Charm++ interface is the raw interface of the library and slightly
more difficult to use but gives more flexibility. To use the charm++
based library, user has to create their own charm arrays which derive
from predefined arrays in library. By overiding default methods, user
can add in additional functions.


For the plane-based library, there are several relevant member
functions:{\it 'doFFT', 'doIFFT', 'doneFFT'} and {\it
'doneIFFT'}. {\it 'doFFT'} and {\it 'doIFFT'} need to be called to
start the computation. {\it 'doneFFT'} and {\it 'doneIFFT'} are
callback functions, and they need to be inheritated.

The sample codes below should shed more light on this. For complete
sample programs, refer to file under
your-charm-dir/pgms/charm++/fftdemo/.

In the sample code below, we will illustrate how to use the
plane-based library in 4 steps: initializing the data struct;
creating array element; starting the computation and finally ending
the computation.
%(sample code here)

For initializing, a NormalFFTinfo struct will be used. Keep in mind
that data storage needs to be allocated and initalized by the
user. Since in-place FFT will occur, user should also make duplicate
copies of data when needed.
\begin{alltt}
    main::main(CkArgMsg *m)
    \{
         ...
         /* Assume FFT of size  nx*ny*nz */
         int srcDim[] = {ny, nx}, destDim[] = {nx, nz};

         complex *plane =  new complex[nx*ny];
         ... // Initialize FFT data here 

         NormalFFTinfo src_fftinfo(srcArray, destArray, 
                                   srcDim, destDim, true, plane, 1, 1);

         ...
     \}
\end{alltt}

Next step is to create the charm array: 
\begin{alltt}
     main::main(CkArgMsg *m)
     \{
          ...
          /* Assume FFT of size  nx*ny*nz */
          int srcDim[] = {ny, nx}, destDim[] = {nx, nz};

          /* create the source array */
          srcArray = CProxy_SrcArray::ckNew();
          for (z = 0; z < dim; z+=1) \{
              complex *plane =  new complex[nx*ny];
              ... // Initialize FFT data here: data needs to be in x-major order 

              NormalFFTinfo src_fftinfo(srcArray, destArray, 
                                       srcDim, destDim, true, plane, 1, 1);

              // insert one plane object: this contains data of x-y plane at z coordinate	      
              srcArray(z).insert(src_fftinfo);  
           \}

          /* destination array will be created in similar fashion */
          ...
      \}
\end{alltt}

Following we will start the FFT computation by making a call to {\it
'doFFT()'}. {\it 'doFFT(int id1, int id2)'} takes two inputs: {\it
id1} defines the ID number of the source FFT, while {\it id2} defines
the ID number of the destination FFT. There is a similar method called
{\it 'doFFT()'} to be used to invoke inverse FFTs. In this example, 3
FFT's are done simultaneously by invoking a {\it 'doAllFFT()'}
method. And {\it 'doAllFFT()'} is defined as:
\begin{alltt}
    void SrcArray::doAllFFT() \{
        doFFT(0, 0);
        doFFT(1, 1);
        doFFT(2, 2);
    \}
\end{alltt}

The last step is to get data at destination side. For this purpose,
inheritance of method {\it 'doneFFT()'} is defined below. {\it
'doneFFT(int id)'} takes the FFT ID number as input. For inverse FFTs,
relevant member function is {\it 'doneIFFT()'}.
\begin{alltt}
    void destArray::doneFFT(int id) \{
        count ++; 
        if(count==3) \{
            count = 0;
            /* A reduction is induced:  this will call the predefined reduction client when all array elements finish their computation */
            contribute(sizeof(int), &count, CkReduction::sum_int);
       \}
    \}
\end{alltt}

Next we will demonstrate the usage of pencil-based library in similar steps. 
%(sample code here)

First is the initialization of data struct {\it LineFFTinfo}:
\begin{alltt}
     main::main()  
    \{
         ...
         /* Assume FFT of size  nx*ny*nz */
         int size[] = {nx, ny, nz};

         complex *pencil =  new complex[nx];
         ... /* Initialize FFT data here */ 

         LineFFTinfo fftinfo(xlinesProxy, ylinesProxy, zlinesProxy, size, true, pencil);
         ...
     \}
\end{alltt}


Second is the creation of array:
\begin{alltt}
     main::main()    
    \{
         ...
         /* Assume FFT of size  nx*ny*nz */
         int size[] = {nx, ny, nz};

         xlinesProxy = CProxy_myXLines::ckNew();
         for (z = 0; z < sizeZ; z++) {
             for (y = 0; y < sizeY; y+=THICKNESS) {
                 complex *pencil =  new complex[nx];
                 ... /* Initialize FFT data here */

                 LineFFTinfo fftinfo(xlinesProxy, ylinesProxy, zlinesProxy, 
                                     size, true, pencil);
                 xlinesProxy(y, z).insert(fftinfo);
             }
         }
         xlinesProxy.doneInserting();

         /* ylinesProxy /zlinesProxy are created in similar fashion */
         ...
     \}
\end{alltt}

Next is the starting of the computation. A method called doFirstFFT()
needs to be called. {\it doFirstFFT(int id, int direction)} takes two
parameters: {\it id} specifies the ID number of the target FFT, {\it
direction} tells whether FFTs is to be done in forward({\it
direction=1}) or backward({\it direction=0}) direction.
\begin{alltt}
    void myXLines::doAllFFT() \{
        doFirstFFT(0, 1);
        doFirstFFT(1, 1);
        doFirstFFT(2, 1);
    \}
\end{alltt}

Finally, it's the step to finish the FFT at receiver side. In this
case, we call the array of destination {\it myZLines}. Similarly as in
the plane-based version, {\it doneFFT()} is inherited.  {\it
doneFFT(int id, int direction)} takes two inputs, which are explained
the same as in {\it doFirstFFT(int id, int direction)}.
\begin{alltt}
    void myZLines::doneFFT(int id, int direction) \{
        count ++;
        if(count==3) \{
            count = 0;
            contribute(sizeof(int), &count, CkReduction::sum_int);
        \}
    \}
\end{alltt}

\subsection {AMPI Interface}
The MPI-like interface aims at easy migration of MPI program to
use the library. {not available in CVS yet.}

The AMPI interface has five functions:
\begin{itemize}
\item init\_fftlib - initialization of library. This will create
charm++ level data structures, prepare for FFT computation.
\item start\_fft - start the FFT. 
\item wait\_fft - wait for the FFT to finish. 
\item start\_ifft - start the inverse FFT. (similar as start\_fft)
\item wait\_ifft - wait for the inverse FFT to finish. ( similar as wait\_fft)
\end{itemize}

(sample code here)
\begin{alltt}
\end{alltt}

