
\chapter{About \pastix{}}

\section{Introduction}

\pastix{} is a complete, parallelized, and multi-threaded library for
the resolution of huge linear systems of equations.
It is developped by \bacchus{} team from INRIA\footnote{Institut
  National de Recherche en informatique et 
  Automatique.}.\\

Depending on charateristics of the matrix \texttt{A}, the solution of $A x =
b$ can be computed in several ways~:
\begin{itemize}
  \item if the matrix \texttt{A} is symmetric positive-definite, we
    can use the Cholesky \nomenclature[Ala]{Cholesky
      decomposition~:}{In mathematics, the Cholesky decomposition is
      named after Andr\'e-Louis Cholesky, who found that a symmetric
      positive-definite matrix can be decomposed into a lower
      triangular matrix and the transpose of the lower triangular
      matrix. The lower triangular matrix is the Cholesky triangle of
      the original, positive-definite matrix. Cholesky's result has
      since been extended to matrices with complex
      entries. (en.wikipedia.org)} 
    ($A = LL^t$, $L$ 
    lower triangular matrix, $L^t$ its transpose) or
    Cholesky-Crout ($A = LDL^t$, $D$ diagonal matrix) with, or without,
    numerical  pivoting,
  \item if the matrix \texttt{A} is not symmetric, the $LU$
    decomposition ($U$ upper triangular matrix) with static pivoting
    will be used.
\end{itemize}

\pastix{} steps~:
\begin{itemize}
\item Reordering the unknowns in order to reduce the fill-in induced
  by the decomposition,
\item Symbolic factorization, to predict the structure of the
  factorized matrix,
\item Distributing matrix blocks among the processors,
\item Decompostion of the matrix \texttt{A},
\item Solving the system (up-down),
\item Refining the solution because we use static pivoting, which can
    reduce the precision of the solution.
\end{itemize}

\section{Ordering}
\label{ordering}

\pastix{} computes the reordering by calling the \scotch{}
package. (\metis{} can also be used.) \\
During direct decomposition, new nonzero terms, called ``fill-in'', appear in the
decomposed matrix.\\
In a graph $G(V,E)$, whose vertices are the unknowns, and
whose edges are defined by~: $(i,j) \in
E(G) \leftrightarrow a_{ij} \neq 0$.\\
One fill-in edge will be created betwen two unknowns \texttt{i} and
\texttt{j} if there is a path from \texttt{i} to \texttt{j}
passing only through vertices with number lower than \texttt{min(i,j)}.\\
To save memory and computation time during decomposition, this fill-in
has to be minimized. To manage this several algorithms are used.\\
First, we use a nested dissection
(\hyperref[fig:dissection]{fig. \ref{fig:dissection},
p. \pageref{fig:dissection}})~: we search in the graph for a
separator\nomenclature[Ala]{Separator~:}{A set of vertices that connects two disjoint parts
  of the graph.} \texttt{S} of minimal size that cuts the graph into two
parts of about the same size.\\
Then the separator nodes are indexed with the highest numbers
available, so that no fill-in terms will appear during decomposition
between the unknowns of the two separated parts of the graph. The
algorithm is then repeated on each subgraph.
This step also produces a dependency graph used to distribute
computations onto the processors.\\

\begin{figure}
  \includegraphics[width= \textwidth]{dissection}
  %\begin{tikzpicture}
  %  \foreach \y/\ytext in {1.125/30,2.275/25,3.625/20,4.875/15,6.125/10,7.375/5,8.625/1}{
  %    \node at (0.5,\y) {\ytext};
  %  }
  %  \foreach \y in {1,1.25,...,8.5}{
  %    \draw (1,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,8.25}{
  %    \draw (1.25,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,8}{
  %    \draw (1.5,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,7.75}{
  %    \draw (1.75,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,7.5}{
  %    \draw (2,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,7.25}{
  %    \draw (2.25,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,7}{
  %    \draw (2.5,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,6.75}{
  %    \draw (2.75,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,6.5}{
  %    \draw (3,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,6.25}{
  %    \draw (3.25,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,6}{
  %    \draw (3.5,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,5.75}{
  %    \draw (3.75,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,5.5}{
  %    \draw (4,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,5.25}{
  %    \draw (4.25,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,5}{
  %    \draw (4.5,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,4.75}{
  %    \draw (4.75,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,4.5}{
  %    \draw (5,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,4.25}{
  %    \draw (5.25,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,4}{
  %    \draw (5.5,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,3.75}{
  %    \draw (5.75,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,3.5}{
  %    \draw (6,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,3.25}{
  %    \draw (6.25,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,3}{
  %    \draw (6.5,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,2.75}{
  %    \draw (6.75,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,2.5}{
  %    \draw (7,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,2.25}{
  %    \draw (7.25,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,2}{
  %    \draw (7.5,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,1.75}{
  %    \draw (7.75,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25,...,1.5}{
  %    \draw (8,\y) rectangle ++(.25,.25);
  %  }
  %  \foreach \y in {1,1.25}{
  %    \draw (8.25,\y) rectangle ++(.25,.25);
  %  }
  %  \draw (8.5,1) rectangle ++(.25,.25);
  %  \node (1) at (-1,7) {1};
  %  \node (2) at (-1,6) {2};
  %  \node (11) at (-1,5) {11};
  %  \node (5) at (-1,4) {5};
  %  \node (9) at (-1,3) {9};
  %  \node (7) at (-1,2) {7};
  %  \node (4) at (-2,7) {4};
  %  \node (3) at (-2,6) {3};
  %  \node (12) at (-2,5) {12};
  %  \node (6) at (-2,4) {6};
  %  \node (10) at (-2,3) {10};
  %  \node (8) at (-2,2) {8};
  %  \node (30) at (-3,7) {30};
  %  \node (29) at (-3,6) {29};
  %  \node (28) at (-3,5) {28};
  %  \node (27) at (-3,4) {27};
  %  \node (26) at (-3,3) {26};
  %  \node (25) at (-3,2) {25};
  %  \node (20) at (-4,7) {20};
  %  \node (21) at (-4,6) {21};
  %  \node (18) at (-4,5) {18};
  %  \node (23) at (-4,4) {23};
  %  \node (16) at (-4,3) {16};
  %  \node (14) at (-4,2) {14};
  %  \node (19) at (-5,7) {19};
  %  \node (22) at (-5,6) {22};
  %  \node (17) at (-5,5) {17};
  %  \node (24) at (-5,4) {24};
  %  \node (15) at (-5,3) {15};
  %  \node (13) at (-5,2) {13};
  %  \draw (1) -- (4) -- (30) -- (20) -- (19);
  %  \draw (2) -- (3) -- (29) -- (21) -- (22);
  %  \draw (17) -- (18) -- (28) -- (12) -- (11);
  %  \draw (24) -- (23) -- (27) -- (6) -- (5);
  %  \draw (15) -- (16) -- (26) -- (10) -- (9);
  %  \draw (13) -- (14) -- (25) -- (8) -- (7);
  %  \draw (19) -- (22) -- (17) -- (24) -- (15) -- (13);
  %  \draw (20) -- (21) -- (18) -- (23) -- (16) -- (14);
  %  \draw (30) -- (29) -- (28) -- (27) -- (26) -- (25);
  %  \draw (4) -- (3) -- (12) -- (6) -- (10) -- (8);
  %  \draw (1) -- (2) -- (11) -- (5) -- (9) -- (7);
  %  \foreach \p in {(8.5,1),(8.25,1.25),(8,1.5),(7.75,1.75),(7.5,2),(7.25,2.25),(7,2.5),(6.75,2.75),(6.5,3),(6.25,3.25),(6,3.5),(5.75,3.75),(5.5,4),(5.25,4.25),(5,4.5),(4.75,4.75),(4.5,5),(4.25,5.25),(4,5.5),(3.75,5.75),(3.5,6),(3.25,6.25),(3,6.5),(2.75,6.75),(2.5,7),(2.25,7.25),(2,7.5),(1.75,7.75),(1.5,8),(1.25,8.25),(1,8.5),(1.25,8.5),(1.75,8.5),(1.5,8.5)}{
  %    \filldraw[fill=black!80!white] \p rectangle ++(0.25,0.25);
  %  }
  %\end{tikzpicture}
  \caption{Nested dissection : Left, the graph before fill-in; right, the matrix with fill-in.}
  \label{fig:dissection}
\end{figure}

When the subgraph is smaller than a specified threshold, the \texttt{Halo
Approximate Minimum Degree} algorithm 
(\hyperref[fig:halo]{fig.~\ref{fig:halo}, p.~\pageref{fig:halo}}) is used.\\
This algorithm further reduces the number of ``fill-in'' terms even more by
assigning the smallest available numbers to the nodes with lowest
degree. Direct and foreign neighbors (\texttt{halo}) are considered, in order
not to neglect interactions with the whole graph.
\begin{figure}
%\includegraphics[width= \textwidth]{halo}
\begin{center}
\begin{tikzpicture}
\tikzstyle halo=[color=blue!50,very thick,dotted] 
\tikzstyle remplissage=[color=red!50,very thick,dashed] 

\begin{scope}[shape=circle,minimum size=1cm,fill=orange!20!white] 
\tikzstyle{every node}=[draw,fill] 
\node (a) at (0,0) {4}; 
\node (b) at (6,0) {1}; 
\node (c) at (2,0) {2}; 
\node (d) at (4,2) {5}; 
\node (e) at (4,-2) {3};
\end{scope} 
\draw (a) -- (c) -- (d) -- (b) -- (e) --(c);
\draw (d) -- (a);
\draw[halo] (a) -- (0,2) -- (d);
\draw[halo] (d) -- (e);
\draw[remplissage] (a) .. controls +(75:-1.4cm) and +(90:-1.4cm) .. (e); 
\draw (7,2) -- (7.5,2) node[right]{original graph};
\draw[halo] (7,1.5) -- (7.5,1.5) node[right,color=black]{halo};
\draw[remplissage] (7,1) -- (7.5,1) node[right,color=black]{fill-in edge};
\end{tikzpicture}
\end{center}
\caption{Halo Approximate Minimum Degree.}
\label{fig:halo}
\end{figure}

An elimination tree can be constructed by using the following rule~: there is
an edge between \texttt{i} and\texttt{j} if the first nonzero entry of
column \texttt{i} is at row \texttt{j}.\\
This tree represents the dependancies between the computation steps.
The wider and deeper it is, the less the computation are dependent
from each other.\\

The goal of the reordering step is to minimize ``fill-in'' and to
reduce the dependancies between computations.\\

This algorithm also compute a partition of the columns. This partition
results from the fusion of the separators and the sub-graphs reordered
using the \texttt{Halo Approximate Minimum Degree} algorithm.

\section{Symbolic factorization}
\label{symbol}
The symbolic factorization step computes the structure of the
factorized matrix from the original matrix \texttt{A}.\\
The factorization is done by blocks. This computation is cheap, its
complexity depend on the number of extra-diagonal blocs in the
decomposed matrix.

\section{Distribution and scheduling.}
\label{blend}
\pastix uses static ordering by default. Computation is organized, in
advance, for maximum alignment with the parallel architecture being
used.\\
The algorithm is composed of two steps.\\
The first step is the partitioning step. The largest blocks are cut
into smaller pieces to be computed by several processors. \textit{Full
  block parallel computation} parallelism is used. It is
applied on biggest separators of the elimination tree, after
the decomposition.\\
During this step, the elimination tree is traversed from the root to the
leaves and for each block, a set of processor candidates able to handle it is
computed. This idea of ``processor candidate'' is essential for the preservation of 
communication locality; this locality is obtain following the
elimination tree, because only the processors which compute a part of
the subtree of a node would be candidate for this node.

The second step is the distributing phase, each block will be
associated to one of its candidates wich will compute it.\\
To do so, the elimination tree is climbed up from the leaves to the
root and, to distribute every node of the tree on processors, the
computation and communication time is computed for each block. Thus,
each block is allocated to the processor which will finish its
processing first. \\
This step  schedules computation and communication precisely; it needs
an accurate calibration of \texttt{BLAS3}\nomenclature{BLAS~:}{Basic
  Linear Algebra Subprograms, de facto application programming interface
  standard for publishing libraries to perform basic linear algebra
  operations such as vector and matrix multiplication. They were first
  published in 1979, and are used to build larger packages such as
  LAPACK. Heavily used in high-performance computing, highly optimized
  implementations of the BLAS interface have been developed by
  hardware vendors such as by Intel as well as by other authors
  (e.g. ATLAS is a portable self-optimizing BLAS). The LINPACK
  benchmark relies heavily on DGEMM, a BLAS subroutine, for its
  performance. (en.wikipedia.org)} and communications operations on
the destination architecture. The perf module contains information
about this calibration.

Thus, three levels of parallelism are used~:
\begin{itemize}
\item Coarse grained parallelism, induced by independant computation
  between two subtrees of one node. This parallelism is also called
  sparse induced parallelism,
\item Medium grained parallelism, induced by dense blocks
  decomposition. This is a node level parallelism, due to the
  possibility of cutting elimination tree nodes to distribute them
  onto different processors, 
\item thin grained parallelism, obtained by using \texttt{BLAS3}
  operations. This requires that the block size be correctly choosen.
\end{itemize}


\section{Factorization}
\label{factorization}
The numeric solution of the problem computes a parallel
Crout ($LL^t$ or $LDL^t$) or Cholesky ($LU$) decomposition with a
one-dimension column block distribution of the matriX.\\
Two type of algorithm exist, \textit{super-nodal}, coming directly from 
column elimination algorithmn rewritten by blocks, and
\textit{multi-frontal}.\\
Among \textit{super-nodals} methods, the \textit{fan-in} or
\textit{left-looking} method involves getting previous column blocks
modifications to compute next column blocks.\\
The contrary, the \textit{fan-out} or \textit{right-looking} method
involves computing the current column block and modifying next column
blocks.

In \pastix{}, the \textit{fan-in} method, more efficient in parallel,
has been implemented. This method has the advantage of considerably
reducing communication volume.



\section{Solve}
\label{solve}
The solve step uses the up-down method. Since this computation is really cheap
compared to decomposition the decomposition distribution is kept.

\section{Refinement}
\label{refinement}

If the precision of the result of the Solve step is unsufficient, iterative refinement
can be performed.

This section described the several iterative methods implemented in
\pastix{}.

\subsection{GMRES}

The \texttt{GMRES} \nomenclature[Ala]{GMRES~:}{In mathematics, the
  generalized minimal residual method is an iterative method for the
  numerical solution of a system of linear equations. The method
  approximates the solution by the vector in a Krylov subspace with
  minimal residual. The Arnoldi iteration is used to find this
  vector.} algorithm used in \pastix{} is based on work with 
Youssef Saad that can be found in \cite{saad96}.
%% 
%% \begin{algorithm}[H]
%%   \caption{GMRES algorithm}
%%   \label{algo:raff:gmres}
%%   \begin{algorithmic}
%%     \STATE $X = 0, out\_flag = true$
%%     \WHILE{$out\_flag$}
%%     \STATE $r_0 = ||b - AX||$
%%     \IF{$r_0 \leq \epsilon$}
%%     \STATE $exit$
%%     \ENDIF
%%     \STATE $v[0] = \frac{b - AX}{r_0}$
%%     \STATE $rs[0] = r_0$
%%     \STATE $in\_flag = 1, i = -1$
%% 
%%     \WHILE{$in\_flag$}
%%     \STATE $i = i  + 1, i_1 = i + 1$
%%     \STATE $w[i] = solve(A, v[i])$
%%     \STATE $v[i_1] = Aw[i]$
%%     \FOR{$j = 0$ to $i$}
%%     \STATE $h[i][j] = v[i_1].v[j]$
%%     \STATE $v[i_1] = v[i_1] - h[i][j] v[j]$
%%     \ENDFOR
%%     \STATE $h[i][i_1] = ||v[i_1]||$
%%     \IF{$||v[i_1]|| > 10^{-50}$}
%%     \STATE $v[i_1] = v[i_1]/||v[i_1]||$
%%     \ENDIF
%%     \FOR{$j = 1$ to $i$ ($i > 0$)}
%%     \STATE $st = h[i][j-1]$
%%     \STATE $h[i][j-1] = \overline{c[j-1]}.st +
%%     \overline{s[j-1]}.h[i][j]$
%%     \STATE $h[i][j] = -\overline{s[j-1]}.st +
%%     \overline{c[j-1]}.h[i][j]$
%%     \ENDFOR
%%     \STATE $st = MAX (\sqrt{h[i][i]^2 + h[i][i_1]^2}, \epsilon)$
%%     \STATE $c[i] = h[i][i]/st$
%%     \STATE $s[i] = h[i][i_1]/st$
%%     \STATE $rs[i_1] = -s[i].rs[i]$
%%     \STATE $r_0 = |rs[i_1]|$
%%     \IF{$i+1 \geq krylov\_space\_size$ or $iter \geq iter\_max$ or
%%       $\frac{r_0}{||b||} \leq \epsilon$}
%%     \STATE $in\_flag = false$
%%     \ENDIF
%%     \STATE $iter = iter + 1$
%%     \ENDWHILE
%% 
%%     \STATE $rs[i] = s[i]/h[i][i]$
%%     \FOR{$ii = 1$ to $i$}
%%     \STATE $k = i - ii$
%%     \STATE $st = rs[k]$
%%     \FOR{$j = k+1$ to $i$}
%%     \STATE $st = st - h[j][k].rs[j]$
%%     \ENDFOR
%%     \STATE $rs[k]=\frac{st}{kk[k][k]}$
%%     \ENDFOR
%%     \FOR{$j=0$ to $i$}
%%     \STATE $X = X + \frac{w[j]}{rs[j]}$
%%     \ENDFOR
%%     \IF{$\frac{r_0}{||b||} \leq \epsilon$ or $iter \geq itermax$}
%%     \STATE $out\_flag = false$
%%     \ENDIF
%%     \ENDWHILE
%%   \end{algorithmic}
%% \end{algorithm}
%% 
\subsection{Conjuguate gradiant}

The conjugate gradient method is available with $LL^t$ and $LDL^t$
factorizations. 

\subsection{Simple iterative refinement}
 
This method, described in
\hyperref[algo:raff:staticpivoting]{
  algorithm  \ref{algo:raff:staticpivoting} 
  p.\pageref{algo:raff:staticpivoting}},
computes the difference $b_i$ between $b$ and $A x$ and solves the
system $A  x = b_i$. The solution $x_i$ are added to the
initial solution, and the process repeated until the relative error ($Max_k(\frac{|b-A \times
  x|_k}{||A||x| +|b||_k}$) is small enough.

This method is only available with $LU$ factorization.
\begin{algorithm}[H]
  \caption{Static pivoting refinement algorithm}
  \label{algo:raff:staticpivoting}
  \SetKwFunction{solve}{solve}
  \begin{algorithm2e}[H]
    $lerr \leftarrow 0$\;
    $flag \leftarrow TRUE$\;
    \While{$flag$}{
      $r \leftarrow b - A \times x$\;
      $r^{\prime} \leftarrow |A||x| + |b|$\;    
      $err \leftarrow max_{i = 0..n} (\frac{r[i]}{r^{\prime}[i]})$\;
      \If{$last\_err = 0$}{
        $last\_err \leftarrow 3 \times err$\;
      }
      $rberror \leftarrow ||r|| / ||b||$\;
      \If{($iter < iparm[IPARM\_ITERMAX]$ \textbf{and} \\
        \hskip 0.5cm $err > dparm[DPARM\_EPSILON\_REFINEMENT]$ \textbf{and} \\
        \hskip 0.5cm $err \leq \frac{lerr}{2}$)\\}{
        $r^{\prime} \leftarrow x$\;
        $x \leftarrow $ \solve{$A \times x = r$}\;
        $r^{\prime} \leftarrow r^{\prime} + x$\;
        $last\_err \leftarrow err$\;
        $iter \leftarrow iter + 1$\;
      }
      \Else{
        $flag \leftarrow FALSE$\;
      }
    }
  \end{algorithm2e}
\end{algorithm}
%% \begin{algorithm}[H]
%%   \caption{Static pivoting algorithm}
%%   \label{algo:raff:staticpivoting}
%%   \begin{algorithmic}
%%     \STATE $lerr = 0$
%%     \STATE $flag = TRUE$
%%     \WHILE{$flag$}
%%     \STATE $r = b - A \times x$
%%     \STATE $r^{\prime} = |A||x| + |b|$    
%%     \STATE $err = max_{i = 0..n} (\frac{r[i]}{r^{\prime}[i]})$
%%     \IF{$last\_err = 0$} 
%%     \STATE $last\_err = 3 \times err$ 
%%     \ENDIF
%%     \STATE $rberror = ||r|| / ||b||$
%%     \IF{$iter < iparm[IPARM\_ITERMAX]$ \textbf{and} $err >
%%       dparm[DPARM\_EPSILON\_REFINEMENT]$ \textbf{and} $err \leq \frac{lerr}{2}$}
%%     \STATE $r^{\prime} = x$
%%     \STATE $x = solve(A \times x = r)$
%%     \STATE $r^{\prime} = r^{\prime} + x$
%%     \STATE $last\_err = err$
%%     \STATE $iter = iter + 1$
%%     \ELSE
%%     \STATE $flag = FALSE$
%%     \ENDIF
%%     \ENDWHILE
%%   \end{algorithmic}
%% \end{algorithm}

\section{Out-of-core}
\label{section:description:ooc}

An experimental out-of-core version of \pastix{} has been written.

Directions for compiling this version are given in section
\ref{subsection:compilation:ooc}.

For the moment, only multi-threaded version is supported, the hybrid
``MPI+threads'' version is likely to lead to unresolved deadlocks.

The disk inputs and outputs are managed by a dedicated thread.
This thread will prefetch column blocks and communication buffers so
that computing threads will be abble to use them.

As the whole computation as been predicted, the out-of-core thread
will follow the scheduled computation and
prefetch needed column blocks. It will also save them, depending on
their next acces, if the memory limit is reached.

The prefetch algorithm is not dependant of the number of computation
threads, it will follow the computation algorithm as if there was only
one thread.
