Using FramerD From C Ken Haase |
--- | ---

Quick ref: [Function name morphology] [Lisp pointers] [Exception Handling] [Unicode Representation] [Input from strings] [Output to strings] [Lisp Types] (esp. immediates, strings, composites) [Evaluator Types] [Evaluator Functions] [Implementing Modules] [Directory Structure] [Source Files]

The kernel of FramerD is written in ANSI C and one way of using FramerD is by writing C or C++ code and using the FramerD libraries in either static or dynamic form. This provisional document describes programming with these libraries and also describes the structure of the FramerD source tree.

# Programming with FramerD

In order to program using FramerD from C, you need to know what compile flags to use and where to find headers and include files. The script `fdxs` provides this information in the form of command line strings which can be passed directly to compilers and linkers. For these purposes, there are three forms of the `fdxs` commands:

`fdxs cflags` | returns the command line options to use for compiling programs which use the FramerD libraries. This includes options specifying where the FramerD include files live as well as other flags such as (on some platforms) `-fshort-enums`.
`fdxs libs` | returns the arguments for linking against the FramerD libraries, including the `-L` argument indicating where the libraries live and individual `-llib` arguments for the various FramerD libraries themselves.
`fdxs modules` | returns the directory where FramerD modules should be installed.

The fdxs command can be used to write portable makefiles by using the backquoted shell escape to execute an `fdxs` and include its results in the compilation commands. For instance, here is the makefile for the `xframerd` module:

    CFLAGS = `fdxs cflags` `libwww-config --cflags` $(XCFLAGS)

    all:  w3fd.so

    tidy:
    	rm -f w3fd.o *~
    clean: tidy
    	rm -f w3fd.so

    w3fd.so: w3fd.o makefile
    	ld -shared -o $@ w3fd.o `libwww-config --libs` `fdxs libs`

    install: w3fd.so
    	cp -f w3fd.so `fdxs modules`

which uses ``fdxs cflags`` to get the C compiler command line, ``fdxs libs`` to get the linker command line arguments, and ``fdxs modules`` to get the modules directory.

### Function name morphology

There are a few heuristics for determining the semantics of functions and macros based on their names. First, nearly all library functions and macros start with the prefix `fd_` or `FD_`; if the preprocessor variable `FD_SOURCE` is defined to be `1` before the FramerD include files are referenced, some aliases without the `fd_` prefix will be defined.

Function names starting with `_fd_` are internal library functions typically used by preprocessor macros, and should not be directly called by user code. Most functions in the library should be threadsafe, but some functions have variants with the suffix `_nolock` indicating that they are not threadsafe.

Case is used to distinguish "well-behaved type-checking" functions from riskier functions. Lower case functions (or preprocessor macros which can be called as functions) typically do type checking (when handed LISP pointers) and obey standard conventions for interaction with the garbage collector (if they hold onto an argument they will incref it and if the return a lisp pointer it will need to be decrefed unless further returned. Some functions have upper case names (and are often implemented as macros or inline functions) and make no such guarantees. Other uppercase identifiers act as semi-syntactic elements, rather than as functions, and these are always implemented by C preprocessor macros (though they may have helper functions, generally starting with `_fd_`. These are mostly used in the exception handling system.

The FramerD libraries are divided into five core collections of functions:

dtype
  provides utility functions, the core LISP data types used by FramerD, and binary DType I/O allowing these data types transmission and storage;
framerd
  provides the basic object database, the associative indices used to do fast searches for objects, the core of the FDScript evaluator (including threading and the module and security system), and basic procedures for accessing frames and making simple inferences;
scheme
  provides much of the R4RS scheme functionality, including the full arithmetic system, string and character operations, and I/O primitives; currently, these functions are lumped into the fdscript library from the developer's point of view;
fdscript
  provides FDScript's special extensions, including more advanced FramerD access and maintenance functions, operating system access, the `PRINTOUT` family of functions, imports from Common LISP (including sequence functions, `DOLIST`, `DOTIMES`, etc.
fdtext
  provides FDScript's text analysis tools, including a powerful and extensible textual pattern matcher, HTML and XML generation and parsing (including XML namespace handling), MIME and RFC-822 parsing, MD5 hashing, the Porter stemmer, and a simple proper name extractor.

All externally available functions are prefixed by a preprocessor macro providing information to the linker. In the core FramerD code, these macros are:

> `DTYPES_EXPORT`, `FRAMERD_EXPORT`, `FDSCRIPT_EXPORT`, `FDTEXT_EXPORT`, `EXPORTED`

# LISP Pointers: `fd_lisp`

This is a typed pointer. In the standard implementation, this is a struct of type `struct FD_LISP_PTR` consisting of a type tag (of enumerated type `fd_lisp_type` stored in the `type` field) and a data pointer (of union type fd_lisp_data stored in the `->data` field). The type of a pointer can be directly tested with the macro `FD_PRIM_TYPEP(x,type)` where `x` is the lisp pointer and `type` is one of the values of `fd_lisp_type`. The data portion can be extracted with the function `FD_PTR_DATA(x,field)` where `field` indicates a particular kind of data.

Garbage Collection FramerD uses a reference counting garbage collector, the important external functions/macros are:

- `fd_incref(x)` increases the "reference count" of the object referred to by the lisp pointer x. Note that if x is a fixnum, immediate, symbol, or object reference, this is simply the identity.
- `fd_decref(x)` reduces the "reference count" of the object referred to by the lisp pointer x and --- if the reference count reaches zero --- frees the storage it is using for other purposes.
- `fd_copy_lisp(x)` returns a copy of the object referred to be the lisp pointer x. Note that if x is a fixnum, immediate, symbol, or object reference, this is simply the identity.

The contract between `fd_copy_lisp` and `fd_decref` is that if an object y is created by `fd_copy_lisp(x)`, it will be unaffected by a subsequent `fd_decref(x)`. Nearly all normal FramerD library functions and preprocessor macros which take lisp pointers as arguments will incref those pointers if they will hang onto them; likewise nearly all functions returning LISP pointers will implicitly incref their argument. Functions and macros which do not do this will have either uppercase names or begin with an underscore "_", indicating that the function is an internal function.

Comparing Objects Lisp objects can also be compared with one another:

- `FD_LISP_EQUAL(obj1,obj2)`
  Returns 1 if obj1 and obj2 are equal, zero otherwise.
- `FD_LISP_EQ(obj1,obj2)`
  Returns 1 if obj1 and obj2 have identical types and literal data fields; this is the same as `LISP_EQUAL` for fixnums, flonums, immediate values, symbols, and OIDs.
- `fd_memberp(object,list)`
  Returns 1 if an object is in a list.
- fd_choice_containsp(object,coice)
  Returns 1 if an object is an element of the choice choice.

Functions and macros for generating or accessing particular types of objects are shown below.

## Printing LISP objects

fd_object_to_string(object)
  Returns a string containing a printed representation for object.
fd_parse_lisp_from_stream(stream)
  Returns an object generated from a printed representation read from stream (of type `FILE *`).
fd_parse_lisp_from_string
  Returns an object generated from a printed representation stored in string
fd_print_dtype(object,file stream)
  Emits a printed representation of an object onto file stream.
fd_print_dtype_to_stdout(object)
  Writes a printed representation of object to the standard output, followed by a carriage return.
fd_print_dtype_to_string(object,string stream)
  Writes a printed representation of object to a string data structure.
fd_read_dtype_from_file(filestring)
fd_str2lisp(string)
  Converts a string containing a printed representation into a lisp object.
fd_arg2lisp(string)
  Returns an object based on the printed representation in string. It is designed for parsing command line arguments as lisp objects and favors returning LISP strings. In particular, only arguments beginning with a colon are interpreted as symbols.

## Reading and Writing DTypes

`fd_dread_dtype`(data_buffer)
  Takes a data buffer and returns the object specified by the DType representation within it.
`fd_dtype_compare`(object,stream)
  Returns 1 if the stream specifies a DTYPE representation for OBJECT.
`fd_dwrite_dtype`(object,data_buffer)
  Writes a dtype representation for an object into a data buffer.
`fd_fignore_dtype`(stream)
  Skips over one dtype representation on stream.
`fd_fread_dtype`(filestream)
  Reads a dtype representation from a binary filestream, returning the corresponding object.
`fd_fwrite_dtype`(object,filestream)
  Writes a DType representation of object to a filestream.
`fd_write_dtype_to_file`(object,filestring)
  Writes a dtype representation of an object to the specified file.

# Exception Handling

The type `fd_exception` is used by FramerD's exception handling system, this is a pointer to a string (`char *`) describing an error. The functions for dealing with errors can be found in `src/os/except.c`. The key functions for signalling errors are

- `fd_raise_exception(exception)`
- `fd_raise_detailed_exception(exception,details string)`
- `fd_raise_lisp_exception(exception,details string,lisp irritant)`

Catching exceptions is done with macros that make C look a little like LISP. The basic schemas are:

    {
      WITH_HANDLING {
        ... do some stuff ... }
      ON_EXCEPTION {
        ... do stuff when an exception gets raised ... }
      END_HANDLING
    }

for catching exceptions and

    {
      UNWIND_PROTECT {
        ... do some stuff ... }
      ON_UNWIND {
        ... do this even if an exception is raised ... }
      END_UNWIND
    }

to ensure that cleanup code gets run despite exceptions. In either case, you can use the function `fd_theException()` to get the exception which was signalled, `fd_exception_details()` to get the details string for the exception, and `fd_exception_object()` to get whatever LISP object was associated with the acception (or `FD_VOID` if there wasn't any).

The exception handliing or protection code can call `fd_reraise()` to pass on the current exception, or `fd_clear_exception()` to clear the current exception. The protection code will automatically reraise the exception unless it has been cleared.

### Some Standard Exceptions

`fd_Out_Of_Memory`
  An exception signalled when a system call to malloc() has failed
`fd_Parse_Error`
  An exception signalled when the parser for LISP expressions encounters an unexpected token
`fd_Unexpected_EOD`
  An exception signalled when a data buffer ends unexpectedly.
`fd_Unexpected_EOF`
  An exception signalled when a data stream ends unexpectedly.
`fd_Unknown_Record_Type`
  An exception signalled when the system tries to operate on an unknown record type.
`fd_Unknown_Type_Code`
  An exception signalled when when an unknown DTYPE data code is read. It generally occurs when mis-aligned data has been written out, which none of the built-in procedures should ever do.

# Unicode

FramerD uses UTF-8 as its internal coding for Unicode strings. UTF-8 encodes ASCII characters using a single byte and encodes other Unicode code points using a series of non-ASCII (decimal value > 255) characters. Most strings in FramerD are UTF-8 strings and are typed as either `fd_utf8_string` or (more commonly) a pointer to an array of type `fd_u8char` (which is just an unsigned character).

Translations from external character sets to UTF-8 is managed by FramerD using structures of type `struct FD_TEXT_ENCODING`. The function `fd_get_encoding(char *)` returns a particular named encoding, e.g. `fd_get_encoding("LATIN-1")` returns the LATIN-1 encoding for Western European characters.

The function `fd_make_utf8(char *start,char *end,struct FD_TEXT_ENCODING *e)` returns a newly malloc'd UTF-8 string based on the characters between start and end based on the encoding e. If end is `NULL`, the entire string start is used; if e is NULL, the default encoding is used.

The function `fd_convert_utf8(u8char *start,int len,struct FD_TEXT_ENCODING *e,int *sizep)` does the opposite conversion, taking a UTF-8 string and producing a newly malloc'd string of characters using encoding e. If a given character cannot be represented in this encoding, an error is signalled. The length of the resulting string is stored in sizep if it is non-NULL (this is especially important if the generated string contains NULs ('\0') which will mislead the C libraries `strlen`.

The function `fd_localize_utf8(u8char *start,struct FD_TEXT_ENCODING *e)` does the opposite conversion, taking a UTF-8 string and producing a newly malloc'd string of characters using encoding e. If a given character cannot be represented in this encoding, an error is signalled.

FramerD provides a structure called an `FD_XFILE` (which deals with alien characters) for doing I/O using character sets. The function `fd_get_xfile(FILE *f,struct FD_TEXT_ENCODING *e)` returns an FD_XFILE structure (malloc'ing it if neccessary) which reads its input or writes its output using f together with the encoding e. This structure should be freed with the function `fd_free_xfile`.

The functions for directly using XFILEs are:

- `fd_xputc(int c,struct FD_XFILE *xf)` writes a representation of the unicode code point c to xf
. This signals an error if the encoding of xf cannot represent c.
- `fd_xgetc(struct FD_XFILE *xf)` returns the `int` code point for the character from xf
, based on the encoding assigned to `xf`.
- `fd_xputc_escaped(int c,struct FD_XFILE *xf)` writes a representation of the unicode code point c to xf
. If the encoding for xf cannot represent c, an escape sequence of the form `\uXXXX` or `\UXXXXXXXX` is used to encode the character, where each `XXXX` or `XXXXXXXX` is the hexadecimal code for the character.
- `fd_xgetc_escaped(int c,struct FD_XFILE *xf)` returns the `int` code point for the character from xf
, based on the encoding assigned to `xf`. This will parse escapes generated by `fd_xputc_encoded`.

The FramerD internationalization library also provides a layer for using stdio `FILE *` pointers directly, without the intervening `XFILE`s. The function `fd_set_file_encoding(FILE *f,char *name)` associates the encoding named name with the file f. After this association, the following functions provide automatic encoding and decoding:

- `fd_fgetc(FILE *f)` returns a character
- `fd_fputc(int c,FILE *f)` writes c to f
- `fd_fputs_raw(u8char *s,int len,FILE *f)` writes len bytes of the UTF-8 encoded string s to f, signalling an error if the encoding for f cannot represent a character.
- `fd_fputs_escaped(u8char *s,int len,FILE *f)` writes len bytes of the UTF-8 encoded string s to f, expressing any unrepresentable characters by `\uXXXX` or `\UXXXXXXXX` escape sequences.

The encoding assigned by `fd_set_file_encoding` can be retrieved by `fd_get_file_encoding`.

In addition to file specific and XFILE encodings, there are three special global encodings: the console encoding is used for output to the console (stdin/stdout/stderr); the system encoding is used to convert strings before and after system or library calls, for instance filenames; finally, the default encoding is used whenever neither of these encodings are relevant and no other encoding is explicity provided.

The function `fd_set_default_encoding(struct FD_TEXT_ENCODING *e)` sets the default encoding and the function `fd_get_default_encoding()` retrieves it. Similarly, for the system encoding, the function `fd_set_system_encoding(struct FD_TEXT_ENCODING *e)` sets the default encoding and the function `fd_get_system_encoding()` retrieves it. The console encoding can be modified by applying the functions above

FramerD normally comes with built-in encodings for ASCII, LATIN-[1-9], ISO-8859/[1-15], UTF-8, UCS-2, and KOI-8. In addition, the function `fd_load_encoding(char *name,char *file)` loads a named encoding from an external file. It is able to interpret both the "charset mapping files" provided by the Unicode Consortium (ftp://ftp.unicode.org/Public/MAPPINGS/) and the native character set encoding used by Linux and some other operating systems. Under Linux, these mappings usually live in the `/usr/share/i18n/charmaps/` directory.

The environment variable `ENCODINGS_PATH` can refer to a number of directories where FramerD automatically searches for encodings in this format. In this search, encoding names are canonicalized by removal of of punctuation characters and upper-casing of alphabetic characters.

The following program translates the standard input to the standard output, doing character translations automatically:

    int main()
    {
      struct FD_TEXT_ENCODING *in, *out; int c;
      in=fd_get_encoding(argv[1]); out=fd_get_encoding(argv[2]);
      fd_set_file_encoding(stdin,in); fd_set_file_encoding(stdout,out);
      while ((c=fd_fgetc(stdin)) >= 0) fd_fputc(c,stdout);
      exit(0);
    }

## Composed Characters

Some unicode characters are combining characters which work by modifying preceding or succeeding characters with (for instance) diacritical or other marks. This leads to a potential for ambiguity because some modified characters have two representations: a direct representation as a single unicode character and a combined representation consisting of the unmodified character and a subsequent modifier. The function `fd_decompose_char(unsigned int ch)` returns a UTF-8 string encoding ch as a sequence of base character plus modifiers; the function `fd_recompose_char(fd_u8char *s)` returns the single unicode character (if it exists) represented by the character and modifiers in the UTF-8 string s It returns `-1` if the string does not describe a combined character with a single character equivalent.

The standard I/O functions (such as fd_fgetc) do not normally interpret combined characters, but return base and combining characters separately. The functions above are designed to be used when programs need to interpret such combined characters.

# String Streams

String streams are for generating strings programmatically and are used extensively by FDScript and especially its text processing utilities. String streams grow automatically as output is received. A string stream is initialized by the macro `FD_INITIALIZE_STRING_STREAM(ssp,isize)` where ssp is a pointer to a string stream and isize is the initial number of bytes allocated to its buffer. This buffer is kept in the `->ptr` field of the `FD_STRING_STREAM` (of type `fd_u8char *`). The current capacity of this buffer is an int in the limit and the current actual size is in the size field. These sizes are in bytes and the number of represented unicode characters may differ, since some Unicode characters take more than one byte of UTF-8 encoding.

As indicated by the type of the buffer, string streams use UTF-8 to represent Unicode output. The functions for doing output to string streams are:

- `fd_sputc(fd_string_stream,int c)` writes the unicode character represented by c;
- `fd_sputs(fd_string_stream,u8char *s)` writes the UTF-8 string s to the stream;
- `fd_sputn(fd_string_stream,u8char *s,int n)` writes the UTF-8 string between s and s+n to the stream.

For example, the following procedure returns a string based on a character range:

    u8char *enumerate_char_range(int bot,int top)
    {
      /* Return a string of characters from bot to top */
      struct FD_STRING_STREAM s; int i=bot;
      FD_INITIALIZE_STRING_STREAM(&s,(top-bot)*2);
      while (i < top) {fd_sputc(&s,i); i++;}
      return s.ptr;
    }

The macro `FD_INITIALIZE_FIXED_STRING_STREAM` initializes a string stream of fixed size; output beyond the specified size is a no-op. It takes three arguments: a pointer to the string stream to initialize, the size of the buffer, and a pointer to the buffer (which is typically on the stack). We could use this to implement a limited version of the procedure above:

    u8char *enumerate_char_range(int bot,int top)
    {
      /* Return a string of characters from bot to top */
      struct FD_STRING_STREAM s; int i=bot; char buf[128];
      if ((top-bot)*2 > 128)
         fd_raise_exception("Range might be too big");
      else {
        FD_INITIALIZE_FIXED_STRING_STREAM(&s,128,buf);
        while (i < top) {fd_sputc(&s,i); i++;}
        /* We have to copy it because it is in this stack frame. */
        return fd_strdup(s.ptr);}
    }

The function `fd_printf(fd_string_stream,char *fmt,...)` is a version of the `printf` which takes a string stream as argument and also supports several additional format directives:

- "%q" prints a lisp pointer
- "%Q" prints a lisp pointer using the pretty printer
- "%t" prints the current time
- "%lt" prints the current time and date
- "%m" prints a `message' string which should be processed by GNU `gettext` for translation before printing.

The function `fd_fprintf` takes a `FILE *` pointer as its first argument. It will also obey any encoding assigned to the file pointer by `fd_set_file_encoding`. The function `fd_xprintf(char *fmt,...)` handles exceptional i/o which goes to either the standard error output or a dynamically declared error stream.

The functions `fd_notify(char *fmt,...)` and `fd_warn(char *fmt,...)` are both specialized versions of `fd_printf` which use globally declared handlers to display output to the user. These functions might, for instance, pop up a dialog box rather than doing output to the console. Both functions produce strings which are then passed to handlers. The handler for `fd_notify` is defined by `fd_set_notify_handler`: its argument is a function which takes a string and returns void. The handler for `fd_warn` is defined by `fd_set_warn_handler`: its argument is also a function which takes a string and returns void. The function `fd_disable_notifications()` turns `fd_notify` into a no-op. There is no way to turn off warnings. In the case of both functions, the default action is to do output to the standard error stream (stderr).

## Reading from Strings

UTF-8 strings can also be read from using a set of functions which parallel the FILE and XFILE reading functions. In this case, an input string is simply a pointer to a pointer to a UTF-8 string (i.e. `fd_utf8 **`; as characters are read, the pointer is advanced. The function `fd_sgetc(fd_u8char **ss)` reads a character from such a stream. The function `fd_parse_lisp_from_string(fd_u8char **ss)` parses a LISP object from the representation in the string.

# Lisp Types

There are four basic kinds of lisp types in FramerD:

- immediate types (fixnums, flonums, characters, and constants) take up no additional storage (beyond the pointer itself, which is typically stack-allocated
- consed types (strings or symbols, for instance) point to an internal FramerD structures of some sort; some of these (such as symbols) may be interned, so that LISP equality and pointer identity are equivalent. And in some of these cases, such as symbols, the consed data is permanent and cannot be reclaimed. Note that object identifiers, OIDs, are a special case, depending on how FramerD is compiled; when FramerD is compiled with consed oids (currently the default), OIDs are a native type and point to a lisp strcuture; when FramerD is compiled with lightweight oids, OIDs are an immediate type which may not have a corresponding structure.
- composite types contain other pointers as elements
- foreign types contain reference counted C pointers to other objects

The immediate types are:

- fixnums (tagged with `fixnum_type`)
  `FD_FIXNUMP(x)` is `1` if x is a fixnum;
  `FD_LISPFIX(int)` converts a C int into a tagged lisp pointer; `FD_FIXLISP(lisp_int)` extracts the integer value from a lisp pointer; `fixlisp(lisp_int)` does the same but signals an error if lisp_int is not tagged as a fixnum.
- constants (tagged with `immediate_type` and including true, false, the empty list, the empty choice, the `void' value, and the end of file marker);
  constants are returned by `FD_TRUE`, `FD_FALSE`, `FD_EMPTY_LIST`, `FD_EMPTY_CHOICE`, `FD_VOID`, and `FD_EOF_OBJECT`;
  values are tested by `FD_TRUEP`, `FD_FALSEP`, `FD_EMPTY_LISTP`, `FD_EMPTYP`, `FD_VOIDP`, and `FD_EOF_OBJECTP`;
- characters (tagged with `character_type`)
  `FD_CHARACTERP(x)` is `1` if x is a character object;
  `FD_CODE_CHAR(int)` returns a character object representing the code point int; `FD_CHAR_CODE(x)` returns the integer code point for x; `fd_char_code(x)` does the same but signals an error if v is not a character object `fd_make_character(int x)` returns a LISP character object for the integer code point for x
- OIDs (when OIDS are lightweight) are unsigned ints whose first 8 bits indicate a pseudo pool and whose remaining 24 bits indicate an offset into that pool. These pseudo pools may or may not align with actual pools of OIDs used by the object database.
  `FD_OIDP(x)` is `1` if x is an OID;
  `fd_oid_current_value(x)` returns the value associated with the OID x if it has been set or loaded already; `fd_oid_value(x)` returns the value associated with the OID x, retrieving it if neccessary and incref'ing the result;
  `FD_OID_ADDR_HIGH(x)` returns the top 32-bits of the unique 64-bit ID of x; `FD_OID_ADDR_LOW(x)` returns the lower 32-bits of the unique 64-bit ID of x;

The primary consed types are:

- strings are pointers to structures of type `struct FD_STRING` which store length information
  `FD_STRINGP(x)` is `1` if x is a string; `FD_ASCII_STRINGP(x)` will tell you if x is an ASCII string
  `FD_STRING_DATA(x)` returns the UTF-8 string data of a string; `FD_STRING_LENGTH(x)` returns the number of bytes used in this string; `fd_strdata(x)` and `fd_strlen(x)` do type checking
  `fd_init_string(fd_u8char *s,int len)` creates a new string object, using the string s as data; s is not copied and it's UTF-8 validity is not normally checked; `fd_copy_string(fd_u8char *s)` returns a string object based on the UTF-8 string s, however it also does not validate the UTF-8 string; `fd_make_string(char *s)` returns a string object based on the string s using the default character encoding to convert s to UTF-8; `fd_make_substring(fd_u8char *start,fd_u8char *end)` returns a string object based on the substring starting at start and ending at end>.
- symbols are pointers to structures of type `struct FD_SYMBOL` and have associated names and global values; symbols are interned, so all symbols with the same name will actually be the same `struct FD_SYMBOL` structure and `fd_decref(x)` on a symbol does nothing
  `FD_SYMBOLP(x)` is `1` if x is a symbol;
  `FD_SYMBOL_NAME(x)` returns the name of the symbol x, a UTF-8 string (`u8char *`) which should not be modified. `FD_SYMBOL_VALUE(x)` returns the lisp value globally associated with the symbol x;
  `fd_make_symbol(string)` returns the symbol with the name string (a UTF-8 string which will be copied). This will always return the same pointer with equal arguments and will cons a new `struct FD_SYMBOL` if neccessary;
  `fd_probe_symbol(string)` is like `fd_make_symbol` but returns `FD_EMPTY_CHOICE` if the symbol has not already been returned.
- OIDs (when OIDS are consed) are pointers to structures of type `struct FD_OID` which combine a unique 64-bit identifier with a lisp value; they are also interned, like symbols;
  `FD_OIDP(x)` is `1` if x is an OID;
  `FD_OID_VALUE(x)` returns the value associated with the OID x if it has been set or loaded; `FD_OID_ADDR_HIGH(x)` returns the top 32-bits of the unique 64-bit ID of x; `FD_OID_ADDR_LOW(x)` returns the lower 32-bits of the unique 64-bit ID of x;
  there are a lot of functions and macros for dealing with OIDs; the are described here (to be written).
- flonums (tagged with `flonum_type`) are pointers to C `double`s
  `FD_FLONUMP(x)` is `1` if x is a flonum object;
  `FD_LISPFLOAT(fl)` returns a floating point object with value fl; `FD_FLOATLISP(x)` returns the floating point value of x and `fd_lisp2float(x)` does the same conversion together with a type check on x
- C records are C pointers tagged with a lisp object for type information; they are implemented by the `struct FD_RECORD` structure;
  `FD_RECORDP(x)` is `1` if x is a C record;
  `FD_RECORD_TAG(x)` returns the LISP object used as a type tag for x; `FD_RECORD_DATA(x)` returns a `void *` pointer describing the data for x;
  `fd_record_tag(x)` returns the LISP object used as a type tag for x, signalling an error if x is not a record; `fd_record_data(x,type)` returns a `void *` pointer describing the data for x, but will signal an error if x is not a record with type tag type;
  `fd_make_record(ltag,ptr)` returns a C record with the tag tag and containing the pointer ptr (which should be of type void *

The primary composite types are:

- conses are pairs of two other pointers;
  `FD_PAIRP(x)` is `1` if x is a pair;
  `FD_MAKE_PAIR(x,y)` makes a cons out of x and y (without `incref`ing them); `fd_make_pair(x,y)` does the same but increfs its arguments; `FD_MAKE_LIST1(elt)` returns a list with one element elt (not incref'd);
  `FD_CAR(x)` will get x back, `FD_CDR(x)` will be y back. `fd_car(x)` and `fd_cdr(x)` will do the same with type checking and an incref; `fd_car_noref`, and `fd_cdr_noref` do the type checking but not the incref
  `fd_list_length(x)` returns the number of elements in x
- vectors are composed of a fixed number of other pointers with a specific order;
  `FD_VECTORP(x)` is `1` if x is a vector;
  `FD_VECTOR_LENGTH(x)` returns the number of elements in x; `FD_VECTOR_REF(x,i)` returns the ith element; `FD_VECTOR_SET(v,i,x)` sets the ith element of the vector v to x;
  `fd_vector_length(x)` and `fd_vector_ref(x,i)` do type and bounds checking.
- choices represent unordered sets of objects (pointers) and are used by the evaluator to implement non-deterministic evaluation and by the frame language to implement many-valued slots; technically, a regular lisp object is a singular choice, and most of the macros and functions below will do the right thing with regular objects. When a choice is non-singular and non-empty, it is represented by a pointer to a `struct FD_CHOICE` structure
  `FD_CHOICEP(x)` is `1` if x is a non-singular choice; `FD_CHOICE_SIZE(x)` returns the number of elements in x and returns `1` if x is not a choice and `0` if x is the empty choice;
  the macro `FD_ADD_TO_CHOICE(loc,x)` adds a pointer to x to the choice stored in loc; for example, `FD_ADD_TO_CHOICE(answers,LISPFIX(3));` adds the fixnum `3` to the choice stored in the C variable `answers`. Note that answers should have been initialized to some lisp pointer (typically the empty choice);
  the macro pattern `FD_DO_CHOICES(x,choice)bodyEND_FD_DO_CHOICES;` iterates over the elements of a choice, processing for the next C statement for each element of choice and binding x to the element. Note that `FD_DO_CHOICES` declares some internal variables, so it must occur before any non-declarations in a C block.
  choices are internally divided into proper choices (which have no repeated elements) and sorted choices (which are sorted to optimize certain operations). The macros `FD_PROPER_CHOICEP(x)` and `FD_SORTED_CHOICEP(x)` check the manifest type of x (they don't actually check the data). The functions `fd_return_proper_choice(x)` and `fd_return_sorted_choice(x)` generate proper or sorted choices from x (they also may free their argument x and return a different pointer).
- slotmaps represent sets of assocations between keys (typically symbols or OIDs) and lisp values; they are kept in a `struct FD_SLOTMAP` structure
  `FD_SLOTMAPP(x)` is `1` if x is a slotmap; `FD_SLOTMAP_PTR(x)` returns a pointer to the slotmap structure used for x
  `SLOTMAP_SIZE(x)` returns the number of associations in x; `FD_SLOTMAP_KEY(sm,i)` returns the key for the ith association; `FD_SLOTMAP_VALUE(sm,i)` returns its value;
  `fd_make_slotmap(size)` makes a new slotmap with initial space for size associations; these assocations can be modified by several functions:
  - `fd_slotmap_set(sm,key,val)` sets the value associated with key in the slotmap sm to val;
  - `fd_slotmap_add(sm,key,val)` adds val to the values associated with key in the slotmap sm;
  - `fd_slotmap_get(sm,key,default)` returns the values associated with key in the slotmap sm, or returns default if there are no associations in sm;
  - `fd_slotmap_test(sm,key,val)` returns `1` if val is associated with key in sm;

  there are other slotmap functions as well.

- lisp records are lisp objects tagged with type information represented by a LISP tag (which could be a symbol, a string, an OID, anything);
  `FD_LRECORDP(x)` is `1` if x is a lisp record; `LRECORD_TAG(x)` returns the lisp object being used to tag x; `FD_LRECORD_TAG(x)` returns the lisp type tag of x.
  `FD_LRECORD_DATA(x)` returns the lisp data of x.
  `FD_LRECORD_TYPEP(x,tag)` is `1` if the tag on x is `EQ?` to tag.
  `fd_lrecord_tag(x)` returns the lisp object describing the type of x and signals an error if x is not an LRECORD.
  `fd_lrecord_data(x,type)` returns the lisp data for x but signals an error if x is not an LRECORD with type tag type.
  `fd_make_record(ltag,ldata)` returns a lisp record whose tag is tag and whose data is data.

Foreign types are reference counted C pointers. The pointer of a foreign type can be extracted by the macro `FD_CPTR_DATA(x)`, and an object of foreign type is created by the function `fd_make_cptr(type,ptr)` where type is one of the legal `fd_lisp_type` values.

This table lists the foreign types, the kind of pointer which `CPTR_DATA`, and the chief file where that data structure is defined and used will return:

Type | FD_CPTR_DATA is | file(s)
--- | --- | ---
dtype_server_type | struct FD_SERVER * | src/eval/eval.c
pool_type | struct FD_POOL * | src/odb/odb.c
index_type | struct FD_INDEX * | src/index/index.c
hashtable_type | struct FD_HASHTABLE * | src/cons/hash.c src/fdscript/hashprims.c
hashset_type | struct FD_HASHSET *
input_file_type | FILE * | src/fdscript/ioprims.c
output_file_type | FILE *
io_file_type | FILE *
input_string_type | struct FD_STRING_ISTREAM *
output_string_type | struct FD_STRING_STREAM *
mutex_type | pthread_mutex_t * | src/eval/threads.c
mystery_type | struct FD_MYSTERY *
segmented_stream_type | struct FD_MYSTERY * | src/text/text.c
record_stream_type | struct FD_MYSTERY *
tx_closure_type | struct TX_CLOSURE * | src/text/match.c

In addition, a number of special `foreign' types are used by the evaluator; this table summarizes them. For the time being, if you have any questions, "Use the Source, Luke".

type | FD_PTR_DATA is | Description
tail_call_type | struct FD_VECTOR * | Represents a partially evaluated expression, used to allow constant-space tail recursion
cproc_type | struct FD_CPROC * | a C primitive accessible from FDScript
sproc_type | struct FD_SPROC * | a Scheme procedure definition, closed in a particular environment
ssproc_type | struct FD_SSPROC * | a "synchronized" scheme procedure, which is guaranteed to only be running in any one thread at any one time
continuation_type | a FD_CPTR to a char * | indicates a dynamic return point somewhere up the stack; the continuation passing mechanism uses FramerD's native exception handling to implement its upward continuations
gproc_type | struct FD_GPROC * | a procedure which returns the empty choice when called recursively on the same arguments; used for certain special sorts of programming
rproc_type | struct FD_RPROC * | a procedure which uses a remote server to get its value
multiple_value_type | struct FD_VECTOR * | used to represent multiple values returned from a procedure; note that these multiple values are different from choices
delay_type | struct FD_PAIR * | a delayed value; the `CAR` is the delay's thunk, the `CDR` is the value returned by it and the thunk is freed after itis called.
env_type | struct FD_LISPENV * | a pointer to an environment structure; when this structure has a `->module` field, it is a top level module
quoted_choice_type | struct FD_CHOICE * | This is used to temporarily suspend non-determinism when passing an argument; a quoted choice is automatically unquoted whenever it is passed to a primitive or bound to a variable

# FDScript Evaluator Functions

`fd_add_cproc`("fdscript-name",arity,c_procedure)
  Declares a new primitive in the scripting language with the name "fdscript-name", implemented by c_procedure, and taking arity. Evaluating the expression `(fdscript-name arg1 arg2 ...)` calls the c_procedure on the result of evaluating each of the arguments.
`fd_add_cform`("fdscript-name",c_procedure)
  Declares a new special form in the scripting language with the name "fdscript-name", implemented by c_procedure. Evaluating the expression `(fdscript-name arg1 arg2 ...)` calls the c_procedure on the entire expression without evaluating any arguments.
`fd_add_nary_cproc`("fdscript-name",c_procedure)
  Declares a new primitive in the scripting language with the name "fdscript-name", implemented by c_procedure, and taking any number of arguments. Evaluating the expression `(fdscript-name arg1 arg2 ...)` calls the c_procedure on a list whose elements are the results of evaluating each argi. If the argument evaluations return non-deterministic values, the values are passed as elements of the list, rather than calling c_procedure repeatedly on each combination.
`fd_eval`(expr)
  Evaluates an expression in this process.
`fd_eval_loop`(input,output)
  Enters a read-eval-print loop going to a specified input and output stream.
`fd_get_arg`(expr,arg_number,default)
  Given an expression, returns the arg_numberth element if it exists and the default otherwise. If the default value is `FD_VOID`, an error is signalled if the argument is not provided.
`fd_get_body`(expr,offset)
  Given an expression, returns the sublist starting after the offsetth element.
`fd_interactive_loop`(input file stream,output file stream)
  Iteratively reads expressions from the input file stream (a FILE *), evaluates them, and outputs them to the output file stream.
`fd_load_init`(filestring)
  Loads and evaluates a file of FDScript expressions.

# Writing a Module

To write a C module to link FDScript with some existing library, take a look at the automatic code generator in `extras/c-bindings/cbind.fdx`. Otherwise, this section offers a few pointers and you can look at `samples/c-modules/` for example code.

There are three basic kinds of primitives: cprocs, lexprs, and special forms. A cproc takes some fixed number of arguments, all of which are evaluated; a lexpr takes an arbitrary number of arguments, all of which are evaluted; a special form takes an expression and an environment and does whatever it pleases with it.

Special forms are declared by the function `fd_add_special_form(env,"name",cproc)` which defines a special form bound to the symbol name in env which will be handled by the C function cproc. The C function cproc should have the signature `lisp cproc(lisp expr,fd_lispenv env)`.

CPROCs are declared by the function `fd_add_cproc("name",n_args,cproc)` which defines a CPROC that takes n_args arguments (surprise) and is handled by the C function cproc. The C function cproc should have the signature `lisp cproc(lisp arg1,lisp arg2,...)`. The compiler will not warn you if the number of arguments doesn't match. This binds the symbol name to this primitive in the environment env.

LEXPRs come in two flavors and are declared by the function `fd_add_lexpr("name",lexpr_type,cprc)` where lexpr_type is either `NORMAL_LEXPR_ARGS` or `SPECIAL_LEXPR_ARGS`. If the type is `NORMAL_LEXPR_ARGS`, the evaluator will automatically enumerate any choices passed as arguments (so cproc may be called multiple times). Otherwise, cproc will only be called once, but the argument list it receives may contain non-singular choices. This binds the symbol name to this primitive in the environment env.

A C module normally contains a series of `static` functions which implement the primitives and an initialization function which calls the function `fd_make_module()` to make an empty module (which has C type `fd_lispenv`). It then makes a series of calls to `fd_add_cproc` and its siblings to populate this environment with siblings. Finally, it calls the function `fd_register_module(name,env)` on the module to bind name in the global environment to the module where a call to `USE-MODULE!` can find it. (Alternatively, it can call the function `fd_register_restricted_module` to bind the module in FDScript's `restricted environment'.)

If the env argument to the `fd_add_` functions is `NULL`, the primitive is defined in the global environment. This will allow for faster accesses, but access cannot be restricted and there is a risk of name collisions.

# The Directories of FramerD

The top level of the FramerD directory tree (which you get when you unpack a source tarball or do a CVS checkout) is organized into multiple subdirectories show in this table.

Top Level FramerD directory structure
src | contains the source for the C libraries and executables; the structure of these sources is described below.
include/ | contains C header files used by the FramerD source files
include/framerd/ | contains C header files used by external FramerD libraries as well as the FramerD source files
bin/ | is the directory into which executable files are compiled
lib/ | is the directory into which static and shared libraries are compiled
etc/ | contains the Emacs Lisp (ELISP) interface to FramerD, the FDScript installation scripts, and various shell scripts for server management.
extras/ | contains subdirectories for various FDScript extension libraries written in C. Each has its own makefile and produces static and dynamic libraries
extras/c-bindings | The file `cbind.fdx` implements a package which automatically generates interfaces from FDScript to C libraries. The individual `.fdx' files in this directory implement various bindings for common libraries.
docs/ | contains documentation for FramerD in HTML, together with a handful of `.png` format images.
tests/ | contains a FramerD test suites, including Aubrey Jaffer's R4RS test suite, custom tests for FramerD itself, and regression tests to catch the return of past bugs.
tests/gabriel | contains Scheme versions of the Gabriel benchmarks for evaluating FDScript performance
scripts/ | contains FDScript programs to be installed as command line utilities. These utilities include pool and index maintenance utilities as well as command line tools for browsing FramerD databases.
samples/ | contains subdirectories of various collections of sample code for servers, shell commands, web scripting, and email parsing

In addition, the top level directory contains configuration and make files (configure, configure.in, and makefile.in) as well as text files describing how to compile FramerD (`BUILD`), how to install it (`INSTALL`), news about changes (`README`), and a copy of the GNU General Public License (`LICENSE`).

Conceptually, the FramerD sources can be divided into six modules, summarized in this table.

Structure of FramerD source modules
directories | libraries | Description | header files
--- | --- | --- | ---
src/os | libdtypes | a system module provides for OS-independent internationalization, environment access, and file system operations | include/framerd/os.h include/framerd/except.h include/framerd/strstream.h
src/cons | a data module provides the basic LISP data structures, printed and binary object i/o, and some derived structures, including hashtables and slotmaps | include/framerd/cons.h include/framerd/lisp.h
src/odb src/index src/eval | libframerd | the FramerD core provides management for object (OID) databases, persistent associative indices, and a minimal evaluator for a subset of Scheme | include/framerd/odb.h include/framerd/index.h include/framerd/eval.h
src/scheme | libfdscript | the Scheme module provides most of the R5RS scheme functions, including rational, complex, and arbitrary precision arithmetic | include/framerd/bignum.h include/framerd/bignmint.h
src/fdscript | the FDScript module provides further advancec functions, including operating system access, bindings for internal utilities in the system, data and FramerD modules, formatted output, generic sequences, and console interaction |
src/text | libfdtext | the text module provides pattern matching and parsing functions for natural and artifical languages, especially intended for implementing transducers between existing programs and databases and FramerD. | include/framerd/fdtext.h
src/exe src/misc |  | the executables themsleves which are linked against the libraries above; the directory `src/misc` contains code used with the server and CGI executables. A description of the standard executables can be found here |

The "standard executables" in FramerD are compiled from files in `src/exe`, linked against the above libraries. The server and web gateway executables are also linked together with the files `src/misc/server.c` and `src/misc/cgiparse.c`. These standard executables are summarized in this table.

Standard FramerD executables
print-dtype | prints and ASCII representation of a DTYPE stored in a file
make-dtype | makes a DTYPE in a file given an ASCII representation from the command line
dtcall | does a remote procedure call using the DType protocol.
pool-get | prints the value of an OID in a pool
index-get | prints the values associated with a key in an index
analyze-index | analyzes and index file for hashing properties and statistics
fdscript | runs an FDScript interpreter, either interactively or on a command file; this is used to implement command-line access to FramerD databases
fdcgi | runs an FDScript interpreter, intended to use with a web server's common gateway interface to provide WWW-hosted interaction with FramerD databases
fdserver | starts a FramerD server on a specified port, which may provide access to pools or indices in local files

# The Files of FramerD

src/cons/except.c | implements basic exception handling based on the C runtime `setjmp` and `longjmp`
src/cons/data.c | implements data types such as lists and integers and the facilities for implementing new datatypes in C
src/cons/hash.c | interned datatypes (symbols and object identifiers (OIDs)) which rely on them
src/cons/xdata.c | datatypes, including vectors, slotmaps, bignums, unicode strings, etc
src/cons/io.c src/cons/dtio.c src/cons/txio.c | implements the binary DTYPE format for transmitting data across the network and storing it persistently in disk files; it is included twice in the file `io.c` to implement I/O to and form buffers in memory and I/O using the C stdio library. `src/cons/io.c` provides a wrapper for the binary DTYPE functions as well as functions for producing and parsing printed representations of objects. It also includes the functions for getting miscellaneous enviromental information (including accessing the WIN32 registry)
src/os/network.c | implements basic generic network access functions across operating systems
src/os/fdprintf.c | with a printf syntax extended for the presentation of lisp objects. Output can be either directed to files or to the string streams used throughout FramerD. This file also implements exceptional i/o facilities for managing reports of unexpected events.
src/os/i18n.c | provides an OS independent layer for dealing with Unicode strings and character set conversions
src/os/envfns.c | provides an OS independent layer for dealing with environment queries, including time and system functions
src/os/filefns.c | provides an OS independent layer for dealing with file access and information
src/odb/odb.c | implements the object database which maps OIDs to their values, whether stored in local files or remote servers; it also manages the locking and unlocking of OIDs and the organization of OIDs into disjoint pools.
src/odb/pools.c | implements the management of the pools into which the object address space is divided.
src/odb/file-pool.c | implements access to subsets of the object database stored in disk files, called file pools
src/odb/network-pool.c | implements access to subsets of the object database stored on remote servers, called network pools
src/odb/frames.c | implements frames which are OIDs whose values are slotmaps implementing an attribute value structure with the possibility for complex inferences over some of their values.
src/index/index.c | implements access to persistent indices mapping Scheme objects to each other and especially matching key objects to OIDs
src/index/file-index.c | implements access to on-disk hash tables as indices.
src/index/network-index.c | implements access remote DTYPE servers used as indices.
src/index/search.c | combines the frames implemented in the object database with the indexing facility to do automatic or semi-automatic frame indexing
src/eval/eval.c | contains the evaluator kernel and a small set of evaluator related functions.
src/eval/ndeval.c | contains handlers for dealing with non-determinism in evaluation and application
src/eval/mini.c | contains basic core functions (no binding constructs) to be used by FramerD inference methods
src/eval/lambda.c | implements environment structures and lambdas to support user definitions.
src/eval/sandbox.c | implements basic symbol level access in order to support limited-capability interpreters for more secure FDScript environments
src/eval/threads.c | contains support for multi-threaded programming, including mutexes and spawning new threads.
src/eval/load.c | implements procedures for loading disk files containing FDScript code.
src/scheme/arith.c | contains arithmetic primitives
src/scheme/bignum.c | contains the code for arbitrary precision integers, inherited from MIT Scheme
src/scheme/strings.c | contains special string manipulation primitives
src/scheme/pairs.c | contains special pair/list manipulation primitives
src/scheme/characters.c | contains special character access and manipulation primitives
src/scheme/reflect.c | contains forms for environment manipulation
src/scheme/special.c | contains Scheme and Common Lisp special forms
src/scheme/cxr.c | contains list access primitives of the form "C[AD]+R"
src/scheme/ioprims.c | contains primitives for binary and character I/0, including the R4RS I/0 primitives
src/fdscript/prims.c | contains a miscellaneous primitives for FDScript
src/fdscript/printout.c | implements the `PRINTOUT` facility for producing formatted output.
src/fdscript/seq.c | contains special sequence functions inherited from Common Lisp
src/fdscript/osprims.c | contains primitives for operating system access
src/fdscript/framerd.c | contains primitives for accessing FramerD objects, frames, pools, and indices
src/fdscript/maint.c | contains primitives for maintaining FramerD pools and indices
src/fdscript/hashprims.c | contains primitives for using hashtables and hashsets