FramerD Server Guide Ken Haase |
--- | ---

Quick ref: [Simple Servers] [Security Concerns] [More Complicated Servers] [Administering Servers] [Hairy fdserver invocations]

If the DType protocol is the glue of complex FramerD applications, servers are their struts and skeleton. Both FramerD data (pools of OIDs), annotations (indices), general services (natural language parsing, associative searching), and application services (parsing of particular formats, management of shared state) can be and often are modularized into FramerD servers.

In addition to encouraging modular design, the packaging of functions into servers allows computation and data to be distributed according to the diversity of resources and demand. The portability of FramerD's C kernel (from low end PCs and notebooks to multiple-processor production servers) permits processing to be arranged with respect to available processing, memory, and long term storage resources. The accessibility of servers from very lightweight DType clients permits point-of-interaction applications which can leverage very complex remote processing.

For instance, a FramerD application can consist of a 30K java applet on a palmtop accessing a server on a nearby PC connecting to a distant high-throughput function server on the outside of a firewall which synthesizes results from knowledge bases provided by several high-performance database servers on the inside of the firewall. And this may be only one path of connectivity between these servers and services! Other applications or services may connect these components in different ways.

A FramerD server is essentially a remote Scheme evaluator which processes expressions and returns values. This evaluator typically processes a variant of the Scheme language, reduced (in some ways) to enhance security and extended (in other ways) to provide particular functionality. For example, servers do not usually provide direct access to the local file system or to other servers. However, most servers do provide special functions, which can range from managing object persistence to processing natural language to serving as special gateways to other network services.

In distributed FramerD applications, two special kinds of servers are OID servers and index servers:

- OID servers provide access to the persistent object database by offering retrieval and (possibly) modification for a range of OIDs (object identifiers)
- Index servers provide associational indices from complex structured keys to sets of OIDs or other objects.

FramerD servers are nearly always implemented by the executable `fdserver` and configured by use of a server control file described below.

## Simple Servers

The simplest way to start a new server is to use a server control file as the single argument to the program `fdserver`. A server control file typically has a name of the form name`.fdz` and consists of FDScript code which is processed to initialize the server. Unless overriden, the server evaluates the forms in the control file and then starts listening for remote requests at a particular port name. This port name can either be set in the control file (by the function `set-port-id!`) or will be derived from the control file's name. The port name is converted into a port number (what the network really uses) using either a system-wide database of service names (often found in /etc/services under Unix) or by touch-tone encoding (so "brico" corresponds to the port 27425).

The server control file is simply an FDScript file which initializes the pools, indices, and procedures which the server will provide. Applications or components can be bundled with canned ".fdz" files for setting up a local version of the specified data or processing servers.

For example, the Brico knowledge base comes bundled with the file `brico.fdz`:

>     #!/usr/local/bin/fdserver
>     ;;; This is the file brico.fdz

>     (set-port-id! "brico")

>     ;; Just run locally if the file use-localhost exists in
>     ;;  the same directory as this file
>     (when (file-exists? (get-component "use-localhost"))
>       (use-localhost!))

>     (when (bound? ADMIN-SERVER)
>       (use-pool ADMIN-SERVER)
>       (use-index ADMIN-SERVER))

>     ;;; Intialize brico server
>     (define brico-pool (use-pool (get-component "brico.pool")))
>     (define brico-index (use-index (get-component "brico.index")))
>     (define cyc-pool (use-pool (get-component "cyc/cyc.pool")))
>     (define cyc-index (use-index (get-component "cyc/cyc.index")))

>     (serve-pool brico-pool)
>     (serve-index brico-index)

>     (serve-pool cyc-pool)
>     (serve-index cyc-index)

>     (define (find-synsets word)
>       (find-frames words-index 'words word))
>     (define (get-synonyms x)
>       (fget (find-synsets x) 'words))

>     (define (help)
>       "This server provide the BRICO database and special functions
>     to access it.")

>     (module-export! '{get-synonyms help})

which sets up the server to provide access the OID pools `brico.pool` and `cyc.pool` and to their respective indices as well. Once the server is running, clients can access these pools and indices remotely with specifications such as `(use-pool "brico@myhost")`. It will provide remote servers with access to these pools but not (for instance, the pool and index derived from `ADMIN-SERVER` (which might be used by additional procedures to be defined later. This homegrown `brico@myhost` server can be accessed just like the `brico@db.beingmeta.com` server described in the user's guide.

The particular call to `set-port-id!` in `brico.fdz` is redundant in this case, since `fdserver` would figure it out from the name of the configuration file. The server will then listen to this port on all of the addresses currently assigned to the machine. This can be modified by using a portid of the form `portname@host` where the server will only listen to the port `portid` on the addresses assigned to hostname (which can just be a numeric IP address in dotted decimal format, e.g. `18.85.2.138`). For example,

    (SET-PORT-ID! "brico@127.0.0.1")

would set the server up to listen only on the default localhost address.

The example configuration file `brico.fdz` also defines the procedures `find-synsets`, `get-synonyms` and `help`. The server control file implicitly creates an environment (the "server environment") where these procedures are defined. At the end of the configuration file, the call to `module-export!` determines which of the definitions will actually be accessible by remote servers. In this case, the `help` function and the `get-synonyms` function are available and can be called thus:

>     % dtcall brico@myhost get-synonyms "example"
>     "good_example"
>     "model"
>     "example"
>     "exemplar"
>     "exercise"
>     "instance"
>     "specimen"
>     "sample"
>     "quotation"
>     "representative"
>     "illustration"
>     "object_lesson"
>     "lesson"
>     "deterrent_example"
>     "protoplast"
>     "module"
>     "ensample"
>     "paradigm"

## Security Concerns

Running a FramerD server allows clients on the Internet to run programs on the server's host machine, a potentially risky proposition. This risk is reduced in fdserver by taking advantage of FDScript's native security model. The design of FDScript separates out "risky" functions --- including access to the local filesystem, external program execution, establishment of network connections, etc --- into a separate module. The evaluator used by fdserver does not have access to this module but can only access "safe" functions and whatever functions are defined in the server configuration file(s).

The initialization files read by `fdserver` are processed in the full FDScript environment, so the definitions they provide can access restricted functions. This allows an implementor to write "safe" wrappers around system functions that remote users can access. For instance, even though proscription of `fopen` keeps anonymous clients from opening arbitrary files, a user procedure defined in the initialization files can still open a particular file and provide access to it.

The "sandbox" provided by the limited server environment provides substantial protection, but additional levels of protection can be provided by having the server run as a limited user. The "identity" of a server can be set by the `SET-UID!` procedure in the server control file. This only works when the system is originally running as a privleged user (root under unix). In order to avoid inadvertently running a server as root, servers started as root will always change their uid to the string specified by the configuration variable `FDSERVER_USER` (or `"nobody"` it is not defined) and will change their group id to the group specified by the string `FDSERVER_GROUP` (or `"nogroup"` if it is not defined).

## Controlling Client Access

If the initialization files for a server defines a `CLIENT-OK?` procedure in the server environment, this is called on all client addresses before a connection is established. The client address will typically be in standard `.edu' or `.com' form or as dotted decimal addresses if the symbolic hostname cannot be determined. For example, the following `CLIENT-OK?` definition permits access only from within the MIT Media Lab domain:

    (define (client-ok? client-addr)
      (has-suffix "media.mit.edu" client-addr))

In addition, the procedure `(CLIENT-ID)` returns the address of the client being served. This allows servers to customize their processing based on client identity. For instance, the following definition uses two different parser servers depending on the domain of the client being served:

    (define (parse string)
     (if (has-suffix ".media.mit.edu" (client-id))
         (dtcall "parse@inside-parse" parse string)
       (dtcall "parse@outside-parse" parse string)))

## Maintainer Access

FramerD servers can provide access to an unrestricted environment with the `maintainer` procedure. This procedure takes an expression and evaluates it in an unrestricted environment; however, it can only be called by clients on particular remote hosts. The procedure `MAINTAINER-OK?`, which must be defined to use the `maintainer` procedure, validates a maintenance connection, as in:

    (define (maintainer-ok? client-addr)
      (equal? "admin.framerd.org" client-addr))

This definition only lets connections from the host `admin.framerd.org` use `maintainer`. Note that this particular method is subject to IP spoofing attacks. A better approach would be to require that forms to be evaluated in an unrestricted environment be cryptographically signed (volunteers?).

## A more complicated `fdserver`

A more complicated server control file can combines pool/index access together with special functions related to the data it is providing. For example, the following file provides read-only access to a pool and index recording user comments but also provides an external function for creating a new comment:

    (define comments-pool (use-pool (get-component "comments.pool")))
    (define comments-index (use-index (get-component "comments.index")))

    (serve-pool comments-pool)
    (serve-pool comments-index)

    ;; External clients can't write the pool or index
    (set-read-only!)

    (define (record-comment user text (time #f))
      (let* ((tstamp (or time (timestamp)))
    	 (frame (frame-create comments-pool
                      'user user 'comment text 'time tstamp)))
        (index-frame comments-index frame 'user)
        ;; This will allow free text search on the comment
        (index-frame comments-index frame
                    'comment (stem-english-word (elts (segment text))))
        ;; This will allow more clever searching on the user name too
        (index-frame comments-index frame 'user (elts (segment user)))
        ;; And the same for the timestamp; e.g. get-timekeys might
        ;;  return something like: {JAN 2001 D5 WINTER 3AM MORNING}
        (index-frame comments-index frame 'time
                     (get-timekeys (get frame 'time)))
        frame))

    (module-export! 'record-comment)

In this example, external clients cannot write directly to the database, but they can search and examine it. They can also add to it in a particular, rigorous way defined by the procedure `record-comment`. This method ensures that the frame created is timestamped and indexed in a useful and consistent way.

## Administering Servers

The FramerD executable `fdmanager` provides a way of easily managing multiple servers. Based on a master control file, `fdmanager` starts a set of FramerD servers, arranges for their standard and error output to go to specific log files, and will restart them if they terminate. The master control file has multiple lines of the form:

     [wait=secs] [dependent]
    <server control file> fdserver args...

The parameter `wait` specifies how long `fdmanager` should wait for the server to start before moving onto the next server in the file. The flag `dependent` indicates that this server is dependent on the preceding server and should be restarted if the preceding server is restarted. The server control file is an `.fdz` file passed to `fdserver` and the fdserver args are passed along with it. Additional information about `fdmanager` can be found on its man page.

FramerD comes with an `fdservers` init file designed to work with the SYSV init scheme common on many Unix systems. This file is a shell script which tags the commands `start`, `stop`, `restart`, and `status`. It works by starting an `fdmanager` process using the file `framerd dir/servers` as the control file (e.g. `/usr/local/share/framerd/servers`).

When started from the `fdservers` init script or some other system-wide startup mechanism (for instance, `/etc/rc.local`), servers begin by running as root. However, as mentioned above, they almost immediately change their user and group ids to either `nobody` or `nogroup` or the values of the environment variables `FDSERVER_USER` and `FDSERVER_GROUP` if they are valid user and group names. Since the server running under these names will need to access the directory of the `.fdz` file (to write status files such as `.pid` and `.nid` as well as reading the data files there), it is usually best to have the groups on the files in this directory set to the FDSERVER_GROUP and to have appropriate group access permissions set.

The `fdservers` init script sets above the environment variables to the user and group name `fdaemon`. Creating this user and group (or changing the init script) is probably a good idea for non-experimental installations.

## Invoking `fdserver`

Like most of the FramerD shell commands, fdserver can be paased configuration variables of the form `VAR=VAL` or `VAR=+VAL` on the command line. Additional options for invoking fdserver are listed when fdserver is passed the `--help` option:

    Usage: fdserver [options*] ( |  | )

     where options may define configuration variables, e.g.
        FRAMERD_USER=hal9k
     or may be direct options for fdserver:
        --local                Run the server locally (e.g. as localhost)
        --log        Log connections and statistics to
                               (use '-' for stdout, '--' for stderr)
        --trace      Log transaction details to
                               (use '-' for stdout, '--' for stderr)
        --access (read-only | locking | run-with-scissors)
                               Limit OID and association access according
                               to ruleset specified
        -c | --config     Load symbol definitions from
        -m | --module     has the server use the module
        -f | --file
                               Load  before starting
        -e | --eval ''
                               Evaluate  before starting

The arguments to the `--access` option have the following meaning:

read-only
  the served pools or indices cannot be remotely modified at all
locking
  oid-level locking is provided for served pools
sloppy
  OIDs can be written but no locking is provided

The value of this option will have no effect if the server itself cannot modify one of its pools or indices; in such a case, the pool or index is read-only no matter what access is specified.

For debugging purposes, two useful arguments to fdserver are `--log` and `--trace`. These activiate logging and tracing (respectively) and send the results to a specified file. If a dash (-) is specified as the file, results go to the standard output; if two dashes (--) are specified, results go to the standard error. Logging reports each connections and disconnections of clients with the server. Tracing reports each request and the value returned.

One can specify that a server only runs locally by passing the command line arguments `--local yes` to `fdserver`. You can then access this server through server ids of the form "port@localhost". This is useful if you are using FramerD on a machine which is only intermittenly connected to the network.

Each of these options can also be specified in the server control file, by the following functions:

set-read-only!
  sets the server to be externally read-only
set-sloppy!
  sets the server to be externally writable, without locking
set-log!
  takes two stream arguments: one for connection logging, one for comprehensive transaction logging