<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="generator" content="AsciiDoc 8.6.8">
<title>Unicode</title>
<link rel="stylesheet" href="./asciidoc.css" type="text/css">
<link rel="stylesheet" href="./pygments.css" type="text/css">


<script type="text/javascript" src="./asciidoc.js"></script>
<script type="text/javascript">
/*<![CDATA[*/
asciidoc.install();
/*]]>*/
</script>
<link rel="stylesheet" href="./mlton.css" type="text/css"/>
</head>
<body class="article">
<div id="banner">
<div id="banner-home">
<a href="./Home">MLton 20130715</a>
</div>
</div>
<div id="header">
<h1>Unicode</h1>
</div>
<div id="content">
<div id="preamble">
<div class="sectionbody">
<div class="paragraph"><p>The current release of MLton does not support Unicode.  We are working
on adding support.</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">WideChar</span> structure.
</p>
</li>
<li>
<p>
UTF-8 encoded source files.
</p>
</li>
</ul></div>
<div class="paragraph"><p>There is no real support for Unicode in the <a href="DefinitionOfStandardML">Definition</a>;
there are only a few throw-away sentences along the lines of "ASCII
must be a subset of the character set in programs".</p></div>
<div class="paragraph"><p>Neither is there real support for Unicode in the <a href="BasisLibrary">Basis Library</a>.
The general consensus (which includes the opinions of the
editors of the Basis Library) is that the <span class="monospaced">WideChar</span> structure is
insufficient for the purposes of Unicode.  There is no <span class="monospaced">LargeChar</span>
structure, which in itself is a deficiency, since a programmer can not
program against the largest supported character size.</p></div>
<div class="paragraph"><p>MLton has some preliminary support for 16 and 32 bit characters and
strings.  It is even possible to include arbitrary Unicode characters
in 32-bit strings using a <span class="monospaced">\Uxxxxxxxx</span> escape sequence.  (This
longer escape sequence is a minor extension over the Definition which
only allows <span class="monospaced">\uxxxx</span>.)  This is by no means completely
satisfactory in terms of support for Unicode, but it is what is
currently available.</p></div>
<div class="paragraph"><p>There are periodic flurries of questions and discussion about Unicode
in MLton/SML.  In December 2004, there was a discussion that led to
some seemingly sound design decisions.  The discussion started at:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>http://www.mlton.org/pipermail/mlton/2004-December/026396.html</pre>
</div></div>
<div class="paragraph"><p>There is a good summary of points at:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>http://www.mlton.org/pipermail/mlton/2004-December/026440.html</pre>
</div></div>
<div class="paragraph"><p>In November 2005, there was a followup discussion and the beginning of
some coding.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>http://www.mlton.org/pipermail/mlton/2005-November/028300.html</pre>
</div></div>
<div class="paragraph"><p>We are optimistic that support will appear in the next MLton release.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_also_see">Also see</h2>
<div class="sectionbody">
<div class="paragraph"><p>The <a href="fxp">fxp</a> XML parser has some support for dealing with Unicode
documents.</p></div>
</div>
</div>
</div>
<div id="footnotes"><hr></div>
<div id="footer">
<div id="footer-text">
</div>
<div id="footer-badges">
</div>
</div>
</body>
</html>
