guile/doc/mbapi.texi

\input texinfo
@setfilename mbapi.info
@settitle Multibyte API
@setchapternewpage off

@c Open issues:

@c What's the best way to report errors?  Should functions return a
@c magic value, according to C tradition, or should they signal a
@c Guile exception?

@c 


@node Working With Multibyte Strings in C
@chapter Working With Multibyte Strings in C

Guile allows strings to contain characters drawn from a wide variety of
languages, including many Asian, Eastern European, and Middle Eastern
languages, in a uniform and unrestricted way.  The string representation
normally used in C code --- an array of @sc{ASCII} characters --- is not
sufficient for Guile strings, since they may contain characters not
present in @sc{ASCII}.

Instead, Guile uses a very large character set, and encodes each
character as a sequence of one or more bytes.  We call this
variable-width encoding a @dfn{multibyte} encoding.  Guile uses this
single encoding internally for all strings, symbol names, error
messages, etc., and performs appropriate conversions upon input and
output.

The use of this variable-width encoding is almost invisible to Scheme
code.  Strings are still indexed by character number, not by byte
offset; @code{string-length} still returns the length of a string in
characters, not in bytes.  @code{string-ref} and @code{string-set!} are
no longer guaranteed to be constant-time operations, but Guile uses
various strategies to reduce the impact of this change.

However, the encoding is visible via Guile's C interface, which gives
the user direct access to a string's bytes.  This chapter explains how
to work with Guile multibyte text in C code.  Since variable-width
encodings are clumsier to work with than simple fixed-width encodings,
Guile provides a set of standard macros and functions for manipulating
multibyte text to make the job easier.  Furthermore, Guile makes some
promises about the encoding which you can use in writing your own text
processing code.

While we discuss guaranteed properties of Guile's encoding, and provide
functions to operate on its character set, we do not actually specify
either the character set or encoding here.  This is because we expect
both of them to change in the future: currently, Guile uses the same
encoding as GNU Emacs 20.4, but we hope to change Guile (and GNU Emacs
as well) to use Unicode and UTF-8, with some extensions.  This will make
it more comfortable to use Guile with other systems which use UTF-8,
like the GTk user interface toolkit.

@menu
* Multibyte String Terminology::  
* Promised Properties of the Guile Multibyte Encoding::  
* Functions for Operating on Multibyte Text::  
* Multibyte Text Processing Errors::  
* Why Guile Does Not Use a Fixed-Width Encoding::  
@end menu


@node Multibyte String Terminology, Promised Properties of the Guile Multibyte Encoding, Working With Multibyte Strings in C, Working With Multibyte Strings in C
@section Multibyte String Terminology 

In the descriptions which follow, we make the following definitions:
@table @dfn

@item byte
A @dfn{byte} is a number between 0 and 255.  It has no inherent textual
interpretation.  So 65 is a byte, not a character.

@item character
A @dfn{character} is a unit of text.  It has no inherent numeric value.
@samp{A} and @samp{.} are characters, not bytes.  (This is different
from the C language's definition of @dfn{character}; in this chapter, we
will always use a phrase like ``the C language's @code{char} type'' when
that's what we mean.)

@item character set
A @dfn{character set} is an invertible mapping between numbers and a
given set of characters.  @sc{ASCII} is a character set assigning
characters to the numbers 0 through 127.  It maps @samp{A} onto the
number 65, and @samp{.} onto 46.

Note that a character set maps characters onto numbers, @emph{not
necessarily} onto bytes.  For example, the Unicode character set maps
the Greek lower-case @samp{alpha} character onto the number 945, which
is not a byte.

(This is what Internet standards would call a "coding character set".)

@item encoding
An encoding maps numbers onto sequences of bytes.  For example, the
UTF-8 encoding, defined in the Unicode Standard, would map the number
945 onto the sequence of bytes @samp{206 177}.  When using the
@sc{ASCII} character set, every number assigned also happens to be a
byte, so there is an obvious trivial encoding for @sc{ASCII} in bytes.

(This is what Internet standards would call a "character encoding
scheme".)

@end table

Thus, to turn a character into a sequence of bytes, you need a character
set to assign a number to that character, and then an encoding to turn
that number into a sequence of bytes.

Likewise, to interpret a sequence of bytes as a sequence of characters,
you use an encoding to extract a sequence of numbers from the bytes, and
then a character set to turn the numbers into characters.

Errors can occur while carrying out either of these processes.  For
example, under a particular encoding, a given string of bytes might not
correspond to any number.  For example, the byte sequence @samp{128 128}
is not a valid encoding of any number under UTF-8.

Having carefully defined our terminology, we will now abuse it.

We will sometimes use the word @dfn{character} to refer to the number
assigned to a character by a character set, in contexts where it's
obvious we mean a number.

Sometimes there is a close association between a particular encoding and
a particular character set.  Thus, we may sometimes refer to the
character set and encoding together as an @dfn{encoding}.


@node Promised Properties of the Guile Multibyte Encoding, Functions for Operating on Multibyte Text, Multibyte String Terminology, Working With Multibyte Strings in C
@section Promised Properties of the Guile Multibyte Encoding

Internally, Guile uses a single encoding for all text --- symbols,
strings, error messages, etc.  Here we list a number of helpful
properties of Guile's encoding.  It is correct to write code which
assumes these properties; code which uses these assumptions will be
portable to all future versions of Guile, as far as we know.

@b{Every @sc{ASCII} character is encoded as a single byte from 0 to 127, in
the obvious way.}  This means that a standard C string containing only
@sc{ASCII} characters is a valid Guile string (except for the terminator;
Guile strings store the length explicitly, so they can contain null
characters).

@b{The encodings of non-@sc{ASCII} characters use only bytes between 128
and 255.}  That is, when we turn a non-@sc{ASCII} character into a
series of bytes, none of those bytes can ever be mistaken for the
encoding of an @sc{ASCII} character.  This means that you can search a
Guile string for an @sc{ASCII} character using the standard
@code{memchr} library function.  By extension, you can search for an
@sc{ASCII} substring in a Guile string using a traditional substring
search algorithm --- you needn't add special checks to verify encoding
boundaries, etc.

@b{No character encoding is a subsequence of any other character
encoding.}  (This is just a stronger version of the previous promise.)
This means that you can search for occurrences of one Guile string
within another Guile string just as if they were raw byte strings.  You
can use the stock @code{memmem} function (provided on GNU systems, at
least) for such searches.  If you don't need the ability to represent
null characters in your text, you can still use null-termination for
strings, and use the traditional string-handling functions like
@code{strlen}, @code{strstr}, and @code{strcat}.

@b{You can always determine the full length of a character's encoding
from its first byte.}  Guile provides the macro @code{scm_mb_len} which
computes the encoding's length from its first byte.  Given the first
rule, you can see that @code{scm_mb_len (@var{b})}, for any @code{0 <=
@var{b} <= 127}, returns 1.

@b{Given an arbitrary byte position in a Guile string, you can always
find the beginning and end of the character containing that byte without
scanning too far in either direction.}  This means that, if you are sure
a byte sequence is a valid encoding of a character sequence, you can
find character boundaries without keeping track of the beginning and
ending of the overall string.  This promise relies on the fact that, in
addition to storing the string's length explicitly, Guile always either
terminates the string's storage with a zero byte, or shares it with
another string which is terminated this way.


@node Functions for Operating on Multibyte Text, Multibyte Text Processing Errors, Promised Properties of the Guile Multibyte Encoding, Working With Multibyte Strings in C
@section Functions for Operating on Multibyte Text

Guile provides a variety of functions, variables, and types for working
with multibyte text.

@menu
* Basic Multibyte Character Processing::  
* Finding Character Encoding Boundaries::  
* Multibyte String Functions::  
* Exchanging Guile Text With the Outside World in C::  
* Implementing Your Own Text Conversions::  
@end menu


@node Basic Multibyte Character Processing, Finding Character Encoding Boundaries, Functions for Operating on Multibyte Text, Functions for Operating on Multibyte Text
@subsection Basic Multibyte Character Processing

Here are the essential types and functions for working with Guile text.
Guile uses the C type @code{unsigned char *} to refer to text encoded
with Guile's encoding.

Note that any operation marked here as a ``Libguile Macro'' might
evaluate its argument multiple times.

@deftp {Libguile Type} scm_char_t
This is a signed integral type large enough to hold any character in
Guile's character set.  All character numbers are positive.
@end deftp

@deftypefn {Libguile Macro} scm_char_t scm_mb_get (const unsigned char *@var{p})
Return the character whose encoding starts at @var{p}.  If @var{p} does
not point at a valid character encoding, the behavior is undefined.
@end deftypefn

@deftypefn {Libguile Macro} int scm_mb_put (unsigned char *@var{p}, scm_char_t @var{c})
Place the encoded form of the Guile character @var{c} at @var{p}, and
return its length in bytes.  If @var{c} is not a Guile character, the
behavior is undefined.
@end deftypefn

@deftypevr {Libguile Constant} int scm_mb_max_len
The maximum length of any character's encoding, in bytes.  You may
assume this is relatively small --- less than a dozen or so.
@end deftypevr

@deftypefn {Libguile Macro} int scm_mb_len (unsigned char @var{b})
If @var{b} is the first byte of a character's encoding, return the full
length of the character's encoding, in bytes.  If @var{b} is not a valid
leading byte, the behavior is undefined.
@end deftypefn

@deftypefn {Libguile Macro} int scm_mb_char_len (scm_char_t @var{c})
Return the length of the encoding of the character @var{c}, in bytes.
If @var{c} is not a valid Guile character, the behavior is undefined.
@end deftypefn

@deftypefn {Libguile Function} scm_char_t scm_mb_get_func (const unsigned char *@var{p})
@deftypefnx {Libguile Function} int scm_mb_put_func (unsigned char *@var{p}, scm_char_t @var{c})
@deftypefnx {Libguile Function} int scm_mb_len_func (unsigned char @var{b})
@deftypefnx {Libguile Function} int scm_mb_char_len_func (scm_char_t @var{c})
These are functions identical to the corresponding macros.  You can use
them in situations where the overhead of a function call is acceptable,
and the cleaner semantics of function application are desireable.
@end deftypefn


@node Finding Character Encoding Boundaries, Multibyte String Functions, Basic Multibyte Character Processing, Functions for Operating on Multibyte Text
@subsection Finding Character Encoding Boundaries

These are functions for finding the boundaries between characters in
multibyte text.

Note that any operation marked here as a ``Libguile Macro'' might
evaluate its argument multiple times, unless the definition promises
otherwise.

@deftypefn {Libguile Macro} int scm_mb_boundary_p (const unsigned char *@var{p})
Return non-zero iff @var{p} points to the start of a character in
multibyte text.

This macro will evaluate its argument only once.
@end deftypefn

@deftypefn {Libguile Function} {const unsigned char *} scm_mb_floor (const unsigned char *@var{p})
``Round'' @var{p} to the previous character boundary.  That is, if
@var{p} points to the middle of the encoding of a Guile character,
return a pointer to the first byte of the encoding.  If @var{p} points
to the start of the encoding of a Guile character, return @var{p}
unchanged.
@end deftypefn

@deftypefn {libguile Function} {const unsigned char *} scm_mb_ceiling (const unsigned char *@var{p})
``Round'' @var{p} to the next character boundary.  That is, if @var{p}
points to the middle of the encoding of a Guile character, return a
pointer to the first byte of the encoding of the next character.  If
@var{p} points to the start of the encoding of a Guile character, return
@var{p} unchanged.
@end deftypefn

Note that it is usually not friendly for functions to silently correct
byte offsets that point into the middle of a character's encoding.  Such
offsets almost always indicate a programming error, and they should be
reported as early as possible.  So, when you write code which operates
on multibyte text, you should not use functions like these to ``clean
up'' byte offsets which the originator believes to be correct; instead,
your code should signal a @code{text:not-char-boundary} error as soon as
it detects an invalid offset.  @xref{Multibyte Text Processing Errors}.


@node Multibyte String Functions, Exchanging Guile Text With the Outside World in C, Finding Character Encoding Boundaries, Functions for Operating on Multibyte Text
@subsection Multibyte String Functions

These functions allow you to operate on multibyte strings: sequences of
character encodings.

@deftypefn {Libguile Function} int scm_mb_count (const unsigned char *@var{p}, int @var{len})
Return the number of Guile characters encoded by the @var{len} bytes at
@var{p}.

If the sequence contains any invalid character encodings, or ends with
an incomplete character encoding, signal a @code{text:bad-encoding}
error.
@end deftypefn

@deftypefn {Libguile Macro} scm_char_t scm_mb_walk (unsigned char **@var{pp})
Return the character whose encoding starts at @code{*@var{pp}}, and
advance @code{*@var{pp}} to the start of the next character.  Return -1
if @code{*@var{pp}} does not point to a valid character encoding.
@end deftypefn

@deftypefn {Libguile Function} {const unsigned char *} scm_mb_prev (const unsigned char *@var{p})
If @var{p} points to the middle of the encoding of a Guile character,
return a pointer to the first byte of the encoding.  If @var{p} points
to the start of the encoding of a Guile character, return the start of
the previous character's encoding.

This is like @code{scm_mb_floor}, but the returned pointer will always
be before @var{p}.  If you use this function to drive an iteration, it
guarantees backward progress.
@end deftypefn

@deftypefn {Libguile Function} {const unsigned char *} scm_mb_next (const unsigned char *@var{p})
If @var{p} points to the encoding of a Guile character, return a pointer
to the first byte of the encoding of the next character.

This is like @code{scm_mb_ceiling}, but the returned pointer will always
be after @var{p}.  If you use this function to drive an iteration, it
guarantees forward progress.
@end deftypefn

@deftypefn {Libguile Function} {const unsigned char *} scm_mb_index (const unsigned char *@var{p}, int @var{len}, int @var{i})
Assuming that the @var{len} bytes starting at @var{p} are a
concatenation of valid character encodings, return a pointer to the
start of the @var{i}'th character encoding in the sequence.

This function scans the sequence from the beginning to find the
@var{i}'th character, and will generally require time proportional to
the distance from @var{p} to the returned address.

If the sequence contains any invalid character encodings, or ends with
an incomplete character encoding, signal a @code{text:bad-encoding}
error.
@end deftypefn

It is common to process the characters in a string from left to right.
However, if you fetch each character using @code{scm_mb_index}, each
call will scan the text from the beginning, so your loop will require
time proportional to at least the square of the length of the text.  To
avoid this poor performance, you can use an @code{scm_mb_cache}
structure and the @code{scm_mb_index_cached} macro.

@deftp {Libguile Type} {struct scm_mb_cache}
This structure holds information that allows a string scanning operation
to use the results from a previous scan of the string.  It has the
following members:
@table @code

@item character
An index, in characters, into the string.

@item byte
The index, in bytes, of the start of that character.

@end table

In other words, @code{byte} is the byte offset of the
@code{character}'th character of the string.  Note that if @code{byte}
and @code{character} are equal, then all characters before that point
must have encodings exactly one byte long, and the string can be indexed
normally.

All elements of a @code{struct scm_mb_cache} structure should be
initialized to zero before its first use, and whenever the string's text
changes.
@end deftp

@deftypefn {Libguile Macro} const unsigned char *scm_mb_index_cached (const unsigned char *@var{p}, int @var{len}, int @var{i}, struct scm_mb_cache *@var{cache})
@deftypefnx {Libguile Function} const unsigned char *scm_mb_index_cached_func (const unsigned char *@var{p}, int @var{len}, int @var{i}, struct scm_mb_cache *@var{cache})
This macro and this function are identical to @code{scm_mb_index},
except that they may consult and update *@var{cache} in order to avoid
scanning the string from the beginning.  @code{scm_mb_index_cached} is a
macro, so it may have less overhead than
@code{scm_mb_index_cached_func}, but it may evaluate its arguments more
than once.

Using @code{scm_mb_index_cached} or @code{scm_mb_index_cached_func}, you
can scan a string from left to right, or from right to left, in time
proportional to the length of the string.  As long as each character
fetched is less than some constant distance before or after the previous
character fetched with @var{cache}, each access will require constant
time.
@end deftypefn

Guile also provides functions to convert between an encoded sequence of
characters, and an array of @code{scm_char_t} objects.

@deftypefn {Libguile Function} scm_char_t *scm_mb_multibyte_to_fixed (const unsigned char *@var{p}, int @var{len}, int *@var{result_len})
Convert the variable-width text in the @var{len} bytes at @var{p}
to an array of @code{scm_char_t} values.  Return a pointer to the array,
and set @code{*@var{result_len}} to the number of elements it contains.
The returned array is allocated with @code{malloc}, and it is the
caller's responsibility to free it.

If the text is not a sequence of valid character encodings, this
function will signal a @code{text:bad-encoding} error.
@end deftypefn

@deftypefn {Libguile Function} unsigned char *scm_mb_fixed_to_multibyte (const scm_char_t *@var{fixed}, int @var{len}, int *@var{result_len})
Convert the array of @code{scm_char_t} values to a sequence of
variable-width character encodings.  Return a pointer to the array of
bytes, and set @code{*@var{result_len}} to its length, in bytes.

The returned byte sequence is terminated with a zero byte, which is not
counted in the length returned in @code{*@var{result_len}}.

The returned byte sequence is allocated with @code{malloc}; it is the
caller's responsibility to free it.

If the text is not a sequence of valid character encodings, this
function will signal a @code{text:bad-encoding} error.
@end deftypefn


@node Exchanging Guile Text With the Outside World in C, Implementing Your Own Text Conversions, Multibyte String Functions, Functions for Operating on Multibyte Text
@subsection Exchanging Guile Text With the Outside World in C

[[This is kind of a heavy-weight model, given that one end of the
conversion is always going to be the Guile encoding.  Any way to shorten
things a bit?]]

Guile provides functions for converting between Guile's internal text
representation and encodings popular in the outside world.  These
functions are closely modeled after the @code{iconv} functions available
on some systems.

To convert text between two encodings, you should first call
@code{scm_mb_iconv_open} to indicate the source and destination
encodings; this function returns a context object which records the
conversion to perform.

Then, you should call @code{scm_mb_iconv} to actually convert the text.
This function expects input and output buffers, and a pointer to the
context you got from @var{scm_mb_iconv_open}.  You don't need to pass
all your input to @code{scm_mb_iconv} at once; you can invoke it on
successive blocks of input (as you read it from a file, say), and it
will convert as much as it can each time, indicating when you should
grow your output buffer.

An encoding may be @dfn{stateless}, or @dfn{stateful}.  In most
encodings, a contiguous group of bytes from the sequence completely
specifies a particular character; these are stateless encodings.
However, some encodings require you to look back an unbounded number of
bytes in the stream to assign a meaning to a particular byte sequence;
such encodings are stateful.

For example, in the @samp{ISO-2022-JP} encoding for Japanese text, the
byte sequence @samp{27 36 66} indicates that subsequent bytes should be
taken in pairs and interpreted as characters from the JIS-0208 character
set.  An arbitrary number of byte pairs may follow this sequence.  The
byte sequence @samp{27 40 66} indicates that subsequent bytes should be
interpreted as @sc{ASCII}.  In this encoding, you cannot tell whether a
given byte is an @sc{ASCII} character without looking back an arbitrary
distance for the most recent escape sequence, so it is a stateful
encoding.

In Guile, if a conversion involves a stateful encoding, the context
object carries any necessary state.  Thus, you can have many independent
conversions to or from stateful encodings taking place simultaneously,
as long as each data stream uses its own context object for the
conversion.

@deftp {Libguile Type} {struct scm_mb_iconv}
This is the type for context objects, which represent the encodings and
current state of an ongoing text conversion.  A @code{struct
scm_mb_iconv} records the source and destination encodings, and keeps
track of any information needed to handle stateful encodings.
@end deftp

@deftypefn {Libguile Function} {struct scm_mb_iconv *} scm_mb_iconv_open (const char *@var{tocode}, const char *@var{fromcode})
Return a pointer to a new @code{struct scm_mb_iconv} context object,
ready to convert from the encoding named @var{fromcode} to the encoding
named @var{tocode}.  For stateful encodings, the context object is in
some appropriate initial state, ready for use with the
@code{scm_mb_iconv} function.

When you are done using a context object, you may call
@code{scm_mb_iconv_close} to free it.

If either @var{tocode} or @var{fromcode} is not the name of a known
encoding, this function will signal the @code{text:unknown-conversion}
error, described below.

@c Try to use names here from the IANA list: 
@c see ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets
Guile supports at least these encodings:
@table @samp 

@item US-ASCII
@sc{US-ASCII}, in the standard one-character-per-byte encoding.

@item ISO-8859-1
The usual character set for Western European languages, in its usual
one-character-per-byte encoding.

@item Guile-MB
Guile's current internal multibyte encoding.  The actual encoding this
name refers to will change from one version of Guile to the next.  You
should use this when converting data between external sources and the
encoding used by Guile objects.

You should @emph{not} use this as the encoding for data presented to the
outside world, for two reasons.  1) Its meaning will change over time,
so data written using the @samp{guile} encoding with one version of
Guile might not be readable with the @samp{guile} encoding in another
version of Guile.  2) It currently corresponds to @samp{Emacs-Mule},
which invented for Emacs's internal use, and was never intended to serve
as an exchange medium.

@item Guile-Wide
Guile's character set, as an array of @code{scm_char_t} values.

Note that this encoding is even less suitable for public use than
@samp{Guile}, since the exact sequence of bytes depends heavily on the
size and endianness the host system uses for @code{scm_char_t}.  Using
this encoding is very much like calling the
@code{scm_mb_multibyte_to_fixed} or @code{scm_mb_fixed_to_multibyte}
functions, except that @code{scm_mb_iconv} gives you more control over
buffer allocation and management.

@item Emacs-Mule
This is the variable-length encoding for multi-lingual text by GNU
Emacs, at least through version 20.4.  You probably should not use this
encoding, as it is designed only for Emacs's internal use.  However, we
provide it here because it's trivial to support, and some people
probably do have @samp{emacs-mule}-format files lying around.

@end table

(At the moment, this list doesn't include any character sets suitable for
external use that can actually handle multilingual data; this is
unfortunate, as it encourages users to write data in Emacs-Mule format,
which nobody but Emacs and Guile understands.  We hope to add support
for Unicode in UTF-8 soon, which should solve this problem.)

Case is not significant in encoding names.

You can define your own conversions; see @ref{Implementing Your Own Text
Conversions}.
@end deftypefn

@deftypefn {Libguile Function} int scm_mb_have_encoding (const char *@var{encoding})
Return a non-zero value if Guile supports the encoding named @var{encoding}[[]]
@end deftypefn

@deftypefn {Libguile Function} size_t scm_mb_iconv (struct scm_mb_iconv *@var{context}, const char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft})
Convert a sequence of characters from one encoding to another.  The
argument @var{context} specifies the encodings to use for the input and
output, and carries state for stateful encodings; use
@code{scm_mb_iconv_open} to create a @var{context} object for a
particular conversion.

Upon entry to the function, @code{*@var{inbuf}} should point to the
input buffer, and @code{*@var{inbytesleft}} should hold the number of
input bytes present in the buffer; @code{*@var{outbuf}} should point to
the output buffer, and @code{*@var{outbytesleft}} should hold the number
of bytes available to hold the conversion results in that buffer.

Upon exit from the function, @code{*@var{inbuf}} points to the first
unconsumed byte of input, and @code{*@var{inbytesleft}} holds the number
of unconsumed input bytes; @code{*@var{outbuf}} points to the byte after
the last output byte, and @code{*@var{outbyteleft}} holds the number of
bytes left unused in the output buffer.

For stateful encodings, @var{context} carries encoding state from one
call to @code{scm_mb_iconv} to the next.  Thus, successive calls to
@var{scm_mb_iconv} which use the same context object can convert a
stream of data one chunk at a time.  

If @var{inbuf} is zero or @code{*@var{inbuf}} is zero, then the call is
taken as a request to reset the states of the input and the output
encodings.  If @var{outbuf} is non-zero and @code{*@var{outbuf}} is
non-zero, then @code{scm_mb_iconv} stores a byte sequence in the output
buffer to put the output encoding in its initial state.  If the output
buffer is not large enough to hold this byte sequence,
@code{scm_mb_iconv} returns @code{scm_mb_iconv_too_big}, and leaves
the shift states of @var{context}'s input and output encodings
unchanged.

The @code{scm_mb_iconv} function always consumes only complete
characters or shift sequences from the input buffer, and the output
buffer always contains a sequence of complete characters or escape
sequences.

If the input sequence contains characters which are not expressible in
the output encoding, @code{scm_mb_iconv} converts it in an
implementation-defined way.  It may simply delete the character.

Some encodings use byte sequences which do not correspond to any textual
character.  For example, the escape sequence of a stateful encoding has
no textual meaning.  When converting from such an encoding, a call to
@code{scm_mb_iconv} might consume input but produce no output, since the
input sequence might contain only escape sequences.

Normally, @code{scm_mb_iconv} returns the number of input characters it
could not convert perfectly to the ouput encoding.  However, it may
return one of the @code{scm_mb_iconv_} codes described below, to
indicate an error.  All of these codes are negative values.

If the input sequence contains an invalid character encoding, conversion
stops before the invalid input character, and @code{scm_mb_iconv}
returns the constant value @code{scm_mb_iconv_bad_encoding}.

If the input sequence ends with an incomplete character encoding,
@code{scm_mb_iconv} will leave it in the input buffer, unconsumed, and
return the constant value @code{scm_mb_iconv_incomplete_encoding}.  This
is not necessarily an error, if you expect to call @code{scm_mb_iconv}
again with more data which might contain the rest of the encoding
fragment.

If the output buffer does not contain enough room to hold the converted
form of the complete input text, @code{scm_mb_iconv} converts as much as
it can, changes the input and output pointers to reflect the amount of
text successfully converted, and then returns
@code{scm_mb_iconv_too_big}.
@end deftypefn

Here are the status codes that might be returned by @code{scm_mb_iconv}.
They are all negative integers.
@table @code

@item scm_mb_iconv_too_big
The conversion needs more room in the output buffer.  Some characters
may have been consumed from the input buffer, and some characters may
have been placed in the available space in the output buffer.

@item scm_mb_iconv_bad_encoding
@code{scm_mb_iconv} encountered an invalid character encoding in the
input buffer.  Conversion stopped before the invalid character, so there
may be some characters consumed from the input buffer, and some
converted text in the output buffer.

@item scm_mb_iconv_incomplete_encoding
The input buffer ends with an incomplete character encoding.  The
incomplete encoding is left in the input buffer, unconsumed.  This is
not necessarily an error, if you expect to call @code{scm_mb_iconv}
again with more data which might contain the rest of the incomplete
encoding.

@end table


Finally, Guile provides a function for destroying conversion contexts.

@deftypefn {Libguile Function} void scm_mb_iconv_close (struct scm_mb_iconv *@var{context})
Deallocate the conversion context object @var{context}, and all other
resources allocated by the call to @code{scm_mb_iconv_open} which
returned @var{context}.
@end deftypefn


@node Implementing Your Own Text Conversions,  , Exchanging Guile Text With the Outside World in C, Functions for Operating on Multibyte Text
@subsection Implementing Your Own Text Conversions

[[note that conversions to and from Guile must produce streams
containing only valid character encodings, or else Guile will crash]]

This section describes the interface for adding your own encoding
conversions for use with @code{scm_mb_iconv}.  The interface here is
borrowed from the GNOME Project's @file{libunicode} library.

Guile's @code{scm_mb_iconv} function works by converting the input text
to a stream of @code{scm_char_t} characters, and then converting
those characters to the desired output encoding.  This makes it easy
for Guile to choose the appropriate conversion back ends for an
arbitrary pair of input and output encodings, but it also means that the
accuracy and quality of the conversions depends on the fidelity of
Guile's internal character set to the source and destination encodings.
Since @code{scm_mb_iconv} will be used almost exclusively for converting
to and from Guile's internal character set, this shouldn't be a problem.

To add support for a particular encoding to Guile, you must provide one
function (called the @dfn{read} function) which converts from your
encoding to an array of @code{scm_char_t}'s, and another function
(called the @dfn{write} function) to convert from an array of
@code{scm_char_t}'s back into your encoding.  To convert from some
encoding @var{a} to some other encoding @var{b}, Guile pairs up
@var{a}'s read function with @var{b}'s write function.  Each call to
@code{scm_mb_iconv} passes text in encoding @var{a} through the read
function, to produce an array of @code{scm_char_t}'s, and then passes
that array to the write function, to produce text in encoding @var{b}.

For stateful encodings, a read or write function can hang its own data
structures off the conversion object, and provide its own functions to
allocate and destroy them; this allows read and write functions to
maintain whatever state they like.

The Guile conversion back end represents each available encoding with a
@code{struct scm_mb_encoding} object.

@deftp {Libguile Type} {struct scm_mb_encoding}
This data structure describes an encoding.  It has the following
members:

@table @code

@item char **names
An array of strings, giving the various names for this encoding.  The
array should be terminated by a zero pointer.  Case is not significant
in encoding names.

The @code{scm_mb_iconv_open} function searches the list of registered
encodings for an encoding whose @code{names} array matches its
@var{tocode} or @var{fromcode} argument.

@item int (*init) (void **@var{cookie})
An initialization function for the encoding's private data.
@code{scm_mb_iconv_open} will call this function, passing it the address
of the cookie for this encoding in this context.  (We explain cookies
below.)  There is no way for the @code{init} function to tell whether
the encoding will be used for reading or writing.

Note that @code{init} receives a @emph{pointer} to the cookie, not the
cookie itself.  Because the type of @var{cookie} is @code{void **}, the
C compiler will not check it as carefully as it would other types.

The @code{init} member may be zero, indicating that no initialization is
necessary for this encoding.

@item int (*destroy) (void **@var{cookie})
A deallocation function for the encoding's private data.
@code{scm_mb_iconv_close} calls this function, passing it the address of
the cookie for this encoding in this context.  The @code{destroy}
function should free any data the @code{init} function allocated.

Note that @code{destroy} receives a @emph{pointer} to the cookie, not the
cookie itself.  Because the type of @var{cookie} is @code{void **}, the
C compiler will not check it as carefully as it would other types.

The @code{destroy} member may be zero, indicating that this encoding
doesn't need to perform any special action to destroy its local data.

@item int (*reset) (void *@var{cookie}, char **@var{outbuf}, size_t *@var{outbytesleft})
Put the encoding into its initial shift state.  Guile calls this
function whether the encoding is being used for input or output, so this
should take appropriate steps for both directions.  If @var{outbuf} and
@var{outbytesleft} are valid, the reset function should emit an escape
sequence to reset the output stream to its initial state; @var{outbuf}
and @var{outbytesleft} should be handled just as for
@code{scm_mb_iconv}.

This function can return an @code{scm_mb_iconv_} error code
(@pxref{Exchanging Guile Text With the Outside World in C}).  If it
returns @code{scm_mb_iconv_too_big}, then the output buffer's shift
state must be left unchanged.

Note that @code{reset} receives the cookie's value itself, not a pointer
to the cookie, as the @code{init} and @code{destroy} functions do.

The @code{reset} member may be zero, indicating that this encoding
doesn't use a shift state.

@item enum scm_mb_read_result (*read) (void *@var{cookie}, const char **@var{inbuf},  size_t *@var{inbytesleft}, scm_char_t **@var{outbuf}, size_t *@var{outcharsleft})
Read some bytes and convert into an array of Guile characters.  This is
the encoding's read function.

On entry, there are *@var{inbytesleft} bytes of text at *@var{inbuf} to
be converted, and *@var{outcharsleft} characters available at
*@var{outbuf} to hold the results.

On exit, *@var{inbytesleft} and *@var{inbuf} indicate the input bytes
still not consumed.  *@var{outcharsleft} and *@var{outbuf} indicate the
output buffer space still not filled.  (By exclusion, these indicate
which input bytes were consumed, and which output characters were
produced.)

Return one of the @code{enum scm_mb_read_result} values, described below.

Note that @code{read} receives the cookie's value itself, not a pointer
to the cookie, as the @code{init} and @code{destroy} functions do.

@item enum scm_mb_write_result (*write) (void *@var{cookie}, scm_char_t **@var{inbuf}, size_t *@var{incharsleft}, **@var{outbuf}, size_t *@var{outbytesleft})
Convert an array of Guile characters to output bytes.  This is
the encoding's write function.

On entry, there are *@var{incharsleft} Guile characters available at
*@var{inbuf}, and *@var{outbytesleft} bytes available to store output at
*@var{outbuf}.

On exit, *@var{incharsleft} and *@var{inbuf} indicate the number of
Guile characters left unconverted (because there was insufficient room
in the output buffer to hold their converted forms), and
*@var{outbytesleft} and *@var{outbuf} indicate the unused portion of the
output buffer.

Return one of the @code{scm_mb_write_result} values, described below.

Note that @code{write} receives the cookie's value itself, not a pointer
to the cookie, as the @code{init} and @code{destroy} functions do.

@item struct scm_mb_encoding *next
This is used by Guile to maintain a linked list of encodings.  It is
filled in when you call @code{scm_mb_register_encoding} to add your
encoding to the list.

@end table
@end deftp

Here is the enumerated type for the values an encoding's read function
can return:

@deftp {Libguile Type} {enum scm_mb_read_result}
This type represents the result of a call to an encoding's read
function.  It has the following values:

@table @code

@item scm_mb_read_ok
The read function consumed at least one byte of input.

@item scm_mb_read_incomplete
The data present in the input buffer does not contain a complete
character encoding.  No input was consumed, and no characters were
produced as output.  This is not necessarily an error status, if there
is more data to pass through.

@item scm_mb_read_error
The input contains an invalid character encoding.

@end table
@end deftp

Here is the enumerated type for the values an encoding's write function
can return:

@deftp {Libguile Type} {enum scm_mb_write_result}
This type represents the result of a call to an encoding's write
function.  It has the following values:

@table @code

@item scm_mb_write_ok
The write function was able to convert all the characters in @var{inbuf}
successfully.

@item scm_mb_write_too_big
The write function filled the output buffer, but there are still
characters in @var{inbuf} left unconsumed; @var{inbuf} and
@var{incharsleft} indicate the unconsumed portion of the input buffer.

@end table
@end deftp


Conversions to or from stateful encodings need to keep track of each
encoding's current state.  Each conversion context contains two
@code{void *} variables called @dfn{cookies}, one for the input
encoding, and one for the output encoding.  These cookies are passed to
the encodings' functions, for them to use however they please.  A
stateful encoding can use its cookie to hold a pointer to some object
which maintains the context's current shift state.  Stateless encodings
will probably not use their cookies.

The cookies' lifetime is the same as that of the context object.  When
the user calls @code{scm_mb_iconv_close} to destroy a context object,
@code{scm_mb_iconv_close} calls the input and output encodings'
@code{destroy} functions, passing them their respective cookies, so each
encoding can free any data it allocated for that context.

Note that, if a read or write function returns a successful result code
like @code{scm_mb_read_ok} or @code{scm_mb_write_ok}, then the remaining
input, together with the output, must together represent the complete
input text; the encoding may not store any text temporarily in its
cookie.  This is because, if @code{scm_mb_iconv} returns a successful
result to the user, it is correct for the user to assume that all the
consumed input has been converted and placed in the output buffer.
There is no ``flush'' operation to push any final results out of the
encodings' buffers.

Here is the function you call to register a new encoding with the
conversion system:

@deftypefn {Libguile Function} void scm_mb_register_encoding (struct scm_mb_encoding *@var{encoding})
Add the encoding described by @code{*@var{encoding}} to the set
understood by @code{scm_mb_iconv_open}.  Once you have registered your
encoding, you can use it by calling @code{scm_mb_iconv_open} with one of
the names in @code{@var{encoding}->names}.
@end deftypefn


@node Multibyte Text Processing Errors, Why Guile Does Not Use a Fixed-Width Encoding, Functions for Operating on Multibyte Text, Working With Multibyte Strings in C
@section Multibyte Text Processing Errors

This section describes error conditions which code can signal to
indicate problems encountered while processing multibyte text.  In each
case, the arguments @var{message} and @var{args} are an error format
string and arguments to be substituted into the string, as accepted by
the @code{display-error} function.

@deffn Condition text:not-char-boundary func message args object offset
By calling @var{func}, the program attempted to access a character at
byte offset @var{offset} in the Guile object @var{object}, but
@var{offset} is not the start of a character's encoding in @var{object}.

Typically, @var{object} is a string or symbol.  If the function signalling
the error cannot find the Guile object that contains the text it is
inspecting, it should use @code{#f} for @var{object}.
@end deffn

@deffn Condition text:bad-encoding func message args object
By calling @var{func}, the program attempted to interpret the text in
@var{object}, but @var{object} contains a byte sequence which is not a
valid encoding for any character.
@end deffn

@deffn Condition text:not-guile-char func message args number
By calling @var{func}, the program attempted to treat @var{number} as the
number of a character in the Guile character set, but @var{number} does
not correspond to any character in the Guile character set.
@end deffn

@deffn Condition text:unknown-conversion func message args from to
By calling @var{func}, the program attempted to convert from an encoding
named @var{from} to an encoding named @var{to}, but Guile does not
support such a conversion.
@end deffn

@deftypevr {Libguile Variable} SCM scm_text_not_char_boundary
@deftypevrx {Libguile Variable} SCM scm_text_bad_encoding
@deftypevrx {Libguile Variable} SCM scm_text_not_guile_char
These variables hold the scheme symbol objects whose names are the
condition symbols above.  You can use these when signalling these
errors, instead of looking them up yourself.
@end deftypevr


@node Why Guile Does Not Use a Fixed-Width Encoding,  , Multibyte Text Processing Errors, Working With Multibyte Strings in C
@section Why Guile Does Not Use a Fixed-Width Encoding

Multibyte encodings are clumsier to work with than encodings which use a
fixed number of bytes for every character.  For example, using a
fixed-width encoding, we can extract the @var{i}th character of a string
in constant time, and we can always substitute the @var{i}th character
of a string with any other character without reallocating or copying the
string.

However, there are no fixed-width encodings which include the characters
we wish to include, and also fit in a reasonable amount of space.
Despite the Unicode standard's claims to the contrary, Unicode is not
really a fixed-width encoding.  Unicode uses surrogate pairs to
represent characters outside the 16-bit range; a surrogate pair must be
treated as a single character, but occupies two 16-bit spaces.  As of
this writing, there are already plans to assign characters to the
surrogate character codes.  Three- and four-byte encodings are
too wasteful for a majority of Guile's users, who only need @sc{ASCII}
and a few accented characters.

Another alternative would be to have several different fixed-width
string representations, each with a different element size.  For each
string, Guile would use the smallest element size capable of
accomodating the string's text.  This would allow users of English and
the Western European languages to use the traditional memory-efficient
encodings.  However, if Guile has @var{n} string representations, then
users must write @var{n} versions of any code which manipulates text
directly --- one for each element size.  And if a user wants to operate
on two strings simultaneously, and wants to avoid testing the string
sizes within the loop, she must make @var{n}*@var{n} copies of the loop.
Most users will simply not bother.  Instead, they will write code which
supports only one string size, leaving us back where we started.  By
using a single internal representation, Guile makes it easier for users
to write multilingual code.

[[What about tagging each string with its encoding?
"Every extension must be written to deal with every encoding"]]

[[You don't really want to index strings anyway.]]

Finally, Guile's multibyte encoding is not so bad.  Unlike a two- or
four-byte encoding, it is efficient in space for American and European
users.  Furthermore, the properties described above mean that many
functions can be coded just as they would for a single-byte encoding;
see @ref{Promised Properties of the Guile Multibyte Encoding}.

@bye