diff options
Diffstat (limited to 'gcc-4.4.3/libstdc++-v3/doc/xml/manual/codecvt.xml')
| -rw-r--r-- | gcc-4.4.3/libstdc++-v3/doc/xml/manual/codecvt.xml | 730 |
1 files changed, 730 insertions, 0 deletions
diff --git a/gcc-4.4.3/libstdc++-v3/doc/xml/manual/codecvt.xml b/gcc-4.4.3/libstdc++-v3/doc/xml/manual/codecvt.xml new file mode 100644 index 000000000..c836f9d0a --- /dev/null +++ b/gcc-4.4.3/libstdc++-v3/doc/xml/manual/codecvt.xml @@ -0,0 +1,730 @@ +<sect1 id="manual.localization.facet.codecvt" xreflabel="codecvt"> +<?dbhtml filename="codecvt.html"?> + +<sect1info> + <keywordset> + <keyword> + ISO C++ + </keyword> + <keyword> + codecvt + </keyword> + </keywordset> +</sect1info> + +<title>codecvt</title> + +<para> +The standard class codecvt attempts to address conversions between +different character encoding schemes. In particular, the standard +attempts to detail conversions between the implementation-defined wide +characters (hereafter referred to as wchar_t) and the standard type +char that is so beloved in classic <quote>C</quote> (which can now be +referred to as narrow characters.) This document attempts to describe +how the GNU libstdc++ implementation deals with the conversion between +wide and narrow characters, and also presents a framework for dealing +with the huge number of other encodings that iconv can convert, +including Unicode and UTF8. Design issues and requirements are +addressed, and examples of correct usage for both the required +specializations for wide and narrow characters and the +implementation-provided extended functionality are given. +</para> + +<sect2 id="facet.codecvt.req" xreflabel="facet.codecvt.req"> +<title>Requirements</title> + +<para> +Around page 425 of the C++ Standard, this charming heading comes into view: +</para> + +<blockquote> +<para> +22.2.1.5 - Template class codecvt +</para> +</blockquote> + +<para> +The text around the codecvt definition gives some clues: +</para> + +<blockquote> +<para> +<emphasis> +-1- The class codecvt<internT,externT,stateT> is for use when +converting from one codeset to another, such as from wide characters +to multibyte characters, between wide character encodings such as +Unicode and EUC. +</emphasis> +</para> +</blockquote> + +<para> +Hmm. So, in some unspecified way, Unicode encodings and +translations between other character sets should be handled by this +class. +</para> + +<blockquote> +<para> +<emphasis> +-2- The stateT argument selects the pair of codesets being mapped between. +</emphasis> +</para> +</blockquote> + +<para> +Ah ha! Another clue... +</para> + +<blockquote> +<para> +<emphasis> +-3- The instantiations required in the Table ?? +(lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and +codecvt<char,char,mbstate_t>, convert the implementation-defined +native character set. codecvt<char,char,mbstate_t> implements a +degenerate conversion; it does not convert at +all. codecvt<wchar_t,char,mbstate_t> converts between the native +character sets for tiny and wide characters. Instantiations on +mbstate_t perform conversion between encodings known to the library +implementor. Other encodings can be converted by specializing on a +user-defined stateT type. The stateT object can contain any state that +is useful to communicate to or from the specialized do_convert member. +</emphasis> +</para> +</blockquote> + +<para> +At this point, a couple points become clear: +</para> + +<para> +One: The standard clearly implies that attempts to add non-required +(yet useful and widely used) conversions need to do so through the +third template parameter, stateT.</para> + +<para> +Two: The required conversions, by specifying mbstate_t as the third +template parameter, imply an implementation strategy that is mostly +(or wholly) based on the underlying C library, and the functions +mcsrtombs and wcsrtombs in particular.</para> +</sect2> + +<sect2 id="facet.codecvt.design" xreflabel="facet.codecvt.design"> +<title>Design</title> + +<sect3 id="codecvt.design.wchar_t_size" xreflabel="codecvt.design.wchar_t_size"> + <title><type>wchar_t</type> Size</title> + + <para> + The simple implementation detail of wchar_t's size seems to + repeatedly confound people. Many systems use a two byte, + unsigned integral type to represent wide characters, and use an + internal encoding of Unicode or UCS2. (See AIX, Microsoft NT, + Java, others.) Other systems, use a four byte, unsigned integral + type to represent wide characters, and use an internal encoding + of UCS4. (GNU/Linux systems using glibc, in particular.) The C + programming language (and thus C++) does not specify a specific + size for the type wchar_t. + </para> + + <para> + Thus, portable C++ code cannot assume a byte size (or endianness) either. + </para> + </sect3> + +<sect3 id="codecvt.design.unicode" xreflabel="codecvt.design.unicode"> + <title>Support for Unicode</title> + <para> + Probably the most frequently asked question about code conversion + is: "So dudes, what's the deal with Unicode strings?" + The dude part is optional, but apparently the usefulness of + Unicode strings is pretty widely appreciated. Sadly, this specific + encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10, + etc etc etc) are not mentioned in the C++ standard. + </para> + + <para> + A couple of comments: + </para> + + <para> + The thought that all one needs to convert between two arbitrary + codesets is two types and some kind of state argument is + unfortunate. In particular, encodings may be stateless. The naming + of the third parameter as stateT is unfortunate, as what is really + needed is some kind of generalized type that accounts for the + issues that abstract encodings will need. The minimum information + that is required includes: + </para> + + <itemizedlist> + <listitem> + <para> + Identifiers for each of the codesets involved in the + conversion. For example, using the iconv family of functions + from the Single Unix Specification (what used to be called + X/Open) hosted on the GNU/Linux operating system allows + bi-directional mapping between far more than the following + tantalizing possibilities: + </para> + + <para> + (An edited list taken from <code>`iconv --list`</code> on a + Red Hat 6.2/Intel system: + </para> + +<blockquote> +<programlisting> +8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7, +ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD, +GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3, +ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, +ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14, +ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4, +ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4, +UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8, +UTF-16, UTF8, UTF16). +</programlisting> +</blockquote> + +<para> +For iconv-based implementations, string literals for each of the +encodings (i.e. "UCS-2" and "UTF-8") are necessary, +although for other, +non-iconv implementations a table of enumerated values or some other +mechanism may be required. +</para> +</listitem> + +<listitem><para> + Maximum length of the identifying string literal. +</para></listitem> + +<listitem><para> + Some encodings require explicit endian-ness. As such, some kind + of endian marker or other byte-order marker will be necessary. See + "Footnotes for C/C++ developers" in Haible for more information on + UCS-2/Unicode endian issues. (Summary: big endian seems most likely, + however implementations, most notably Microsoft, vary.) +</para></listitem> + +<listitem><para> + Types representing the conversion state, for conversions involving + the machinery in the "C" library, or the conversion descriptor, for + conversions using iconv (such as the type iconv_t.) Note that the + conversion descriptor encodes more information than a simple encoding + state type. +</para></listitem> + +<listitem><para> + Conversion descriptors for both directions of encoding. (i.e., both + UCS-2 to UTF-8 and UTF-8 to UCS-2.) +</para></listitem> + +<listitem><para> + Something to indicate if the conversion requested if valid. +</para></listitem> + +<listitem><para> + Something to represent if the conversion descriptors are valid. +</para></listitem> + +<listitem><para> + Some way to enforce strict type checking on the internal and + external types. As part of this, the size of the internal and + external types will need to be known. +</para></listitem> +</itemizedlist> +</sect3> + +<sect3 id="codecvt.design.issues" xreflabel="codecvt.design.issues"> + <title>Other Issues</title> +<para> +In addition, multi-threaded and multi-locale environments also impact +the design and requirements for code conversions. In particular, they +affect the required specialization codecvt<wchar_t, char, mbstate_t> +when implemented using standard "C" functions. +</para> + +<para> +Three problems arise, one big, one of medium importance, and one small. +</para> + +<para> +First, the small: mcsrtombs and wcsrtombs may not be multithread-safe +on all systems required by the GNU tools. For GNU/Linux and glibc, +this is not an issue. +</para> + +<para> +Of medium concern, in the grand scope of things, is that the functions +used to implement this specialization work on null-terminated +strings. Buffers, especially file buffers, may not be null-terminated, +thus giving conversions that end prematurely or are otherwise +incorrect. Yikes! +</para> + +<para> +The last, and fundamental problem, is the assumption of a global +locale for all the "C" functions referenced above. For something like +C++ iostreams (where codecvt is explicitly used) the notion of +multiple locales is fundamental. In practice, most users may not run +into this limitation. However, as a quality of implementation issue, +the GNU C++ library would like to offer a solution that allows +multiple locales and or simultaneous usage with computationally +correct results. In short, libstdc++ is trying to offer, as an +option, a high-quality implementation, damn the additional complexity! +</para> + +<para> +For the required specialization codecvt<wchar_t, char, mbstate_t> , +conversions are made between the internal character set (always UCS4 +on GNU/Linux) and whatever the currently selected locale for the +LC_CTYPE category implements. +</para> + +</sect3> + +</sect2> + +<sect2 id="facet.codecvt.impl" xreflabel="facet.codecvt.impl"> +<title>Implementation</title> + +<para> +The two required specializations are implemented as follows: +</para> + +<para> +<code> +codecvt<char, char, mbstate_t> +</code> +</para> +<para> +This is a degenerate (i.e., does nothing) specialization. Implementing +this was a piece of cake. +</para> + +<para> +<code> +codecvt<char, wchar_t, mbstate_t> +</code> +</para> + +<para> +This specialization, by specifying all the template parameters, pretty +much ties the hands of implementors. As such, the implementation is +straightforward, involving mcsrtombs for the conversions between char +to wchar_t and wcsrtombs for conversions between wchar_t and char. +</para> + +<para> +Neither of these two required specializations deals with Unicode +characters. As such, libstdc++ implements a partial specialization +of the codecvt class with and iconv wrapper class, encoding_state as the +third template parameter. +</para> + +<para> +This implementation should be standards conformant. First of all, the +standard explicitly points out that instantiations on the third +template parameter, stateT, are the proper way to implement +non-required conversions. Second of all, the standard says (in Chapter +17) that partial specializations of required classes are a-ok. Third +of all, the requirements for the stateT type elsewhere in the standard +(see 21.1.2 traits typedefs) only indicate that this type be copy +constructible. +</para> + +<para> +As such, the type encoding_state is defined as a non-templatized, POD +type to be used as the third type of a codecvt instantiation. This +type is just a wrapper class for iconv, and provides an easy interface +to iconv functionality. +</para> + +<para> +There are two constructors for encoding_state: +</para> + +<para> +<code> +encoding_state() : __in_desc(0), __out_desc(0) +</code> +</para> +<para> +This default constructor sets the internal encoding to some default +(currently UCS4) and the external encoding to whatever is returned by +nl_langinfo(CODESET). +</para> + +<para> +<code> +encoding_state(const char* __int, const char* __ext) +</code> +</para> + +<para> +This constructor takes as parameters string literals that indicate the +desired internal and external encoding. There are no defaults for +either argument. +</para> + +<para> +One of the issues with iconv is that the string literals identifying +conversions are not standardized. Because of this, the thought of +mandating and or enforcing some set of pre-determined valid +identifiers seems iffy: thus, a more practical (and non-migraine +inducing) strategy was implemented: end-users can specify any string +(subject to a pre-determined length qualifier, currently 32 bytes) for +encodings. It is up to the user to make sure that these strings are +valid on the target system. +</para> + +<para> +<code> +void +_M_init() +</code> +</para> +<para> +Strangely enough, this member function attempts to open conversion +descriptors for a given encoding_state object. If the conversion +descriptors are not valid, the conversion descriptors returned will +not be valid and the resulting calls to the codecvt conversion +functions will return error. +</para> + +<para> +<code> +bool +_M_good() +</code> +</para> + +<para> +Provides a way to see if the given encoding_state object has been +properly initialized. If the string literals describing the desired +internal and external encoding are not valid, initialization will +fail, and this will return false. If the internal and external +encodings are valid, but iconv_open could not allocate conversion +descriptors, this will also return false. Otherwise, the object is +ready to convert and will return true. +</para> + +<para> +<code> +encoding_state(const encoding_state&) +</code> +</para> + +<para> +As iconv allocates memory and sets up conversion descriptors, the copy +constructor can only copy the member data pertaining to the internal +and external code conversions, and not the conversion descriptors +themselves. +</para> + +<para> +Definitions for all the required codecvt member functions are provided +for this specialization, and usage of codecvt<internal character type, +external character type, encoding_state> is consistent with other +codecvt usage. +</para> + +</sect2> + +<sect2 id="facet.codecvt.use" xreflabel="facet.codecvt.use"> +<title>Use</title> +<para>A conversions involving string literal.</para> + +<programlisting> + typedef codecvt_base::result result; + typedef unsigned short unicode_t; + typedef unicode_t int_type; + typedef char ext_type; + typedef encoding_state state_type; + typedef codecvt<int_type, ext_type, state_type> unicode_codecvt; + + const ext_type* e_lit = "black pearl jasmine tea"; + int size = strlen(e_lit); + int_type i_lit_base[24] = + { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184, + 27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696, + 25856, 24832, 2560 + }; + const int_type* i_lit = i_lit_base; + const ext_type* efrom_next; + const int_type* ifrom_next; + ext_type* e_arr = new ext_type[size + 1]; + ext_type* eto_next; + int_type* i_arr = new int_type[size + 1]; + int_type* ito_next; + + // construct a locale object with the specialized facet. + locale loc(locale::classic(), new unicode_codecvt); + // sanity check the constructed locale has the specialized facet. + VERIFY( has_facet<unicode_codecvt>(loc) ); + const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc); + // convert between const char* and unicode strings + unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1"); + initialize_state(state01); + result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next, + i_arr, i_arr + size, ito_next); + VERIFY( r1 == codecvt_base::ok ); + VERIFY( !int_traits::compare(i_arr, i_lit, size) ); + VERIFY( efrom_next == e_lit + size ); + VERIFY( ito_next == i_arr + size ); +</programlisting> + +</sect2> + +<sect2 id="facet.codecvt.future" xreflabel="facet.codecvt.future"> +<title>Future</title> +<itemizedlist> +<listitem> + <para> + a. things that are sketchy, or remain unimplemented: + do_encoding, max_length and length member functions + are only weakly implemented. I have no idea how to do + this correctly, and in a generic manner. Nathan? +</para> +</listitem> + +<listitem> + <para> + b. conversions involving std::string + </para> + <itemizedlist> + <listitem><para> + how should operators != and == work for string of + different/same encoding? + </para></listitem> + + <listitem><para> + what is equal? A byte by byte comparison or an + encoding then byte comparison? + </para></listitem> + + <listitem><para> + conversions between narrow, wide, and unicode strings + </para></listitem> + </itemizedlist> +</listitem> +<listitem><para> + c. conversions involving std::filebuf and std::ostream +</para> + <itemizedlist> + <listitem><para> + how to initialize the state object in a + standards-conformant manner? + </para></listitem> + + <listitem><para> + how to synchronize the "C" and "C++" + conversion information? + </para></listitem> + + <listitem><para> + wchar_t/char internal buffers and conversions between + internal/external buffers? + </para></listitem> + </itemizedlist> +</listitem> +</itemizedlist> +</sect2> + + +<bibliography id="facet.codecvt.biblio" xreflabel="facet.codecvt.biblio"> +<title>Bibliography</title> + + <biblioentry> + <title> + The GNU C Library + </title> + + <author> + <surname>McGrath</surname> + <firstname>Roland</firstname> + </author> + <author> + <surname>Drepper</surname> + <firstname>Ulrich</firstname> + </author> + + <copyright> + <year>2007</year> + <holder>FSF</holder> + </copyright> + <pagenums>Chapters 6 Character Set Handling and 7 Locales and Internationalization</pagenums> + + </biblioentry> + + <biblioentry> + <title> + Correspondence + </title> + + <author> + <surname>Drepper</surname> + <firstname>Ulrich</firstname> + </author> + + <copyright> + <year>2002</year> + <holder></holder> + </copyright> + </biblioentry> + + <biblioentry> + <title> + ISO/IEC 14882:1998 Programming languages - C++ + </title> + + <copyright> + <year>1998</year> + <holder>ISO</holder> + </copyright> + </biblioentry> + + <biblioentry> + <title> + ISO/IEC 9899:1999 Programming languages - C + </title> + + <copyright> + <year>1999</year> + <holder>ISO</holder> + </copyright> + </biblioentry> + + <biblioentry> + <title> + System Interface Definitions, Issue 6 (IEEE Std. 1003.1-200x) + </title> + + <copyright> + <year>1999</year> + <holder> + The Open Group/The Institute of Electrical and Electronics Engineers, Inc.</holder> + </copyright> + + <biblioid> + <ulink url="http://www.opennc.org/austin/docreg.html"> + </ulink> + </biblioid> + + </biblioentry> + + <biblioentry> + <title> + The C++ Programming Language, Special Edition + </title> + + <author> + <surname>Stroustrup</surname> + <firstname>Bjarne</firstname> + </author> + + <copyright> + <year>2000</year> + <holder>Addison Wesley, Inc.</holder> + </copyright> + <pagenums>Appendix D</pagenums> + + <publisher> + <publishername> + Addison Wesley + </publishername> + </publisher> + + </biblioentry> + + + <biblioentry> + <title> + Standard C++ IOStreams and Locales + </title> + <subtitle> + Advanced Programmer's Guide and Reference + </subtitle> + + <author> + <surname>Langer</surname> + <firstname>Angelika</firstname> + </author> + + <author> + <surname>Kreft</surname> + <firstname>Klaus</firstname> + </author> + + <copyright> + <year>2000</year> + <holder>Addison Wesley Longman, Inc.</holder> + </copyright> + + <publisher> + <publishername> + Addison Wesley Longman + </publishername> + </publisher> + + </biblioentry> + + <biblioentry> + <title> + A brief description of Normative Addendum 1 + </title> + + <author> + <surname>Feather</surname> + <firstname>Clive</firstname> + </author> + + <pagenums>Extended Character Sets</pagenums> + <biblioid> + <ulink url="http://www.lysator.liu.se/c/na1.html"> + </ulink> + </biblioid> + + </biblioentry> + + <biblioentry> + <title> + The Unicode HOWTO + </title> + + <author> + <surname>Haible</surname> + <firstname>Bruno</firstname> + </author> + + <biblioid> + <ulink url="ftp://ftp.ilog.fr/pub/Users/haible/utf8/Unicode-HOWTO.html"> + </ulink> + </biblioid> + + </biblioentry> + + <biblioentry> + <title> + UTF-8 and Unicode FAQ for Unix/Linux + </title> + + <author> + <surname>Khun</surname> + <firstname>Markus</firstname> + </author> + + <biblioid> + <ulink url="http://www.cl.cam.ac.uk/~mgk25/unicode.html"> + </ulink> + </biblioid> + + </biblioentry> + + +</bibliography> + +</sect1> |