Comments on: Writing games in Korean – part 3

By: adam

adam — Sat, 03 May 2008 23:38:29 +0000

Thanks, guys, for the good advice!

@dom: of course, as you say, the parser can’t even tell by that point. I keep foolishly half-expecting stream objects in java to reveal the history of their encoding(s) via runtime methods, but they don’t. It would help with debugging and with checking incoming data from external sources. I think there’s a bit too much transparecny here – since character data is almost certainly a point of interface with external systems, more debug info would be good.

By: Dominic Mitchell

Dominic Mitchell — Sat, 03 May 2008 18:43:11 +0000

I’ve learned the hard way that any time you say “read a text file” and don’t explicitly say what encoding to use, you’re in for trouble sooner or later. Unfortunately, FileReader doesn’t appear to support that, so you have to be a bit more explicit. As elFarto mentions above, you need to wrap a FileInputStream using InputStreamReader, and specify the encoding. Then you need to wrap the whole lot in a BufferedReader to get any performance out of it. It’s a pretty nasty concoction, but hey, it’s flexible.

new BufferedReader(new InputStreamReader(new FileInputStream("someFile"), "UTF-8"));

The other thing about the xml parser not complaining that the encoding doesn’t match: well, it can’t. Because you’ve provided a Reader, instead of an InputStream to parse, you are sending characters not bytes into the parser. It can’t know anything about the encoding because the transformation from bytes to characters has already occurred. Whereas if you pass an InputStream, the whole “detect the encoding” work of the XML parser takes place as expected. Like elFarto says, using an InputStream is probably a better idea here. And UTF-8? Well, if you’re dealing exclusively with Korean characters, you may wish to consider UTF-16 for space reasons.

By: elFarto

elFarto — Wed, 23 Apr 2008 21:32:24 +0000

new FileReader(file)

is identical to doing:

new InputStreamReader(new FileInputStream(file))

But this will attempt to use the platform default encoding to read the file (which is generally CP-1252 on Windows). Always use InputStreams when reading XML, as XML comes with it’s own method for determining character encoding, which Readers don’t understand or follow.

If you want a buffered FileInputStream, just wrap it in a BufferedInputStream.

I’ve had “Content is not allowed in prolog.” when I was attempting to use Notepad to create a UTF-8 XML file. Notepad helpfully prepends a BOM (Byte Ordering Mark) to the beginning, which isn’t allowed in XML.

I just did a quick test in Eclipse, and it’s magic automatically re-encoding the file feature is broken for XML. The bit must be the first thing in the file, and it must be in ASCII. When selecting UTF-16 it encodes the whole file, which is wrong.

Use UTF-8 and InputStreams and you should be fine.

Regards
elFarto