convert(1) command can't handle multibyte characters

Post any defects you find in the released or beta versions of the ImageMagick software here. Include the ImageMagick version, OS, and any command-line required to reproduce the problem. Got a patch for a bug? Post it here.
Locked
Sonam
Posts: 4
Joined: 2013-04-04T00:03:24-07:00
Authentication code: 6789

convert(1) command can't handle multibyte characters

Post by Sonam »

Hi,

convert(1) command can't handle multibyte characters properly and results the garbled characters.

It's easy to reproduce.. Create a text file which contains multibyte characters and process it with convert(1) as follows.

$ convert label:@test.txt test.png
convert: FreeType library is not available
`/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available
`/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
:
$

You will see the garbled characters in test.png.

$ cat text.txt
ÆüËܸì

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: convert(1) command can't handle multibyte characters

Post by fmw42 »

convert: FreeType library is not available
Looks like you need to install the FreeType libraray. See http://www.imagemagick.org/download/delegates/

Sonam
Posts: 4
Joined: 2013-04-04T00:03:24-07:00
Authentication code: 6789

Re: convert(1) command can't handle multibyte characters

Post by Sonam »

Freetype Library is already installed on the system.

bash-3.2# pkginfo | grep freetype
system SUNWfreetype2 FreeType2 Font library

bash-3.2# /usr/sfw/bin/convert label:"HELLO" 1.png
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
bash-3.2#

The above command creates a correct png file despite the warning messages.

The issue is only when multibyte characters are used.

This error is seen with both 2byte char(Japanese) is in the text file and 1byte char(alphabet) is in the text file.

convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.

although there is this error, the text file successfully gets converted to png when the text file contains 1byte char(alphabet)

Regards,
Sonam

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: convert(1) command can't handle multibyte characters

Post by fmw42 »

I don't know much about Freetype to comment. But if it is not installed in a place where IM can find it, then IM won't recognize that it is there. Perhaps it is not an acceptable version? Just some things to check.

Sonam
Posts: 4
Joined: 2013-04-04T00:03:24-07:00
Authentication code: 6789

Re: convert(1) command can't handle multibyte characters

Post by Sonam »

What should be the correct location of the freetype fonts for IM to work properly ?

Important thing to notice here is that the text file successfully gets converted to png when the text file contains 1byte char(alphabet).
It's not working only with 2byte char(Japanese).

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: convert(1) command can't handle multibyte characters

Post by fmw42 »

Sonam wrote:What should be the correct location of the freetype fonts for IM to work properly ?
Probably the same locations as where you have installed IM.

Perhaps it is sufficient to be in some location that is in your $PATH. I am not really sure.

gmr
Posts: 8
Joined: 2013-07-15T02:54:33-07:00
Authentication code: 6789

Re: convert(1) command can't handle multibyte characters

Post by gmr »

Any ideas on the issue. The issue is the same as Sonam had mentioned.

bash-3.2# /usr/sfw/bin/convert label:@text_UTF-8.txt out.png
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
bash-3.2# ls /usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z
/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z
bash-3.2#

So the file exists on the system, even though the error says the freetype library is not available.

So to put a straight question, does imagemagick support multi-byte character conversion?
If yes, can I get pointers on what to look for regarding the issue?

Thanks

Jason S
Posts: 103
Joined: 2010-12-14T19:42:12-07:00
Authentication code: 8675308

Re: convert(1) command can't handle multibyte characters

Post by Jason S »

gmr wrote:does imagemagick support multi-byte character conversion?
Testing ImageMagick 6.8.6-5 on Linux...

Code: Select all

$ echo -e 'X\xe2\x98\xbaX' > label.txt
$ convert label:@label.txt test.png
My result: X☺X
(I do not get any warnings about FreeType.)

So, it interpreted the label file as UTF-8, and the non-Latin character was correctly processed. This confirms that IM does support multi-byte UTF-8 characters, at least in some situations.

Code: Select all

$ echo -e 'X\xa9X' > label.txt
$ convert label:@label.txt test.png
My result: X©X

This file is not valid UTF-8, and IM interprets it as Latin-1 or similar. It must be doing some autodetection of the encoding.

Code: Select all

$ echo -e 'X\xe4\xb8\x8eX' > label.txt
$ convert label:@label.txt test.png
My result: X?X

This is a CJK character, encoded in UTF-8. Since it was displayed as a single question mark instead of three garbage characters, I assume this is a limitation of the font that was used, and not a problem with interpreting the UTF-8 encoding.

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: convert(1) command can't handle multibyte characters

Post by fmw42 »

gmr wrote:Any ideas on the issue. The issue is the same as Sonam had mentioned.

bash-3.2# /usr/sfw/bin/convert label:@text_UTF-8.txt out.png
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
convert: FreeType library is not available `/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z'.
bash-3.2# ls /usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z
/usr/openwin/lib/X11/fonts/TTbitmaps/Arial-Bold14.pcf.Z
bash-3.2#

So the file exists on the system, even though the error says the freetype library is not available.

So to put a straight question, does imagemagick support multi-byte character conversion?
If yes, can I get pointers on what to look for regarding the issue?

Thanks
If freetype exists, it is not where IM expects to find it. Perhaps your PATH does not point to that location.

To check about IM, on current versions use

convert -version

and see if freetype is listed as delegates

On older systems, type

convert -list configure

look at the line starting with DELEGATES and see if it is listed there.

gmr
Posts: 8
Joined: 2013-07-15T02:54:33-07:00
Authentication code: 6789

Re: convert(1) command can't handle multibyte characters

Post by gmr »

The running version of imagemagick is 6.8.3-5. "convert -version" lists freetype in the delegates list, as below.
Features: DPC Modules
Delegates: bzlib fftw freetype jng jpeg lcms lzma openexr pango png ps tiff x xml zlib
I also tried the given examples in the previous update ... but I was not able to convert some of them. The copyright symbol was converted correctly. But none of the other characters were converted correctly.

Code: Select all

$ echo -e 'X\xa9X' > label.txt
$ convert label:@label.txt test.png


My result is also as expected : X©X. But, where as the following convert did not yield the result as expected. I got the conversion text as a question mark.

Code: Select all

$ echo -e 'X\xe2\x98\xbaX' > label.txt
$ convert label:@label.txt test.png
Any thoughts on where might be the issue ?? There was a mention of the fonts. How do I check on whether the available fonts are an issue?

Thanks for the help!
gmr!

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: convert(1) command can't handle multibyte characters

Post by fmw42 »

convert label:@label.txt test.png
The text file must support UTF8.

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: convert(1) command can't handle multibyte characters

Post by magick »

And the font must support the Unicode character set such as Arial Unicode, and extended MS font.

Jason S
Posts: 103
Joined: 2010-12-14T19:42:12-07:00
Authentication code: 8675308

Re: convert(1) command can't handle multibyte characters

Post by Jason S »

Use the -font and -pointsize options to select a font that contains the characters you need. "convert -list font" will list the known fonts.

gmr
Posts: 8
Joined: 2013-07-15T02:54:33-07:00
Authentication code: 6789

Re: convert(1) command can't handle multibyte characters

Post by gmr »

Updating to have the closure remarks.

Thanks. Having explicitly pointing to the font file, did the trick.

Locked