Normalizing Chinese is to convert every single chinese character to simplified. Don't use it as a converting feature because it will convert the word wrongly. For example, 乾隆 will be converted to 干隆. In this case, it is ok to insert 干隆 into index for searching, but 干隆 is no longer a valid Chinese word.

Hey Forrest,

Thanks for your patch! I've applied it, however, the patch did not include your updated Tokenizer.php for Chinese with the normalize method. Can you uploade that separately?

Best,

Chris

my2 is the combined and myseperate is just the last patch.

Best, Forrest

Hey Forrest,

I modified your normalize code and have just pushed the result. I added a unit test to ZhTokenizerTest.php . I also tested if from the command line. Let me know if you spot anything.

Best, Chris

Hi Professor Pollett,

I did not test on it, but it looks good.

Best, Forrest

This has been in Yioop for a while, I forgot to close the issue.

Best, Chris

Attachments:

my.patch

my2.patch

myseperate.patch