Yioop Repo - Seekquarry

« Back Issue #484

2020-07-21 05:57 Reported by xianghong
2022-03-27 18:54 Marked fixed

Status: Marked fixed

Priority: Medium

Normalize Chinese character for index, tagging, etc.

xianghong 2020-07-21 05:57

Normalizing Chinese is to convert every single chinese character to simplified. Don't use it as a converting feature because it will convert the word wrongly. For example, 乾隆 will be converted to 干隆. In this case, it is ok to insert 干隆 into index for searching, but 干隆 is no longer a valid Chinese word.

administrator 2020-07-21 11:40

Hey Forrest,

Thanks for your patch! I've applied it, however, the patch did not include your updated Tokenizer.php for Chinese with the normalize method. Can you uploade that separately?

Best,

Chris

xianghong 2020-07-21 15:28

my2 is the combined and myseperate is just the last patch.

Best, Forrest

administrator 2020-07-21 17:44

Hey Forrest,

I modified your normalize code and have just pushed the result. I added a unit test to ZhTokenizerTest.php . I also tested if from the command line. Let me know if you spot anything.

Best, Chris

xianghong 2020-07-21 19:17

Hi Professor Pollett,

I did not test on it, but it looks good.

Best, Forrest

administrator 2022-03-27 18:54

This has been in Yioop for a while, I forgot to close the issue.

Best, Chris

xianghong 2022-03-27 18:54

Attachments:

my.patch

my2.patch

myseperate.patch

- Yioop Repo - Seekquarry

Normalize Chinese character for index, tagging, etc.