Several bugs 1. RMM method won't work if there is an english char in it. It will completely mess up. 2. RMM did not segment the the words correctly in some situation e.g. 一一对应 is a word but 二一对应 is not. previously it would do 二 一对应 instead of 二 一 对应. 3. if a single char is at index 0 and it is not in the dictionary, and the second (index 1) char can be a suffix, the first term will be missing. e.g. い那 will be segmented to 那 because the first character is not in dictionary. 4. English char and Arabic numbers would mess up the entire segmentation. Note: If this preg appears in other similiar languages, it needs to be fixed. Tokenizer::segment() - ['/\d+/', '/[a-zA-Z]+/']); + ['/^\d+$/', '/^[a-zA-Z]+$/']);

Edit 1: my2.patch fixes all the bugs stated above I noticed there was a spacing problem after this patch was applied. (sometimes there are two spaces between each word, it is already fixed in my current copy but I haven't patched it) So I will make a patch 3, which fix this issue and make some updates to segmentation as well.

Additional information: So, I didn't fix the segment_word_grams.ftr because of the other bug I noticed and stated in your email. Please take a look.

Hi Forrest,

I tried to apply your patch but it didn't apply. I hadn't edited any files related to your patch recently, so I wasn't sure what was up. Maybe do a fresh clone and remake? Also, when you upload the patch can you state. "This patch fixes ..." From your description above I couldn't tell if you were reporting bugs or fixing them.

Best, Chris

Hi Professor Pollett, I think you've already applied the patch right after two days I posted this issue? And, Merry Christmas!

Best, Forrest

Here's git log:

commit 4072f99db9e795a624faed51bab3900a06054716 Author: Forrest Sun <sxh19911230@gmail.com> Date: Sat Dec 14 16:33:26 2019 -0800

   fix a minor bug

commit c15dfc5c648db2210a80db91f80189f3d5379243 Merge: 84ec6e0c ccae62a1 Author: Forrest Sun <sxh19911230@gmail.com> Date: Sat Dec 14 16:07:36 2019 -0800

   merge to previous work r=chris

commit 84ec6e0c0b612be69b7c3357ed8275b195f0b569 Author: Forrest Sun <sxh19911230@gmail.com> Date: Sat Dec 14 16:01:17 2019 -0800

   fix RMM bugs; r=chris

Applied in Dec 2019

Attachments:

my-2.patch