diff options
| author | Tom N Harris <tnharris@whoopdedo.org> | 2010-11-16 18:09:53 -0500 |
|---|---|---|
| committer | Tom N Harris <tnharris@whoopdedo.org> | 2010-11-16 18:09:53 -0500 |
| commit | 1c07b9e622d139fa815c955c89569f96342475fb (patch) | |
| tree | 08b1d84b5d1fa7c3b1b22c89a9be6efd3e543704 /lib/exe | |
| parent | 6c528220aaf62f4ba5890483797d6661352500bb (diff) | |
| download | rpg-1c07b9e622d139fa815c955c89569f96342475fb.tar.gz rpg-1c07b9e622d139fa815c955c89569f96342475fb.tar.bz2 | |
Use external program to split pages into words
An external tokenizer inserts extra spaces to mark words in the input text.
The text is sent through STDIN and STDOUT file handles.
A good choice for Chinese and Japanese is MeCab.
http://sourceforge.net/projects/mecab/
With the command line 'mecab -O wakati'
Diffstat (limited to 'lib/exe')
0 files changed, 0 insertions, 0 deletions