Chinese Word Segmentation Evaluation Toolkit

By Joy (joy@cs.cmu.edu). Jan 2004

• Introduction

• Evaluation method and the data set

• Evaluation script

• Submit segmented testing data for evaluation

• Word segmenters and their performances

• Related research: First International Chinese Word Segmentation Bakeoff 

Introduction

Unlike most western languages, written Chinese text has no separators (e.g. empty spaces) between words. If this is the case in English, you will read a sentence like this: itisveryimportanttohaveagoodwordsegmenter. Native speakers usually have no trouble reading the sentence without any separators. But most language processing technologies can't handle it. Language technology applications such as Machine Translation and Information Retrieval, use word as the basic units. Segmenting the Chinese sentence into a sequence of words is   So it is very important to have a good word segmenter. 

Word segmentation is a non-trivial task, and it is hard to have a "good" segmenter. It is almost impossible to segment a sentence perfectly. In fact even human has trouble to segment some ambiguous sentences. There are quite some word segmentation approaches: frequency-based, rule-based, maximum-entropy based approaches, just to name a few. To find the "best" segmenter for the NLP application, we need to evaluate the performance of the segmenter.

Evaluation method and the data set

The evaluation method used here is the Edit Distance of the Word Separator (EDWS).

Given a Chinese sentence without segmentation, we can represent it as a sequence of characters: C₁C₂....C_iC_i+1...C_n. 
A segmenter segments this sentence into a sequence of words by inserting separation marks S between characters. For example, a segmented sentence may look like: C₁C₂SC₃C₄C₅SC₆C₇C₈C₉SC₁₀S....C_iC_i+1...SC_n. Two segmentations may have separation marks inserted at different positions in the sentence. The Edit Distance of the Word Separator (EDWS) thus measures how many edit operations (insertion, deletion and substitution) are needed to modify one segmentation to the standard segmentation (reference).

Example:

       Standard(reference):  C₁C₂SC₃C₄C₅SC₆C₇   C₈C₉SC₁₀SC₁₁C₁₂C₁₃SC₁₄
         from one segmenter:   C₁C₂SC₃C₄C₅SC₆C₇SC₈C₉SC₁₀   C₁₁C₁₂C₁₃SC₁₄ 

In this example there are     substitutions after C₂, C₅, C₉ and C₁₃; insertions after C₁₀; deletions after C₇

We can define the segmentation precision, recall and harmonic mean F as:

Prec= (number of sub) /(number of separators in Hyp)
Recall=(number of sub)/(number of separators in Ref)
F=2*Prec*Recall/(Prec+Recall)

Depends on the application, one can derive other metrics based on the number of substitutions, insertions and deletions. For example, if over-segmentation is more problematic, more penalty can be assigned to deletions.

The data set we used here are extracted from the Chinese Treebank, Out of 1715 sentences which have length between 10~50 words, we randomly selected about half of the sentences as the Development set (845 sentences) and the rest as the testing set (870 sentences). You can tune your segmenter (if it is tunable) on the dev-set. The test-set should not be touched for tuning purposes.

To make the reference set more robust, we used three word segmenters (lrsegmenter, GBMMseg and CTseg) to segment the dev/tst data and created "artificial" references. Plus the human segmented treebank data, we have 4 "reliable" segmentations as reference. The intersection of the 4 segmentations, i.e. insert a separation mark between two characters only when all 4 segmentation agree, is used to measure the optimal Recall. The union of the 4 segmentation, i.e. insert a separation mark when any of the 4 segmentation has it, is used to measure the optimal Precision.

Evaluation scripts

Download the evaluation script (perl5) here

To run the evaluation: 

Step1: download the unsegmented data (dev.gb or tst.gb)
Step2: use your segmenter to segment it, make sure that the segmented data has the same number of lines as the unsegmented data. Suppose the segmented data is called hyp.gb
Step3: run the following: 
        perl5 segmentationPrec.pl dev-ref.gb hyp.gb

The result looks like this:

    Total: Sub=21341
    Ins=3142
    Del=807
    Precision: 84.39%

Submit segmented testing data for evaluation

Step1: download the testing data (tst.gb)
Step2: use your segmenter to segment it,  suppose the segmented file is called hyp.gb
Step3: upload your results and fill in the information to publish your results

Word segmenters and their performance (on test set)