Static Oneplus

New neural parser added to LTP

2015-05-20T16:00:00Z

We recently added a new dependency parser to LTP. This parser is a transiton-based neural network parser, which mainly based on Chen and Manning (2014). Besides the parser originally described in their work, additional features are also incorporated in our parser, including global feature (Zhang and Nivre, 2011), word cluster and dynamic oracle (Goldberg et. al. 2014). Dr. Jiang Guo developed the prototype for this parser in his ACL2015 work.

For quick summary, our new parser’s main features includes

Fast linear parsing time: over 8,000 tokens/second.
High parsing accuracy: 85.24 (UAS) on CTB5

and the key techniques includes

Linear time transition-based parsing.
Neural network classifier with cubic active function
Supporting clustering features and global features
Supporting learning from dynamic oracle

Word Cluster

Word cluster can be the most reliable feature acrossing the syntactic tasks, including POS tagging, NER and parsing. Thus, the performance improvement we gain from word cluster is as expected. In our system, word cluster is added to as another ‘POS tag’.

Global Feature

Zhang and Nivre (2012) study the problem of interaction between searching and learning. Their work can be summarized as

local learning + greedy search: [good]
local learning + beam search: [bad]
global learning + greedy search: [bad]
global learning + beam search: [good]

They also say that

local learning + global features: [bad]
global learning + global features: [good]

We also found that global features contribute nothing if they are added to the parser standalone. But we found that when coupled with word cluster, the global feature can further help. Thus, this feature is add in our released version.

Dynamic Oracle

Dynamic oracle is another reliable stuff! On the NLTK’s issues list, Dr. Honnibal (author of spaCy) quoted that

Second, when you train the parser, you parser, you should really use the Goldberg and Nivre (2012) “dynamic oracle” strategy.

We also found dynamic oracle works great on Chinese dataset. The detailed result will be presented in experiment section.

Experiment

We report our parser’s performance on Chinese treebank 5 and Chinese dependency treebank. Parsing accuray is evaluated using UAS/LAS. We also compare our result with the our old LTP parser and the other state-of-the-art parsers.

CTB5 experiment

CTB5 data is splited according to Zhang and Clark (2008), with 16,091 for training, 1,910 for testing and 803 for development. Word embedding trained on Gigawords xinhua news with word2vec is used as initialization embedding. Brown clustering result with 1000 clusters on the same data are also used in our experiment.

Parser	Dev.UAS	Dev.LAS	Test.UAS	Test.LAS	Test Speed(tokens/sec.)
ZPar (b=64)	85.15	85.45	85.82	84.54	(about-) 700
LTP (o2sib)	84.46	82.96	84.05	82.62	40.99(?)
NN (e=50,h=200)	82.87	81.13	83.22	81.68	16737.41
NN-nondet(e=50,h=200)	83.48	81.75	83.95	82.40
NN-explore(e=50,h=200)	84.42	82.68	84.44	82.70
NN-explore+glob.feat(e=50,h=200)	84.48	82.79	84.74	83.02	14627.79
NN-explore+cluster(e=50,h=200)	85.08	83.26	84.98	83.28	9734.88
NN-explore+glob.feat+cluster(e=50,h=200)	85.16	83.54	85.24	83.61	9325.72

CDT experiment

Suggested data split is used in CDT experiment. Embedding and cluster settings are almost identical to the CTB5 experiment except that data are segmented with a PKU standard segmentor.

Parser	Dev.UAS	Dev.LAS	Test.UAS	Test.LAS	Test Speed(tokens/sec.)
ZPar (b=64)	86.43	83.68	85.48	82.41
LTP (o2sib)	84.95	82.09	83.99	81.31	263.89
NN (e=50,h=400)	84.33	81.57	83.04	80.01	11478.48
NN-explore(e=50,h=400)	85.02	82.41	83.74	80.81
NN-explore+glob.feat(e=50,h=400)	84.81	82.23	83.56	80.65
NN-explore+cluster(e=50,h=400)	85.82	83.25	83.49	80.60
NN-explore+glob.feat+cluster(e=50,h=400)	85.80	83.19	84.11	81.17	8448.45

Generally speaking, the dynamic oracle shows to improve parsing accuracy on both dataset. What’s more, brown cluster is also a great feature to boost up the score.

“In a way to kill old people”

We can see the parsing accuracy and speed benchmark with our the old LTP parser in the following chart.

In a way to kill old people, as the phrase is.

Reference

Danqi Chen and Christopher Manning, 2014, A Fast and Accurate Dependency Parser using Neural Networks, In Proc. EMNLP2014
Yue Zhang and Joakim Nivre, 2011, Transition-based Dependency Parsing with Rich Non-local Features, In Proc ACL2011
Yoav Goldberg, Francesco Sartorioand Giorgio Satta, 2014, A Tabular Method for Dynamic Oracles in Transition-Based Parsing, In TACL2014
Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang and Ting Liu, 2015, Cross-lingual Dependency Parsing Based on Distributed Representations, (to apper) In Proc ACL2015
Yue Zhang and Joakim Nivre, 2012, Analyzing the Effect of Global Learning and Beam-Search on Transition-Based Dependency Parsing, In Proc COLING2012
Yue Zhang and Stephen Clark, A Tale of Two Parsers: Investigating and Combining Graph-based and Transition-based Dependency Parsing, In Proc ACL2008

ZGen v0.2.0 Released

2015-03-28T16:00:00Z

[Sourceforge Project Home] [Github Mirror]

I’ve been working on the ZGen project since Sept. the last year. It’s a linearization system that constructs natural language sentences from bags of words, given optional input syntactic constraints. Depending on the amount of input constraints, ZGen can perform free ordering, partial tree linearization and full tree linearization. It’s the system I used in my NAACL2015 paper: Transition-Based Syntactic Linearization. Any feedback is welcome!

Sentence generation is an interesting but less studied topic, partly because the complexity of the generation system. The current version of ZGen can never make up a generation system, but hopefully it can perform generation from logic form or key words or a sentence topic.

ZGen is named after the ZPar project from my supervisor and corporator Dr. Yue Zhang. He must be marching on the road of ZNLP :-)

A benchmark on mapping few keys

2015-03-09T16:00:00Z

Recently, I’ve worked on optimizing my transition-based parser and came across such situation:

I need an associated, or key-value, structure to store the cached scored for each transition action at a certain state. In practice, the number of transition actions is relatively small (less than 100), so I became curious about the best associated data structure for small set of keys.

Previous posts on such performance benchmark mainly focus on the comparsion of different hash maps on large scale, like this one. This post differs from theirs by setting the number of keys to a small number. Time consumption on random insertion, ordered insertion, random retrieve and ordered retrieve is evaluated. I didn’t give a try on the delete operation cause there is no such operation in my problems.

Three types of mapping facilities are adopted in this posts. They are:

Tree: std::map
Ordered List: boost::container::flat_map
Hashtable: std::unordered_map, boost::unordered_map, google::sparse_hash_map, google::dense_hash_map

std::map is a several-time loser in previous benchmark because of the O(log n) time complexity. I choose the std::map and hope it can win a round in this special situation. boost::container::flat_map is suggested by this StackOverflow question. It seems the key are ordered and stored in a list in boost::container::flat_map, rather than a tree like the std::map does. Expectingly, this data structure should have some nice memory allocation feature compared with the std::map. The Hashtable group is actually my old friends and I used google::dense_hash_map and std::unordered_map a lot.

Results

All the experiments are conducted on a Xeon(R) CPU E5-2620 @ 2.00GHz server. My GCC is a new-version one, the 4.8.2. The code used in this benchmark can be found at this gist. -O3 optimizing level is configured in compling the code.

The benchmark results are shown below.

Ordered insertion

number of keys

Random insertion

number of keys

Ordered queries

number of keys

Random queries

number of keys

Summary

Generally speaking, the google::dense_hash_map is still the best choice if you are aggressively purchasing the speed performance. Experimental result confirms that the dense_hash_map achieves both the fastest insertion and fastest querying.

Also, we see that on the condition that the number of keys is relatively small, the sorted group (std::map and boost::container::flat_map) performs better than the hash group. One thing we need to note that, after the number of key increase to about 100, the sorted group starts to perform poor compared to the hash group.

By comparing the std::map and boost::container::flat_map, their performances are almost identical. I am a little disappointed by this fact because I always considering the boost::container::flat_map as a silver bullet for the few key mapping problems.

For practice, there are several suggestion:

For the mapping problem with less than 100 keys, std::map is still a solution. It’s especially true if ordered keys is in your concern.
google::dense_hash_map is still number one choice, if memory and the annoying set_empty_key() are both not the problems.

Parallel and HPC with Python (or numpy)

2014-04-24T16:00:00Z

For guys working with natural language processing problems, it’s daily task to process tons of data. To handle the millions of lines of sentences, I would prefer C/C++ or Java in the past, especially at certain scenario like performing machine learning algorithm onto the data. However, in this days, I wrote a very slow python program (and working around numpy, it’s an important clue for future story). After wasting too much time on this single thread program, I decided to parallel it.

Buzz in the task

Let me briefly introduce my task (It’s usually important for choosing the appropriate parallel model). I have a collection of data which contains about 200 thousand entries. My algorithm is some kind of loop-loop and can be illustrated as the following pseudocode.

while not terminal-condition:
    init(global_vector)
    for instance in instances:
        global_vector += time-consuming-process(instance)
    do-something(global_vector)

Since the time-consuming-process is very time consuming, we can easily use a producer to distribute these tasks onto several consumers. What a textbook parallel model! To make it more clear, also for convenience of future discussion, let me put it into some meaningless but runnable code.

#!/usr/bin/env python
import numpy as np
import time

nr_instances = 2000
nr_dim = 257241

def do_something(vector):
    pass

def consumer():
    for _ in xrange(3):                 # use to simulate the time consuming
        x = np.random.rand(nr_dim)      # numpy array operation.
    ret = np.zeros(nr_dim)
    ret[np.random.randint(0, nr_dim, 20)] += np.random.rand(20)
    return ret

def producer():
    global_vector = np.zeros(nr_dim)
    instances = range(nr_instances)
    for index, instance in enumerate(instances):
        global_vector += consumer()
    do_something(global_vector)

if __name__=="__main__":
    for i in xrange(5):
        producer()
        print "iter %d is done." % i

A simple time command shows the above code runs 1m29.155s on my server.

Threading

As I mentioned before, I decide to paralled the above code. First thing that came into my mind is the threading. According to my past experience, multi-threaded programming is always the best choice when you have a server with several cores.

Distributing the producer’s to several thread can be painless done with python threading module. The producer’s job is dividing the instances into several groups, feed them to each of the thread and wait all these threads finish their work. A wrapper for the consumer is implemented for recieveing data and invoke meta-consumer process.

After a slight modification on the above program, it became the multi-threaded version.

#!/usr/bin/env python
import numpy as np
import time
import threading
from basic import consumer, do_something, nr_instances, nr_dim

nr_threads = 4

def consumer_wrapper(instances, results, index):
    global_vector= np.zeros(nr_dim)
    for instance in instances:
        global_vector += consumer()
    results[index] = global_vector

def producer():
    global_vector = np.zeros(nr_dim)
    instances = range(nr_instances)
    threads = [None] * nr_threads
    results = [None] * nr_threads
    fence = nr_instances / nr_threads
    for i in xrange(nr_threads):
        chunk = instances[i*fence: (L if i+1==nr_instances else i*fence+fence)]
        threads[i] = threading.Thread(target=consumer_wrapper, args=(chunk, results, i))
        threads[i].start()
    for i in xrange(nr_threads):
        threads[i].join()
    global_vector = sum(results)
    do_something(global_vector)

if __name__=="__main__":
    for i in xrange(5):
        producer()
        print "iter %d is done." % i

I was expecting that the multi-threaded version will bring 2 to 3 times speed up if I config the program with 4 threads. However, this code run 1m33.678s on the same server. I can’t even believe that a multi-threaded program runs slower than the single-threaded program.

After a survey on this issue, I found the answer. It suffer from the Python GIL which prevent the script running on two cores. There are lots of article talking about the GIL problems, so I won’t write more on this. The conclusion is that multi-threaded in Python doesn’t work for my task.

Multiprocessing

The failure of multi-threaded program drive me to seek for some alternative and I found the multiprocessing module. At the begining of its document page, it says,

effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads

To my understanding, the mechanism of multiprocessing module is treating the each thread as a process. When creating a thread, it actually copy(fork) the entire processing into a new process.

#!/usr/bin/env python
import numpy as np
import time
from multiprocessing import Pool
from basic import consumer, do_something, nr_dim, nr_instances

nr_threads = 4

def consumer_wrapper(instances):
    global_vector= np.zeros(nr_dim)
    for instance in instances:
        global_vector += consumer()
    return global_vector

def producer():
    instances = range(nr_instances)
    pool = Pool(processes = nr_threads)
    L = len(instances)
    fence = nr_instances / nr_threads
    arguments = [(instances[i*fence:(L if i+1==nr_threads else (i+1)*fence)]) \
            for i in xrange(nr_threads)]

    global_vector = sum(pool.map(consumer_wrapper, arguments))
    pool.close()
    pool.join()
    do_something(global_vector)

if __name__=="__main__":
    for i in xrange(5):
        producer()
        print "iter %d fin" % i

It takes 0m33.970s for the multiprocessing version to run. The multiprocessing module bring in about 2.5 times speed up.

However, one disappointing feature of multiprocessing is it copy entire program, therefore resulting in large memory consumption. This feature make it very unscalable if the single process version consume a lot of memory. In my experiments, my script consume 8G memory. If I apply it to 8 processors, the program explode into 64G (or more), almost reach the limit of the server.

MPI

The first time I meet MPI is that I read some source code of a machine-translation toolkit. The MPI module is embeded in a mess of C++ code and make it very difficult to understand.

Now it came to me again because the document page claims that

MPI is not an IEEE or ISO standard, but has in fact, become the “industry standard” for writing message passing programs on HPC platforms.

My supervisor also endorse it. It seems a widely used library for parallel programming. And its Python embedding also makes it less painful (or painless) to use.

In MPI, the producer-consumer model can be very clearly implemented by letting the zero-ranked (or master) program distribute the instances (or tasks), keep recieveing data from consumer. Running status of the consumers can be easily obtain by check the tag field of the recieved data.

Revisiting my problem, the MPI version is shown blow.

#!/usr/bin/env python
import numpy as np
import time
from mpi4py import MPI
from basic import consumer, do_something, nr_dim, nr_instances

READY, START, DONE, EXIT = 0, 1, 2, 3
comm = MPI.COMM_WORLD
size = comm.size
rank = comm.rank
status = MPI.Status()

def consumer_daemon():
    name = MPI.Get_processor_name()
    while True:
        comm.send(None, dest=0, tag=READY)
        task = comm.recv(source=0, tag=MPI.ANY_TAG, status=status)
        tag = status.Get_tag()

        if tag == START:
            instances = task
            global_vector = np.zeros(nr_dim)
            for instance in instances:
                global_vector += consumer()
            comm.send(global_vector, dest=0, tag=DONE)
        elif tag == EXIT:
            break
    comm.send(None, dest=0, tag=EXIT)

def producer():
    instances = range(nr_instances)
    L = len(instances)
    fence = L/ (size - 1)
    arguments = [(instances[i*fence:(L if i+1==size-1 else (i+1)*fence)]) for i in xrange(size - 1)]
    for i in xrange(1, size):
        comm.send(arguments[i-1], dest=i, tag=START)
    finished = 0
    global_vector = np.zeros(nr_dim)
    while finished < size - 1:
        data = comm.recv(source=MPI.ANY_SOURCE, tag=MPI.ANY_TAG, status=status)
        tag = status.Get_tag()
        if tag == DONE:
            global_vector += data
            finished += 1
    do_something(global_vector)

if __name__=="__main__":
    if rank == 0:
        for i in xrange(5):
            producer()
            print "iter %d is done." % i
        for i in xrange(1, size):
            comm.send(None, dest=i, tag=EXIT)
    else:
        consumer_daemon()

Under same settings, the above program runs 0m31.332s and it memory performance is better than the multiprocessing version. One reason for the faster speed, I think, is the consumer wasn’t shut down between each iterations.

Summary

When it comes to the topic of paralleling in Python, my advice is avoid using threading especially with some code that will trigger GIL. If the task only takes a little memory, my advice is the multiprocessing, because it’s easier to use and easier to switch from threading-oriented program. If you decide to make it a real paralleled program (aka, using it on multi-cores server or even across several servers), mpi4py is no doubt a better choice.

Reference

There are also some unlucky guys who found Python threading is slower: Python threading unexpectedly slower
Introduction to MPI : Message Passing Interface (MPI)
And the Python embedding : MPI4Py - SciPy
And some fancy examples! : jbornschein/mpi4py-examples

研一这一年吧

2013-08-04T17:07:11Z

给这一年列一个时间表吧，这篇文章想写很久了，虽然有预感写出来又是满满的负能量。

2012年

8月-9月

写了一篇叫《基于序列标注的中文分词、词性标注模型比较分析》，投了一个学生会议。大概是想论证用分类方法做分词、词性标注这些序列性问题可以取得与序列标注模型类似的性能。还想强调分类速度比较快。不过实验并没有获得符合预期的结果。所以这篇论文的论点比较奇怪，不管是写还是修改都很痛苦。在后来被转投中文信息学报，我又不得不痛苦地改了一遍。

9月-10月

参加了微博分词的评测。最后提交的系统是一个混了一大坨预处理的特征、半监督特征的CRF模型。除了把别人论证过的东西实现了一通后，几乎没有引入什么有新意的东西。由于官方没公布评测排名，并不知道自己的系统排名如何，但开会时统计了一下，大概是第二的位置。大概半年后，我review这个系统时发现当时使用crfsuite工具训练模型时没把负特征开关打开，所以最终结果应该是高于提交的系统的结果的。但主办方也没公布数据，也没法去做实验。

10月-11月

拿出不多的时间学习了机器学习，包括实现了一些简单的机器学习算法。重组织了博客。调研了一部分domain adaptation的论文，改之前的水文。还有一些杂七杂八的上课实验什么的。

11月-12月

写微博分词的评测报告，同时也看一段gibbs抽样。准备去上海开会的报告以及去开会，接着改论文投中文信息学报。

12月-1月

前半段考试，做了一坨课程实验，后半段去天津开了微博分词的会，其间都是准备poster一些杂事。这月后半段开始接了网管的工作。这其间还草草做了一个字聚类帮助分词的实验。后来这个非常烂的idea投了ccl2013。

2013年

1月-2月

前半段修硬盘来着。刚接手网管，实验室的磁盘阵列就挂了。大概原因就是raid5坏了两块后没人知道，第三块坏后就彻底挂了。大冬天抱着硬盘跑数据恢复，反正是非常苦逼。后来十天草草准备了一下托福，一战成了挂逼。

2月-3月

打算考G，利用过年时间背了一个多月单词，约了5月的G和6月的T。

3月-5月

开始调研用deep learning做分词，基本把rnnlm看了一遍。最初的思路是用语言模型做分词，结果做出来就让人没什么信心，后来又把思路换成用embedding做semi-supervised，也没什么信心。当时觉得主要问题在embedding结果对于分词模型不是线性作用的。也想过用c&w直接做一个分词的神经网络，不过后来看ccl好像有中科院的同学用实现了这个思路，不过效果似乎也让人没什么信心。不知道dl是不是不适合参加自然语言处理的battle against state-of-the-art。

断断续续地准备英语。

这两个月中做的另外一件事是机房上新设备。配上机器，电路改造，修空调，反正很少能安心下来看看书或者读读论文，跑实验也总出错。

5月-6月

面对一坨考试，终于撑不住了。取消的5月的G。不过考试还是成功考出两科60分。

6月-8月

在万念俱灰的情绪下，用一个星期整理了之前cluster帮助分词的工作。想论证怎样做字的表示才能帮助分词任务，结论是对单个字做不靠谱，对字聚类时要考虑字的上下文信息。这个水文投出去后重构了正华师兄的依存句法分析器dparser。当时估计工作量大概是一个月，1万行代码以下。但是实际做起来发现可以顺势将ltp里面的其他模块也重构一下。结果就是重写了本科毕设的序列标注统一框架，整个项目下来有1.7万行代码。还有9K字的文档，再加上写了python和ruby两个版本的client和一部分web页面。总之这两个月彻底做了一只代码狗。

6月末托福二战，再次准备十天，再次挂逼，考了个什么都不能做的93。基本是死了出国这条心了。再一件事是实验室网站被挂马，被网络中心关站了。没办法用静态页生成器重构了主页，迁移+升级了服务器，跑了几趟网络中心。现在还有一些服务没恢复，拙计。

这一年，基本上是在一种忙碌而压抑的状态中度过。没怎么过过周末，也没时间去运动。体重又回到了110斤以下，可悲的是肚子好像有胖的趋势。前个周末，爸妈来哈尔滨，幸亏没买到车票。因为那天综合楼停电，结果来电后机房就烧到50度了。若是他们来，恐怕只能把他们撂旅馆里一天了。

有的时候，我也不知道想过什么应该怎样生活。也没时间去想一想，反正一直被无形的力量推动，不由自主。

他们大四的毕业那会儿，我有几次被三点唱歌的醉汉们吵醒。只好去公寓平台的椅子里乘凉，有时看天色由黑转灰，继而一片青蓝，觉得很陌生。

脱臼

2013-07-21T04:28:17Z

昨天睡觉前，躺在床上打了个哈欠，结果一不小心下巴脱臼了。

下巴脱臼已经不是一次两次了。第一次脱臼好像是由于吃苹果张太大口，后来则以打哈欠打脱臼了居多。我把这种病症归咎于基因，因为印象中我妈好像也脱臼过。可能是骨骼的构造不合适，也可能是韧带的力度不够，不过这些一定是由基因决定的。由于接受了这种设定，我也就比较释然，重来没埋怨过苹果或者是哈欠。

出于多次脱臼的经验，我已经学会了一些基本的救治手段。一般是将下巴向左稍用力顶一下，感觉头骨错动，也可能会听到咯噔一声。如果运气好，脱臼就归位了。运气不好就多试几次，总有那么一次成功的。

我对自己的身体还有自己的技术都是有一定信心的，所以躺在床上左手扶头，右手扶下巴，向左用了一下力。可惜没出现咯噔一声。我知道我失败了，休息了几秒，换了个姿势，又向左使了一下力，当然还是失败。“或许是躺着不方便”，这么想后，我便坐了起来，哈尔滨夏夜的凉风慢悠悠地从窗子吹了进来。我额头上有汗，所以敏锐地捕捉到了这阵不期而遇的夏天的风。

坐起来后，我又重复了几次相同的自救策略，都以失败告终。好在我并没有着急，大学这几年唯一教会我的就是在着急之前先平静一下。平静的片刻之后，一大坨信息就冲入我的脑中：“首都机场好像发生爆炸案了”；“是不是城管最近又打死人了”；“托福口语如何提高成绩，真着急”；“昨天复华那边打架来着”；“不知道当年温岚有没有和周杰伦好过”；“女神建议我换个孙燕姿的桌面，要不要考虑一下”；“最近好像写了很多代码，都快成狗了”；“哎，活着真没意思”；“不过我还是不要告诉别人，免得看起来像个怨妇”。

在一阵激烈的对于宇宙终极问题的思考之后，我意识到“这次脱臼好像是自己弄不好了。”

而这时，寝室里另外一个哥们打起了呼噜，配着青黑天空中闪烁的繁星，显得非常有节奏。

大概又经历了一次尝试的失败，我放弃了治疗。小心翼翼穿上衣服。我要去医院，这件事不怎么严重，不必惊动其他人，我自己就够了。

“我是多么独立的一个人”，我这么想着瞄了一眼寝室的镜子，以确保自己脱臼的嘴脸在旁人看起来不那么可憎。

然后，一切妥当后，我拖着着这副夸张的对世界惊呆了的表情走入了孤独的夜色之中。