Knock out post. Keep helping me understand this world.

Stephen Arnold, October 2, 2008

1. Agreed. Multi-core isn’t going to solve the I/O bandwidth bottleneck for indexing applications. (But once the indexing is done, we can leverage it to solve the I/O bandwidth problem for large scale classification applications [ http://www.hpl.hp.com/techreports/2008/HPL-2008-29R1.html ])

2. If you’re concerned about random words colliding to the same hash value, then just increase the output hash width so collisions are arbitrarily rare. Google published their entire Web dictionary of word counts on a CD set [1TWeb] and it only has fewer than 14 million unique words that occur with any reasonable frequency. This number of tokens uses only 0.3% of the numbers in a 2**32 hash space, so collisions are pretty rare— probably less frequent than mispellings or synonyms that get in your way. But we can do better, if you like: since their average word length was 8.1 characters, we could use 8 byte hashes without using additional disk space. This results in 64-bit hashes, making collisions less frequent than 1 in 18.4 trillion.

3. The comparison with Lucene v2.2 just shows there’s room for improvement, not that Lucene v2.2 is particularly slow compared to other packages. If speed is paramount, then use SpeedyFx in concert with Lucene, i.e. just as another text analyzer.