Right now the inverted index is split into partitions (250) by word. It is further split by generation. The first 250,000 documents/micro documents map to the first generation, the next 250,00 go to the second generation, etc. Within a generation, within a partition word data is stored in a web archive with a complicated header format. The goal of this feature is to try to store data for a single generation, partition pair into a single 1.5Mb file which is read/written out in one go. The format would look like:

word_key1 (8bytes):4bytes num documents 4bytes doc_ref1+rel_score1 ...4bytes docref_n+rel_scoren ... word_key_n ....

doc_1 (sorted by key) ... doc_m

A doc_ref with value 5 would refer to doc_5.

Fixed with commit dfde58ae, cpollett

As the earlier note said, this was fixed with dfde58ae. The index isn't yet as compressed as the original description hoped, but now there is a class IndexShard and it has the basic structure described in that note.