Support more sophisticated querying

[SamHames]

The current implementation of the index is a boolean information retrieval system - documents are represented and indexed as a set of attributes, and can be recalled based on the attributes present in the document (the features). The base feature clustering algorithm can be interpreted within this framework as creating a series of disjunctive queries (feature1 OR feature2 OR feature2) for documents that share these features. This is conceptually and computationally simple to work with, but obviously falls short of the expressive power of even early relevance ranking search engines.

The question is, what can we do about this? There are a few different directions we can consider:

1. Support more boolean search operations (AND, OR, NOT and grouping), for the following purposes:
   1. Directly querying for specific documents (outside the context of the model)
   2. Creating new features from combinations of existing features
   3. Extend the cluster model, by allow modification of the set of terms and how they influence the document retrieved - instead of a topic being a cluster of features, a topic starts as a disjunction of features and can be incrementally modified to exclude or require certain features to be present
2. Relevance ranking: either by identifying existing approaches that work with the current binary model (preferred), or by incorporating new indexing functionality that better accounts for relevance ranking of features. This could be both in the form of general relevance ranking for arbitrary queries, or about relevance ranking specifically in the construction of the clusters/topics.
3. An interface for arbitrary querying/ranking? This might be useful if we want to explore dense representations of documents as well as, or instead of, the current sparse boolean model. Thinking out loud this could also work in conjunction with 1.iii