My passion is building crawlers and search engines. In particular, I specialize in building vertical search engines like Indeed.com, Homethinking.com, Bright.com and Enormo.com (all companies I've worked with). I've also worked on products such as Atlassian Jira and Confluence to improve their search capabilities. Kelvin has posted 22 posts at DZone. You can read more from them at their website. View Full User Profile

Solr Autocomplete with Document Suggestions

03.07.2012
| 7401 views |
  • submit to reddit

Solr 3.5 comes with a nice autocomplete/typeahead component that is based on the SolrSpellCheckComponent.

You provide it a query and a field, and the Suggester returns a list of suggestions based on the query. For example:

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <lst name="spellcheck">
    <lst name="suggestions">
      <lst name="ac">
        <int name="numFound">2</int>
        <int name="startOffset">0</int>
        <int name="endOffset">2</int>
        <arr name="suggestion">
          <str>acquire</str>
          <str>accommodate</str>
        </arr>
      </lst>
      <str name="collation">acquire</str>
    </lst>
  </lst>
</response>

Nice.

Now what if, as part of the autocomplete request, you needed a list of documents that contain the suggested terms for the given field? That's what I'm about to cover here.

TermDocs is your friend

The basic idea here is to call reader.termDocs() for each term, collect the document ids, and use that as the basis of a docslice. Here are relevant bits of code.

AND the doc ids for the various suggestions into a single docset.

NamedList spellcheck = (NamedList) rb.rsp.getValues().get("spellcheck");
NamedList suggestions = (NamedList) spellcheck.get("suggestions");
final SolrIndexReader reader = rb.req.getSearcher().getReader();
OpenBitSet docset = null;
for (int i = 0; i < suggestions.size(); ++i) {
  String name = suggestions.getName(i);
  if ("collation".equals(name)) continue;
  NamedList query = (NamedList) suggestions.getVal(i);
  Set<String> suggestion = (Set<String>) query.get("suggestion");

  OpenBitSet docs = collectDocs(field, reader, result);
  if (docset == null) docset = docs;
  else {
    docset.and(docs);
  }
}
 

collectDocs is implemented here:

private OpenBitSet collectDocs(String field, SolrIndexReader reader, Set<String> terms) throws IOException {
  OpenBitSet docset = new OpenBitSet();
  TermDocs te = reader.termDocs();
  for (String s : terms) {
    Term t = new Term(field, s);
    te.seek(t);
    while (te.next()) {
      docset.set(te.doc());
    }
  }
  te.close();
  return docset;
}

Now with the OpenBitSet of document ids matching the suggested terms, you can return a list of documents.

One problem is that you don't have document scores since no search was actually performed. Ideally, you'd want to return the documents in sorted by some field, and use the field value as the score.



Published at DZone with permission of its author, Kelvin Tan. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags: