Rafal Kuc is a team leader and software developer. Right now he is a software architect and Solr and Lucene specialist. Mainly focused on Java, but open on every tool and programming language that will make the achievement of his goal easier and faster. Rafal is also one of the founders of solr.pl site where he tries to share his knowledge and help people with their problems. Rafał is a DZone MVB and is not an employee of DZone and has posted 75 posts at DZone. You can read more from them at their website. View Full User Profile

Solr Autcomplete, part 4 (Ngram and faceting)

05.29.2012
| 6883 views |
  • submit to reddit

In the previous parts of autocomplete series we presented two methods of autocomplete queries. Than we extended one of those with the ability to define returned information. In today's entry we are back to autocomplete with facet and ngram.

Requirements

Our autocomplete mechanism has the following requirements:

  1. We return whole phrase, not just s single word
  2. Returned phrase can be present multiple times in the index
  3. We want to know the number of results for the returned phrase
  4. Common phrases should be shown higher than the less common ones
  5. Order of words entered by the user doesn’t matter

Solution

Solution given in the first part of the series will not met the requirements because of the first requirement. Of course we could change analysis type, but we wouldn’t return the whole phrase.

Solution to the above requirements is the modified faceting method. Instead of searching all the elements and narrowing results with facet.prefix parameter, we can search only for those elements that have the word fragment we are looking for. We don’t want wildcard query to be used (because of performance) we call ngram’s for the rescue. This means we need to write the ngrams into the index (of course Solr will do that for us). The obvious flaw is the index size growth, but in this case we can live with that.

Schema.xml

We define an additional type:

<fieldType name="text_autocomplete" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter minGramSize="1" maxGramSize="25" />
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

We also define additional fields: one which value we plan to return and one which will be used for searching:

<field name="tag_autocomplete" type="text_autocomplete" indexed="true" stored="true" omitNorms="true" omitTermFreqAndPositions="true"/>
<field name="tag" type="string" indexed="true" stored="true" />

And one copyField to make things easier:

<copyField source="tag" dest="tag_autocomplete"/>

Query

After indexation we are ready to test our queries:

  1. We are narrowing results, only to those which have the interesting word fragment in the tag_autocomplete field, with: q=tag_autocomplete:(PHRASE)
  2. We need all the fragments entered by the user, so we use AND as our logical operator: q.op=AND
  3. We not interested in the actual query results, we will use data returned by faceting, so we say: rows=0
  4. We need faceting: facet=true
  5. We need faceting on the field where we store the original phrase: facet.field=tag
  6. We are not interested in empty tags: facet.mincount=1
  7. We are only interested in 5 autocomplete values: facet.limit=5


And the final query:

?q=tag_autocomplete:(PHRASE)&q.op=AND&rows=0&facet=true&facet.field=tag&facet.mincount=1&facet.limit=5

If we will configure out search handler to include all the constant parameters, we will have the following query:

?q=tag_autocomplete:(PHRASE)

At the end

The basic virtue of the presented method is the ability to use one field for searching and other for returning results. Because of that, we were able to return the whole phrase instead of a single word.



Published at DZone with permission of Rafał Kuć, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Daniel Slazer replied on Tue, 2012/06/12 - 12:24pm

Hi,

I'm a newbie for java. I'm working on birt reporting with php-java-bridge with zend server.

I have a lot of jar files to be added to the class path. Those jars are in a folder and lots of jar files are in sub folders.

I tried adding them one by one. But I guess there is a smarter way.

Can somebody help me please?

Thanks

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.