Rafal Kuc is a team leader and software developer. Right now he is a software architect and Solr and Lucene specialist. Mainly focused on Java, but open on every tool and programming language that will make the achievement of his goal easier and faster. Rafal is also one of the founders of solr.pl site where he tries to share his knowledge and help people with their problems. Rafał is a DZone MVB and is not an employee of DZone and has posted 75 posts at DZone. You can read more from them at their website. View Full User Profile

Sorting by function value in Solr

09.27.2011
| 6904 views |
  • submit to reddit

In Solr 3.1 and later we have a very interesting functionality, which enables us to sort by function value. What does that gives us? Actually, a few interesting possibilities.

Let’s start

The first example that comes to mind, perhaps because of the project on which I worked some time ago, it’s sorting on the basis of distance between two geographical points. So far, to implement such functionality was needed changes in Solr (for example, LocalSolr or LocalLucene). Using Solr 3.1 and later, you can sort your search results using the value returned by the defined functions. For example, is Solr, we have the dist function calculating the distance between two points. One variation of the function is a function accepting five parameters: algorithm and two pairs of points. If, using this feature, we would like to sort your search results in ascending order from the point of latitude and longitude 0.0, we should add the following sort parameter to the Solr query:

 
...sort=dist(2, geo_x, geo_y, 0, 0) asc

I suspect that the most commonly used values of the first parameter will be:

  • 1 – calculation based on the Manhattan metrics
  • 2 – calculation of Euclidean distance

A few words about performance

Everything is fine till now, but how it looks like in terms of performance ? I’ve made a two simple tests.

During the first test, I indexed 200 000 documents, every one of them consisted of four fields: identifier (numeric field), description (a text field) and location (two numeric fields). In order not to obscure the test results for sorting, I used one of the simplest functions currently available in the Solr – the sum function which sums two given arguments. I compared the query time of the default sorting (by score) with the ones which used the value of the function. The following table shows the results of the test:

QueryResults numberQuery timeRe-query time
q=*:*&sort=score+desc200.00031ms0ms
q=*:*&sort=sum(geo_x,geo_y)+desc200.000813ms0ms
q=opis:ala&sort=score+desc200.00047ms1ms
q=opis:ala&sort=sum(geo_x,geo_y)+desc200.000797ms1ms

Another test was based on a comparison of sorting by a string field to sort using function. The test was almost identical to the first test. I’ve indexed 200,000 documents indexed (with additional field: name_sort – type string) and used the sum function. The following table shows the results of the test:

QueryResults numberQuery timeRe-query time
q=*:*&sort=opis_sort+desc200.000267ms0ms
q=*:*&sort=sum(geo_x,geo_y)+desc200.000823ms0ms
q=opis:ala&sort=opis_sort+desc200.000266ms1ms
q=opis:ala&sort=sum(geo_x,geo_y)+desc200.000810ms1ms

Above test shows that sorting using the sort function is much slower than the default sort order (which you’d expect). Sorting on the basis of function value is also slower than sorting with the use of string based field, but the difference is not as significant as in the previous case.

A few words at the end

Of course, the above test just glides through the topic of sorting efficiency using Solr functions, however, shows a direct relationship. Given that, in most cases, this will not be the default sort method and giving us a really powerful tool it seems to me that this is a feature worth remembering. It will definitely be worth using when the requirements says that we have to sort on the value that depends on the query and index values – as in the case of sorting by distance from the point specified by the user.

References
Published at DZone with permission of Rafał Kuć, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags: