.NET Zone is brought to you in partnership with:

Software developer and frequent open-source contributor. Writing mostly for .NET, but also Java and C/C++. Really likes fiddling with data, texts especially, so he frequently finds himself working on databases or search engines, usually combining both. Itamar is a DZone MVB and is not an employee of DZone and has posted 31 posts at DZone. You can read more from them at their website. View Full User Profile

Custom Tokenization and Lucene’s FastVectorHighlighter

06.27.2012
| 4901 views |
  • submit to reddit

Perhaps you have tackled this before: you wanted to use Lucene's FastVectorHighlighter (aka FVH), but since you have a custom CharFilter in your analysis chain, the highlighter fails to produce valid fragments.

In my particular case, I used HTMLStripCharFilter (available to Lucene.Net through my pet contrib project) to extract text content from HTML pages, and then pass it through the rest of the analysis process. This confused FVH, since it was taking the full content from store, where HTML was still present, and token positions were not taking that into account. And any other custom CharFilter that is added to the analysis chain is going to cause the same troubles.

To overcome this, I needed to make sure FVH is aware of all content stripping operations that are made before or while tokenization is happening. All I had to do was to implement a custom FragmentsBuilder, looking as follows (.Net code; a Java version would look almost identical):

public class HtmlFragmentsBuilder : BaseFragmentsBuilder
{
    /// <summary>
    /// a constructor.
    /// </summary>
    public HtmlFragmentsBuilder()
        : base()
    {
    }
 
    /// <summary>
    /// a constructor.
    /// </summary>
    /// <param name="preTags">array of pre-tags for markup terms</param>
    /// <param name="postTags">array of post-tags for markup terms</param>
    public HtmlFragmentsBuilder(String[] preTags, String[] postTags)
        : base(preTags, postTags)
    {
    }
 
    /// <summary>
    /// do nothing. return the source list.
    /// </summary>
    public override List<WeightedFragInfo> GetWeightedFragInfoList(List<WeightedFragInfo> src)
    {
        return src;
    }
 
    protected override String GetFragmentSource(StringBuilder buffer, int[] index, Field[] values, int startOffset, int endOffset)
    {
        string fieldText;
        while (buffer.Length < endOffset && index[0] < values.Length)
        {
            fieldText = GetFilteredFieldText(values[index[0]]);
            if (index[0] > 0 && values[index[0]].IsTokenized() && fieldText.Length > 0)
                buffer.Append(' ');
            buffer.Append(fieldText);
            ++(index[0]);
        }
        var eo = buffer.Length < endOffset ? buffer.Length : endOffset;
        return buffer.ToString().Substring(startOffset, eo - startOffset);
    }
 
    /// <summary>
    /// Gets the field text, after applying custom filtering
    /// </summary>
    /// <param name="field"></param>
    /// <returns></returns>
    protected string GetFilteredFieldText(Field field)
    {
        var theStream = new MemoryStream(Encoding.UTF8.GetBytes(field.StringValue()));
        var reader = CharReader.Get(new StreamReader(theStream));
        reader = new HTMLStripCharFilter(reader);
 
        int r;
        var sb = new StringBuilder();
        while ((r = reader.Read()) != -1)
        {
            sb.Append((char)r);
        }
        return sb.ToString();
    }

 FVH will then need to be configured to use it:

var fvh = new FastVectorHighlighter(FastVectorHighlighter.DEFAULT_PHRASE_HIGHLIGHT,                                             FastVectorHighlighter.DEFAULT_FIELD_MATCH,
                    new SimpleFragListBuilder(), new HtmlFragmentsBuilder());
// ...
var fq = fvh.GetFieldQuery(query);
var fragment = fvh.GetBestFragment(fq, searcher.GetIndexReader(), hits[i].doc, "Content", 300);

If you're using Lucene.Net, you'll have to make sure this patch is applied to your FVH before this could compile.

That was the easiest way to get this working, and fast. Perhaps I could make it more generic, or change the original implementation to allow that and submit it as a patch. Maybe I'll do it someday. Or you could...

Published at DZone with permission of Itamar Syn-hershko, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Rafał Kuć replied on Thu, 2012/06/28 - 5:27am

Hey, guys, I'm not the author of this article :)

Will Martin replied on Thu, 2012/06/28 - 9:25pm

You can't even get it straight sir. The FVH is yabaw (yet another bell and whistlle) from the Solr community. If it were a lucene artifact, and lucene is no longer an independent project (and SOLR is a whore for the Enterprise Market), it would be a highly surfaced, broad thread safe, index serialization compression expertise ... grounded.   Not being such, means that lots and lots of people will become dependent on the FVH and the strategic strength and advancement that would have come from a Lucene API for IR engine experts, will be lost.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.