I'm testing out the default (gap) fragmenter with some simple,
single-word queries on a patched 1.3.0 release populated with some
real-world data. (I think the primary quirk in my setup is that I'm
using ShingleFilterFactory to put word bigrams (aka shingles) into my
index. I was worried that this might mess up highlighting, but
highlighting is *mostly* working.) There are some oddities here, and
I'm wondering if people have any suggestions for debugging my setup
and/or trying to make a good, reproducible test case.
1. The main weird thing is that, the vast majority of the time, the
highlighted term is the last term in the fragment. For example, if I
search for "cat", then almost all my fragments look like this:
fragment 1: "to the *cat*"
fragment 2: "with the *cat*"
fragment 3: "it's what the *cat*"
fragment 4: "Once upon a time the *cat*"
(My actual fragments are longer. The key to note is that all of these
examples end in "cat".)
Sometimes "cat" will appear at somewhere other than the last position,
but this is rare. My expectation, in contrast, is that "cat" would
tend to be more or less evenly distributed throughout fragment
positions.
Note: I tried to reproduce this on 1.3.0 with my patches applied but
using the example dataset/schema from the Solr source tree rather than
my own dataset/schema. With the example dataset this didn't seem to be
an issue.
I've experienced three other highlighting issues, which may or may not
be related:
2. Sometimes, if a term appears multiple times in a fragment, not just
the term but all the words in between the two appearances will get
highlighted too. For example, I searched for "fear", and got this as
one of the snippets:
SETTLEMENT AGREEMENT This Agreement ("the Agreement") is entered
into this 18th day of August, 2008, by
and between Cape <em>Fear Bank Corporation, a North Carolina
corporation (the "Company"), and Cape Fear</em>
In contrast, I would have expected
SETTLEMENT AGREEMENT This Agreement ("the Agreement") is entered
into this 18th day of August, 2008, by
and between Cape <em>Fear</em> Bank Corporation, a North Carolina
corporation (the "Company"), and Cape <em>Fear</em>
3. My install seems to have a curiously liberal interpretation of
hl.fragsize. Now if I put hl.fragsize=0, then things are as expected,
i.e. it highlights the whole field. And it also seems more or less
true (as it should) that as I increase hl.fragsize, the fragments get
longer. However, I was surprised to see that when I put hl.fragsize=1
or hl.fragsize=5, I can get fragments as long as this one:
addition, we believe the wireless feature for our controller will
facilitate exceptional customer services and
response time." About GpsLatitude GpsLatitude, a Montreal-based
company, is a provider of security
solutions and tracking for mobile assets. It is also a developer
of advanced " Videlocalisation" , a cost-effective,
integrated mobile digital <em>video</em>
That seems shockingly long for something of size "five".
4. Very rarely I'll get a fragment that doesn't actually contain any
of the search terms. For example, maybe I'll search for "cat", and
I'll get back "three ounces of milk" as a snippet. I need to explore
this more, though the last time this happened when I opened the
document and found that when I located "three ounces of milk" in the
document text, the word "cat" did appear nearby; so maybe the document
did contain "three ounces of milk for the cat".
Obviously I'm not describing my setup in much detail. Let me know what
you think would be helpful to know more about.
Thanks,
Chris
