I would very much like to hear how people use payloads.
Personally I use them for weight only. And I use them a lot, almost in
all applications. I factor the weight of synonyms, stems,
dediacritization and what not. I create huge indices that contains
lots tokens at the same position but
t involves payloads...
Elias.
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
ually allow you to have more strongly typed payloads which means
you won't have to decode (well, kind of, Lucene will do the decoding
for you). Only problem is they aren't yet supported on the search
side. In other words, your wish for a reusable API is being worked on.
Have a
Hi Karl,
I use payloads for weight only, too, with BoostingTermQuery (see:
http://www.nabble.com/BoostingTermQuery-scoring-td20323615.html#a20323615)
A custom tokenizer looks for the reserved character '\b' followed by a 2
byte 'boost' value. It then creates a special Tok
Is it possible to update the payloads of an existing index? I having
troubles finding any mention of this on the mailing list archives and it
is not obvious that this is possible from the api. I do not want to
change the size of the payloads - just update the values. My payloads
values
Hi,
I started playing with the experimental payload functionality. I have
written an analyzer which adds a payload (some sort of a score/boost)
for each term occurance. The payload/score for each term is dependent on
the document that the term comes from (I guess this is the typoical use
case)
I'm interested in getting the payload information from the
matching span, however it's unclear from the javadocs why
NearSpansUnordered is different than NearSpansOrdered in this
regard.
NearSpansUnordered returns payloads in a hash set that's
computed each method call by ite
the offset of its first word.
>
> I guess it involves payloads...
>
> Elias.
'!' is already a query operator. It is equivalent of NOT. So you cannot use it.
Why not use carat operator? Like singleterm^2 "some phrase"^2
[Boosti
I want to override the operator - it is for a project purpose.
-Original Message-
From: AHMET ARSLAN [mailto:iori...@yahoo.com]
Sent: Saturday, December 19, 2009 6:41 PM
To: java-user@lucene.apache.org
Subject: Re: Payloads
> Hi,
>
> I need to add a query operator '!'
> I want to override the operator - it
> is for a project purpose.
Can you explain your requirements more? What do you mean by "an even offset of
the document"?
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apa
is even) - we apply
this doubling of weight only if a '!' operator precedes the term and if its
offset from the document is even.
-Original Message-
From: AHMET ARSLAN [mailto:iori...@yahoo.com]
Sent: Saturday, December 19, 2009 6:48 PM
To: java-user@lucene.apache.org
Subject:
> Sent: Saturday, December 19, 2009 5:54 PM
> To: java-user@lucene.apache.org
> Subject: RE: Payloads
>
> Let's say I have a document that contains the following text:
>
> "Graph Algorithms is one of the most important topics in computer science"
>
> And
About 60 students I think, if you have given some answers I would be
grateful if you could link me to them or quote them again.
-Original Message-
From: Uwe Schindler [mailto:u...@thetaphi.de]
Sent: Saturday, December 19, 2009 7:00 PM
To: java-user@lucene.apache.org
Subject: RE: Payloads
to "Graph|2.0 Algorithms" you
can use DelimitedPayloadTokenFilter to set payload of marked term.
Additionally you need to everride QueryParser to return PayloadTermQuery
and scorePayload method of DefaultSimilarity.
By doing so payloads will be included in score calculation.
public
oaded ?
-Original Message-
From: AHMET ARSLAN [mailto:iori...@yahoo.com]
Sent: Saturday, December 19, 2009 8:34 PM
To: java-user@lucene.apache.org
Subject: RE: Payloads
> Let's say I have a document that
> contains the following text:
>
> "Graph Algorithms is one of t
> If I need to override the QueryParser
> to return PayloadTermQuery, what
> function for PayloadFunction should I use in the
> constructor (If you can
> show me an example).
I am not sure about that. Maybe custom one.
> In your code I didn't see an indexer, will this work with
> the regular
> I
What do you mean by a custom one - please explain. I must use a
PayloadTermQuery ?
And for the TermPositionPayloadTokenFilter there is a method that is not
used - incrementToken (only used in the main method) ... I didn't see in the
code the place that examines if the query term is at an even offs
hits2 = searcher.search(query2, 10).scoreDocs;
for (int i = 0; i < hits2.length; i++) {
Document hitDoc = searcher.doc(hits2[i].doc);
System.out.println(hitDoc.get("title"));
}
}
}
-Original Message-
From: AHMET ARSLAN [mail
.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de
> -Original Message-
> From: Elias Khsheibun [mailto:eli...@gmail.com]
> Sent: Sunday, December 20, 2009 2:51 PM
> To: java-user@lucene.apache.org
> Subject: RE: Payloads
>
>
> I'm t
er@lucene.apache.org
Subject: RE: Payloads
> Let's say I have a document that
> contains the following text:
>
> "Graph Algorithms is one of the most important topics in computer
> science"
>
> And a query "!Graph Algorithms" then the term Graph in the q
Hi,
I'm trying to extract payloads from an index for specific tokens the
following way (inserting sample document number and term):
Terms terms = reader.getTermVector(16504, "term");
TokenStream tokenstream = TokenSources.getTokenStream(terms);
while (tokenstream
AUTOMATIC REPLY
LUX is closed until 5th January 2009
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
nk
hacking it is the only way for now. Someone else might have a better
idea, but I confess I haven't thought too much about it.
On Dec 31, 2008, at 2:34 PM, Philip Ogren wrote:
Is it possible to update the payloads of an existing index? I
having troubles finding any mention of this
:34 PM, Philip Ogren wrote:
Is it possible to update the payloads of an existing index? I
having troubles finding any mention of this on the mailing list
archives and it is not obvious that this is possible from the api.
I do not want to change the size of the payloads - just update the
On Apr 21, 2009, at 4:39 AM, Murat Yakici wrote:
Hi,
I started playing with the experimental payload functionality. I
have written an analyzer which adds a payload (some sort of a score/
boost) for each term occurance. The payload/score for each term is
dependent on the document that the t
No, just checking. I will let you everyone know I see one.
Cheers,
Murat
>
> On Apr 21, 2009, at 4:39 AM, Murat Yakici wrote:
>
>> Hi,
>> I started playing with the experimental payload functionality. I
>> have written an analyzer which adds a payload (some sort of a score/
>> boost) for each te
Dear Murat,
I saw your question and wondered how did you implement these changes?
The requirement below are the same ones as I am trying to code now.
Did you modify the source code itself or only used Lucene's jar and just
override code?
I would very much apprecicate if you could give me a short
Here is what I am doing, not so magical... There are two classes, an
analyzer and an a TokenStream in which I can inject my document dependent
data to be stored as payload.
private PayloadAnalyzer panalyzer = new PayloadAnalyzer();
private class PayloadAnalyzer extends Analyzer {
}
public void reset(Reader input) throws IOException
{
tok.reset(input);
}
public void close() throws IOException
{
tok.close();
}
}
**
*Example for the use of payloads:*
PayloadAnalyzer panalyzer = new PayloadAnalyzer();
File ind
ply put, I want to add some scores to documents/terms, but I can't
generate that score before I observe the document/terms. If I do that I
would replicate some of the work that is being already done by
IndexWriter.
If I remember it correctly, there is also some intention to add document
payload
rate that score before I observe the document/terms. If I do that I
> would replicate some of the work that is being already done by
> IndexWriter.
>
> If I remember it correctly, there is also some intention to add document
> payloads functionality. I have the same concerns on thi
gt; IndexWriter
>> already does these somewhere down deep.
>>
>>
>> Simply put, I want to add some scores to documents/terms, but I can't
>> generate that score before I observe the document/terms. If I do that I
>> would replicate some of the work that is be
exWriter.
If I remember it correctly, there is also some intention to add document
payloads functionality. I have the same concerns on this. So I think we
need a clear view on the topic. Where is the payload work moving? How we
can generate a score without duplicating some of the work that
IndexWr
y have an indexing time penalty. If your window size is
> >> the document itself, you will be doing the same job twice (calculating
> >> the
> >> num of times a term occurs in doc X, index time weights etc.).
> >> IndexWriter
> >> already does these somewhere
gt;> 2009/4/26 Murat Yakici
>>>>
>>>>
>>>>
>>>>> Yes, this is more or less what I had in mind. However, for this
>>>>> approach
>>>>> one requires some *prior knowledge* of the vocabulary of the document
>>>>&g
Hi all,
I'm using the new payloads feature to assign types to tokens as I
index. The type is based on the surrounding text in the document, and
I want to filter my searches based on this token type.
For example, I may index the token "house" maybe found in different
places
If a SpanQuery is constructed from one or more BoostingTermQuery(s), the
payloads on the terms are never processed by the SpanScorer. It seems to me
that you would want the SpanScorer to score the document both on the spans
distance and the payload score. So, either the SpanScorer would have to
I started playing with payloads and have been trying to work out how to get the
data into the payload
I have a field where I want to add the following untokenized fields
A1
A2
A3
With these fields, I would like to add the payloads
B1
B2
B3
Firstly, it looks like you cannot add payloads to
regard.
>
> NearSpansUnordered returns payloads in a hash set that's
> computed each method call by iterating over the SpanCell as a
> linked list, whereas NearSpansOrdered stores the payloads in a
> list (which is ordered) only when collectPayloads is true.
>
> At first
On Nov 20, 2009, at 6:49 PM, Jason Rutherglen wrote:
> I'm interested in getting the payload information from the
> matching span, however it's unclear from the javadocs why
> NearSpansUnordered is different than NearSpansOrdered in this
> regard.
>
> NearSpansUno
an NearSpansOrdered in this
>> regard.
>>
>> NearSpansUnordered returns payloads in a hash set that's
>> computed each method call by iterating over the SpanCell as a
>> linked list, whereas NearSpansOrdered stores the payloads in a
>> list (which is ordered) only when co
I don't mind adding the "positions" of the payloads in them. However,
maybe we can be little more clear in the javadocs what's going on
underneath?
On Wed, Nov 25, 2009 at 5:36 AM, Mark Miller wrote:
> Grant Ingersoll wrote:
>> On Nov 20, 2009, at 6:49 PM, Jason Ru
Hello everybody,
I am currently experimenting with Lucene 4.0 and would like to add payloads.
Payload should only be added once per term on the first position. My current
code looks like this:
public final boolean incrementToken() throws java.io.IOException {
String term
I'm looking at the new Payload api and would like to use it in the following
manner. Meta-data is indexed as a special phrase (all terms at same
position) and a payload is stored with the first term of each phrase. I
would like to create a custom query class that extends PhraseQuery and uses
its P
TermVectors are per-document and do not contain payloads. You are reading the
per-document TermVectors which is a "small index" *stored* for each document as
a binary blob. This blob only contains the terms of this document with its
positions/offsets, but no payloads (offsets are use
Actually, term vectors can store payloads now (LUCENE-1888), so if that
field was indexed with FieldType.setStoreTermVectorPayloads they should be
there.
But I suspect the TokenSources.getTokenStream API (which I think un-inverts
the term vectors to recreate the token stream = very slow?) wasn&#
Am 23.04.2013 13:21, schrieb Michael McCandless:
> Actually, term vectors can store payloads now (LUCENE-1888), so if that
> field was indexed with FieldType.setStoreTermVectorPayloads they should be
> there.
>
> But I suspect the TokenSources.getTokenStream API (which I think u
Am 23.04.2013 13:47, schrieb Carsten Schnober:
> I'm trying to figure out a way to use a query as Uwe suggested. My
> scenario is to perform a query and then retrieve some of the payloads
> upon user request, so there no obvious way to wrap this into a query as
> I can't know
g to figure out a way to use a query as Uwe suggested. My
>> scenario is to perform a query and then retrieve some of the payloads
>> upon user request, so there no obvious way to wrap this into a query as
>> I can't know what (terms) to query for.
>
> I wonder: is
d on a SpanQuery which depends on a specific
expression to search for. In my use case, I need to retrieve Spans
specified by their offsets only, and then get their payloads and process
them further. Alternatively, I could query for the occurence of certain
string patterns in the payloads and c
Hi Carsten,
It doesn't sound as though an inverted index is really what you want to be
querying here, if I'm reading you right. You want to get the payloads for
spans at a specific position, but you don't particularly care about the actual
term at that position? You
Am 23.04.2013 16:17, schrieb Alan Woodward:
> It doesn't sound as though an inverted index is really what you want to be
> querying here, if I'm reading you right. You want to get the payloads for
> spans at a specific position, but you don't particularly care about th
ver, when I print the document off and look at
the index in Luke, the field I'm trying to append the payload to has no
terms or payloads associated with it even though I specified both term and
payload attributes in the token stream code. I'm quite confused how these
things are supposed
AUTOMATIC REPLY
LUX is closed until 5th January 2009
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
That sounds pretty cool Karl, and I also dig your use of Motorhead as an
example : )
I recently built an application where payloads were a lifesaver, but my
usage of them is pretty basic. I am indexing pages of text, so I use
payloads to store metadata about each word on the page - size, color
That sounds pretty cool Karl, and I also dig your use of Motorhead as an
example : )
I recently built an application where payloads were a lifesaver, but my
usage of them is pretty basic. I am indexing pages of text, so I use
payloads to store metadata about each word on the page - size, color
6 feb 2008 kl. 23.10 skrev Matt Ronge:
I may index the token "house" maybe found in different places with
different types. If the user query contains house, I want to report
the number of instances of the token house of type A, type B and so
on.
Should I be using payloads fo
type B and so
on.
Should I be using payloads for this? If so, I'd like to be able to
count up all the instances of for each type. Then I can show the
results, along with TypeA (100 hits), TypeB (1000 hits) so on.
Pehaps, what do you do with these numbers you extract?
If I could use
type B and so
on.
Should I be using payloads for this? If so, I'd like to be able to
count up all the instances of for each type. Then I can show the
results, along with TypeA (100 hits), TypeB (1000 hits) so on.
Pehaps, what do you do with these numbers you extract?
I would like to
instances of the token house of type A, type
B and so on.
Should I be using payloads for this? If so, I'd like to be able to
count up all the instances of for each type. Then I can show the
results, along with TypeA (100 hits), TypeB (1000 hits) so on.
Pehaps, what do you do with th
I'm not fully following what you want. Can you explain a bit more?
Thanks,
Grant
On Jul 9, 2008, at 2:55 PM, Peter Keegan wrote:
If a SpanQuery is constructed from one or more BoostingTermQuery(s),
the
payloads on the terms are never processed by the SpanScorer. It
seems to me
tha
erall score based on the payloads assigned to them. Thus the score from
the SpanNearQuery would be higher if :
a) the terms have payloads that boost their scores
b) the terms are positionally next to each other (minimal slop - as it works
now)
Does this make sense?
Peter
On Thu, Jul 10, 20
uery,
SpanFirstQuery) recognize that the terms in the phrase should boost
the
overall score based on the payloads assigned to them. Thus the score
from
the SpanNearQuery would be higher if :
a) the terms have payloads that boost their scores
b) the terms are positionally next to each other (mi
s looking for a way to have SpanNearQuery (also SpanOrQuery,
>> SpanFirstQuery) recognize that the terms in the phrase should boost the
>> overall score based on the payloads assigned to them. Thus the score from
>> the SpanNearQuery would be higher if :
>>
>> a) the
lacks a recursive traversal of the query tree), which addresses the same
problem.
So, I made an attempt to modify SpanNearQuery to expand a nested
BoostingTermQuery, but soon realized while debugging that since
BoostingTermQuery loads payloads from all term positions in the document,
not just the
IIRC first versions of patches that added payloads support had this notion
of payload by field rather than by token, but later it was modified to be by
token only.
I have seen two code patterns to add payloads to tokens.
The first one created the field text with a reserved separator/delimiter
Thanks for your comments Doron. I found the earlier discussions on the dev list
(21/12/06), where this issue is discussed - my use case is similar to Nadav Har'El.
Implementing payloads via Tokens explicitly prevents the use of payloads for
untokenized fields, as they only su
>
> Implementing payloads via Tokens explicitly prevents the use of payloads
> for untokenized fields, as they only support field.stringValue(). There
> seems no way to override this.
I assume you already know this but just to make sure what I meant was clear
- on tokenizati
Hi to everyone,
I started to use payloads in my indexes.
However, I didn't find a class that permits to perform queries using
more than one term.
The only way that works is to use the BoostingTermQuery class, but by
using this class, I can search only one term a time.
Does exist a way to
Hi Guys,
I've been using Lucene for more than 5 years and it is a great tool -
great job! Thanks for everything...
Lately I encountered the new payloads support and it looks its a great
solution for my project.
*The problem:*
The use case is as follows:
I need to support a w
You cannot do it because TermPositions is read in the
PhraseWeight.scorer(IndexReader) method (or MultiPhraseWeight) and
loaded into an array which is passed to PhraseScorer. Extend the Weight
as well and pass the payload to the Scorer as well is a possibility.
- Mark
Peter Keegan wrote:
I'm
Could you get what you need combining the BoostingTermQuery with a
SpanNearQuery to produce a score? Just guessing here..
At some point, I would like to see more Query classes around the
payload stuff, so please submit patches/feedback if and when you get
a solution
On Jun 27, 2007, at 1
I tried to subclass PhraseScorer, but discovered that it's an abstract class
and its subclasses (ExactPhraseScorer and SloppyPhraseScorer) are final
classes. So instead, I extended Scorer with my custom scorer and extended
PhraseWeight (after making it public). My scorer's constructor is passed th
I'm now looking at using payloads with SpanNearQuery but I don't see any
clear way of getting the payload(s) from the matching span terms. The term
positions for the payloads seem to be buried beneath SpanCells in the
NearSpansOrdered and NearSpansUnordered classes, which are not publ
: I'm now looking at using payloads with SpanNearQuery but I don't see any
: clear way of getting the payload(s) from the matching span terms. The term
: positions for the payloads seem to be buried beneath SpanCells in the
Isn't Spans.start() and Spans.end() what you are looking
ECTED]> wrote:
: I'm now looking at using payloads with SpanNearQuery but I don't see any
: clear way of getting the payload(s) from the matching span terms. The
term
: positions for the payloads seem to be buried beneath SpanCells in the
Isn't Spans.start() and Spans.end() what
stetter <[EMAIL PROTECTED]> wrote:
: I'm now looking at using payloads with SpanNearQuery but I don't
see any
: clear way of getting the payload(s) from the matching span
terms. The
term
: positions for the payloads seem to be buried beneath SpanCells
in the
Isn't Spans.st
this would be useful. Perhaps this should be discussed on dev.
And the same holds for the payloads, there many be more than one
for a single Span.
Regards,
Paul Elschot
>
> Cheers,
> Grant
>
>
> On Jul 12, 2007, at 8:20 AM, Peter Keegan wrote:
>
> > I'm loo
Yep, totally agree.One way to handle this initially at least is
have isPayloadAvailable() only return true for the SpanTermQuery.
The other option is to come up with some modification of the
suggested methods below to return all the payloads in a span.
I have a basic implementation
.
The other option is to come up with some modification of the
suggested methods below to return all the payloads in a span.
I have a basic implementation for just the SpanTermQuery (i.e. via
TermSpans) in the works. I will take a crack at fleshing out the
rest at some point soon.
-Grant
On Jul 1
> I'm looking for Spans.getPositions(), as shown in
...
: >> : I'm now looking at using payloads with SpanNearQuery but I don't
: >> see any
: >> : clear way of getting the payload(s) from the matching span
: >> terms. The
Hmm... okay so the issue is
he Payload data
c) add methods to the Spans API to get the subspans (if any) ... this
would be the Spans corrilary to getTerms() and would always return
TermSpans which would have TermPositions for getting payload data.
This could be a good alternative.
When we first talked about payloads we wond
? The
reason I ask is that my custom analyzer's 'tokenStream' method creates a
custom filter which produces a payload based on the existence of each field
occurrence. However, if DocumentsWriter was changed and combined all the
occurrences before inversion, my scheme wouldn't
hod creates a
> custom filter which produces a payload based on the existence of each field
> occurrence. However, if DocumentsWriter was changed and combined all the
> occurrences before inversion, my scheme wouldn't work. Since payloads are
> created by filters/tokenizers, it hel
Hi All,
I'm trying to create an index with payloads using Lucene 4.5 but the payloads
don't seem to be getting into the index.
I looked into the code and it looks like storePayloads argument to
addOrUpdateInternal when called from addOrUpdate in FieldInfos is set to false
in sequences of payloads.
(trying not to be too pedantic, a regular chunker looks for 'chunks' based
on part-of-speech tags, e.g. noun phrases can be searched for with patterns
like "(DT)?(JJ)*(NN|NP)+", that is, an optional determinant and zero or
more adjectives preceding
Hello,
To support payloads we rewrite SynonymQuery to a pair of SpanTerm queries which
we then can wrap in the PayloadScoreQuery. This is not the right way to do this
because if both clauses match, both are also scored. We could try to rewrite
SynonymQuery to a SpanOrQuery but i suppose that
hile now I have more
questions on payloads.
1. it seems that in order to use payload I must write my own Query
class like the BoostingTermQuery. Since I need to use payload with a
lot of the Lucene built-in Query classes, does it mean that I have to
write all those Query class again?
2. it seems t
Hi,
I'm running a SpanQuery and get the Spans result which tell me the documents
and positions of what I searched for.
I would now like to get the payloads in those documents and positions
without having to iterate on TermPositions since I don't have a term but I
do have the document an
Hi to everyone,
I started to use payloads in my indexes.
However, I didn't find a class that permits to perform queries using
more than one term.
The only way that works is to use the BoostingTermQuery class, but by
using this class, I can search only one term a time.
Does exist a way to do a
ght help you. Otherwise,
> the PayloadTermQuery is still a Query and can be used in a BooleanQuery.
> Beyond that, you may need to write what you need, since it doesn't exist
> yet.
>
>
> On Oct 26, 2009, at 10:25 AM, Mauro Dragoni wrote:
>
>> Hi to everyone,
>>
document and to assign the
language tags as payloads to the terms. But then I need
(i) a search filter that eliminates docs based on a given language tag and
(ii) a way to access the term payloads from the documents returned by the
searcher
For both I haven't found a solution yet. Can I wr
Hi Guys,
I've been using Lucene for more than 5 years and it is a great tool -
great job! Thanks for everything...
Lately I encountered the new payloads support and it looks its a great
solution for my project.
*The problem:*
The use case is as follows:
I need to support a w
ions inline below.
>
>
> Lately I encountered the new payloads support and it looks its a great
> solution for my project.
>
>
> *The problem:*
>
> The use case is as follows:
>
> I need to support a way to calculate statistics on web pages.
>
> Each pa
pert usages
> of Lucene. From what I can tell, you have sent the same question 3 times in
> a matter of less than a day. Sending more than once in a 2-3 day period is
> just going to make it less likely that you will get help, not more likely.
>
> Some suggestions inline below.
those two?
Mike McCandless
http://blog.mikemccandless.com
On Tue, Oct 15, 2013 at 8:13 AM, Kyle Judson wrote:
>
>
>
>
>
>
>
>
> Hi All,
>
>
> I'm trying to create an index with payloads using Lucene 4.5 but the payloads
> don't seem to be getting
That was it. Reversing the order fixed it.
Thanks for wading through the lack of whitespace.
ThanksKyle
> From: luc...@mikemccandless.com
> Date: Tue, 15 Oct 2013 09:24:17 -0400
> Subject: Re: Payloads disabled in 4.5?
> To: java-user@lucene.apache.org
>
> Something catastroph
score (payload * normalize
coefficient)
How can I do that? Custom payload similarity class or custom function
query?
I've followed here:
http://sujitpal.blogspot.com/2013/07/porting-payloads-to-solr4.html#! but
decodeNormValue if a final method anymore. How about that:
http://www.solrtuto
Hi,
It's great to see support for payloads in the suggesters - this is really
helpful, and pretty much addresses LUCENE-4516. Are there any plans to also
support them for WFSTs? We have some cases where we don't need the Analyzer's
capabilities (we look up the completion us
re/4_10_2/core/org/apache/lucene/search/spans/SpanNearPayloadCheckQuery.html>
since
it properly handles the fact that payloads aren't ordered by SpanNearQuery
<http://lucene.apache.org/core/4_10_2/core/org/apache/lucene/search/spans/SpanNearQuery.html>."
?
I used SpanNearQuery a
1 - 100 of 165 matches
Mail list logo
