Re: Solr Document expiration with TTL
Great, Thank you for confirming. This will help other people seeing similar kinds of issues. Regards, Alex. Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/ On 9 March 2015 at 05:02, Makailol Charls 4extrama...@gmail.com wrote: Hi, As suggested, we could manage to activate the document expiration using TTL by shifting the newly added URP setting in the starting of the add-unknown-fields-to-the-schema URP. That populates the expire_at_dt field and makes document to expire after defined TTL. Many Thanks, Makailol
Re: Solr Document expiration with TTL
Hi, As suggested, we could manage to activate the document expiration using TTL by shifting the newly added URP setting in the starting of the add-unknown-fields-to-the-schema URP. That populates the expire_at_dt field and makes document to expire after defined TTL. Many Thanks, Makailol On Fri, Feb 27, 2015 at 7:41 PM, Alexandre Rafalovitch arafa...@gmail.com wrote: Yep, Your default URP chain is probably not being triggered due to the initParams. initParams are new in Solr 5, so this is still rough around the edges advice. But try giving your chain a name and adding explicit update.chain value to the requestHandler section (not initParams) section. Alternatively, since add-unknown-fields is already used, you could move your extra URPs to the start of that instead. In fact, if you are doing both timestamps and dynamically adding fields to the schema, you will need to do that anyway. Regards, Alex. On 27 February 2015 at 08:53, Makailol Charls 4extrama...@gmail.com wrote: initParams path=/update/** lst name=defaults str name=update.chainadd-unknown-fields-to-the-schema/str /lst /initParams Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/
Re: Solr Document expiration with TTL
Hi, As suggested, we could manage to activate the document expiration using TTL by shifting the newly added URP setting in the starting of the add-unknown-fields-to-the-schema URP. That populates the expire_at_dt field and makes document to expire after defined TTL. Many Thanks, Makailol On Fri, Feb 27, 2015 at 10:23 PM, Chris Hostetter hossman_luc...@fucit.org wrote: : There were no changes made in the solrconfig.xml file except added that : updateRequestProcessorChain default=true/updateRequestProcessorChain : block. ok, first off: if you already *had* another updateRequestProcessorChain that said 'default=true' just adding a new one would be weird and would likely give you errors. you have t oconsider the whole context of the config and the other updateRequestProcessorChains when you make edits like that. : initParams path=/update/** : lst name=defaults : str name=update.chainadd-unknown-fields-to-the-schema/str : /lst : /initParams so that says whe nyou make any requests to a /update handler, it's going to use a default request param of update.chain=add-unknown-fields-to-the-schema. so your updates are not going to the default hanler (which you didn't give a name) they are going though the updateRequestProcessorChain/ with the name=add-unknown-fields-to-the-schema you should probably remove the chain you added, and instead put the new processors you want in the add-unknown-fields-to-the-schema chain. that's the simplest way to get what you want in place. -Hoss http://www.lucidworks.com/
Re: Solr Document expiration with TTL
Hi Thanks for the reply. I am just beginning with the solr, so not much familiar with the settings of the solr. I have created solr collection1 core with the following command. bin/solr create -c collection1 Then modified the managed-schema file to add required field definitions There were no changes made in the solrconfig.xml file except added that updateRequestProcessorChain default=true/updateRequestProcessorChain block. I can see below code defined in my solrconfig.xml file by default. initParams path=/update/**,/query,/select,/tvrh,/elevate,/spell,/browse lst name=defaults str name=df_text/str /lst /initParams initParams path=/update/** lst name=defaults str name=update.chainadd-unknown-fields-to-the-schema/str /lst /initParams *While for requestHandler/ I think its below one?* requestHandler name=/update/extract startup=lazy class=solr.extraction.ExtractingRequestHandler lst name=defaults str name=lowernamestrue/str str name=fmap.metaignored_/str str name=fmap.content_text/str /lst /requestHandler Thanks, Makailol On Thu, Feb 26, 2015 at 10:39 PM, Chris Hostetter hossman_luc...@fucit.org wrote: : If your expire_at_dt field is not populated automatically, let's step : back and recheck a sanity setting. You said it is a managed schema? Is : it a schemaless as well? With an explicit processor chain? If that's : the case, your default chain may not be running AT ALL. yeah ... my only guess here is that even though you posted before that you had this configured in your defaut chain... processor class=solr.processor.DocExpirationUpdateProcessorFactory int name=autoDeletePeriodSeconds30/int str name=ttlFieldNametime_to_live_s/str str name=expirationFieldNameexpire_at_dt/str /processor ...perhaps you have an update.chain=foo type default param configured for your /update handler? * what does your /update requestHandler/ config look like? * are you using the new initParams/ feature of solr? what does it's config look like? : So, recheck your solrconfig.xml. Or add another explicit field : population inside the chain, just like the example did with : TimestampUpdateProcessorFactory : : https://lucidworks.com/blog/document-expiration/ yeah ... that would help as a sanity check as well ... point is: we need to verify which chain you are using when adding the doc. : : : : : Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: : http://www.solr-start.com/ : -Hoss http://www.lucidworks.com/
Re: Solr Document expiration with TTL
Yep, Your default URP chain is probably not being triggered due to the initParams. initParams are new in Solr 5, so this is still rough around the edges advice. But try giving your chain a name and adding explicit update.chain value to the requestHandler section (not initParams) section. Alternatively, since add-unknown-fields is already used, you could move your extra URPs to the start of that instead. In fact, if you are doing both timestamps and dynamically adding fields to the schema, you will need to do that anyway. Regards, Alex. On 27 February 2015 at 08:53, Makailol Charls 4extrama...@gmail.com wrote: initParams path=/update/** lst name=defaults str name=update.chainadd-unknown-fields-to-the-schema/str /lst /initParams Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/
Re: Solr Document expiration with TTL
: There were no changes made in the solrconfig.xml file except added that : updateRequestProcessorChain default=true/updateRequestProcessorChain : block. ok, first off: if you already *had* another updateRequestProcessorChain that said 'default=true' just adding a new one would be weird and would likely give you errors. you have t oconsider the whole context of the config and the other updateRequestProcessorChains when you make edits like that. : initParams path=/update/** : lst name=defaults : str name=update.chainadd-unknown-fields-to-the-schema/str : /lst : /initParams so that says whe nyou make any requests to a /update handler, it's going to use a default request param of update.chain=add-unknown-fields-to-the-schema. so your updates are not going to the default hanler (which you didn't give a name) they are going though the updateRequestProcessorChain/ with the name=add-unknown-fields-to-the-schema you should probably remove the chain you added, and instead put the new processors you want in the add-unknown-fields-to-the-schema chain. that's the simplest way to get what you want in place. -Hoss http://www.lucidworks.com/
Re: Solr Document expiration with TTL
Hi, Thanks for the reply. I tried adding following code block in updateRequestProcessorChain default=true in solrconfig.xml. processor class=solr.TimestampUpdateProcessorFactory str name=fieldNametimestamp_dt/str /processor and added field definition in managed-schema field name=timestamp_dt type=date stored=true multiValued=false / But then dont see this field is getting populated in document. Thanks, Makailol On Thu, Feb 26, 2015 at 8:08 PM, Alexandre Rafalovitch arafa...@gmail.com wrote: If your expire_at_dt field is not populated automatically, let's step back and recheck a sanity setting. You said it is a managed schema? Is it a schemaless as well? With an explicit processor chain? If that's the case, your default chain may not be running AT ALL. So, recheck your solrconfig.xml. Or add another explicit field population inside the chain, just like the example did with TimestampUpdateProcessorFactory : https://lucidworks.com/blog/document-expiration/ Regards, Alex. On 26 February 2015 at 07:52, Makailol Charls 4extrama...@gmail.com wrote: since your time_to_live_s and expire_at_dt fields are both stored, can you confirm that a expire_at_dt field is getting popularted by the update processor by doing as simple query for your doc (ie q=id:10seconds) No, expire_at_dt field does not get populated when we have added document with the TTL defined in the TTL field. Like with following query, Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/
Re: Solr Document expiration with TTL
Hi
Thanks for your quick reply.
since your time_to_live_s and expire_at_dt fields are both
stored, can you confirm that a expire_at_dt field is getting popularted by
the update processor by doing as simple query for your doc (ie
q=id:10seconds)
No, expire_at_dt field does not get populated when we have added document
with the TTL defined in the TTL field. Like with following query,
curl -X POST -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/update?commit=true' -d
'[{id:10seconds,time_to_live_s:+10SECONDS}]'
and when document retrieved, it gives following result (Can see that
expire_at_dt field is not showing at all).
curl -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/select?q=id:10secondswt=jsonindent=true
'
{
responseHeader:{
status:0,
QTime:19,
params:{
indent:true,
q:id:10seconds,
wt:json}},
response:{numFound:1,start:0,docs:[
{
id:10seconds,
time_to_live_s:+10SECONDS,
_version_:1494171978430414848}]
}}
While if document is added with the TTL value defined explicitly in
expire_at_dt field, like,
curl -X POST -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/update?commit=true' -d
'[{id:10seconds,expire_at_dt:NOW+10SECONDS}]'
We can see the document with expire_at_dt field populated.
curl -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/select?q=id:10secondswt=jsonindent=true
'
{
responseHeader:{
status:0,
QTime:2,
params:{
indent:true,
q:id:10seconds,
wt:json}},
response:{numFound:1,start:0,docs:[
{
id:10seconds,
expire_at_dt:2015-02-26T12:27:31.983Z,
_version_:1494172190095966208}]
}}
Thanks,
Makailol
On Wed, Feb 25, 2015 at 10:00 PM, Chris Hostetter hossman_luc...@fucit.org
wrote:
: Following query posts a document and sets expire_at_dt explicitly. That
: is working perfectly ok and ducument expires at defined time.
so the delete trigge logic is working correctly...
: But when trying to post with TTL (following query), document does not
: expire after given time.
...which suggests that the TTL-expire_at logic is not being applied
properly.
which is weird.
since your time_to_live_s and expire_at_dt fields are both
stored, can you confirm that a expire_at_dt field is getting popularted by
the update processor by doing as simple query for your doc (ie
q=id:10seconds)
(either way: i can't explain why it's not getting deleted, but it would
help narrow down where the problem is)
-Hoss
http://www.lucidworks.com/
Re: Solr Document expiration with TTL
If your expire_at_dt field is not populated automatically, let's step back and recheck a sanity setting. You said it is a managed schema? Is it a schemaless as well? With an explicit processor chain? If that's the case, your default chain may not be running AT ALL. So, recheck your solrconfig.xml. Or add another explicit field population inside the chain, just like the example did with TimestampUpdateProcessorFactory : https://lucidworks.com/blog/document-expiration/ Regards, Alex. On 26 February 2015 at 07:52, Makailol Charls 4extrama...@gmail.com wrote: since your time_to_live_s and expire_at_dt fields are both stored, can you confirm that a expire_at_dt field is getting popularted by the update processor by doing as simple query for your doc (ie q=id:10seconds) No, expire_at_dt field does not get populated when we have added document with the TTL defined in the TTL field. Like with following query, Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/
Re: Solr Document expiration with TTL
Hi Alex,
Thanks for the reply.
Yes, we have already tried to set the autoDeletePeriodSeconds period to
some low value like 5 seconds and tried checking the document expiration
after 30 seconds or minute or even after an hour. But result is same and
document does not get expired automatically.
Thanks,
Makailol
On Thu, Feb 26, 2015 at 6:22 PM, Makailol Charls 4extrama...@gmail.com
wrote:
Hi
Thanks for your quick reply.
since your time_to_live_s and expire_at_dt fields are both
stored, can you confirm that a expire_at_dt field is getting popularted by
the update processor by doing as simple query for your doc (ie
q=id:10seconds)
No, expire_at_dt field does not get populated when we have added document
with the TTL defined in the TTL field. Like with following query,
curl -X POST -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/update?commit=true' -d
'[{id:10seconds,time_to_live_s:+10SECONDS}]'
and when document retrieved, it gives following result (Can see that
expire_at_dt field is not showing at all).
curl -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/select?q=id:10secondswt=jsonindent=true
'
{
responseHeader:{
status:0,
QTime:19,
params:{
indent:true,
q:id:10seconds,
wt:json}},
response:{numFound:1,start:0,docs:[
{
id:10seconds,
time_to_live_s:+10SECONDS,
_version_:1494171978430414848}]
}}
While if document is added with the TTL value defined explicitly in
expire_at_dt field, like,
curl -X POST -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/update?commit=true' -d
'[{id:10seconds,expire_at_dt:NOW+10SECONDS}]'
We can see the document with expire_at_dt field populated.
curl -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/select?q=id:10secondswt=jsonindent=true
'
{
responseHeader:{
status:0,
QTime:2,
params:{
indent:true,
q:id:10seconds,
wt:json}},
response:{numFound:1,start:0,docs:[
{
id:10seconds,
expire_at_dt:2015-02-26T12:27:31.983Z,
_version_:1494172190095966208}]
}}
Thanks,
Makailol
On Wed, Feb 25, 2015 at 10:00 PM, Chris Hostetter
hossman_luc...@fucit.org wrote:
: Following query posts a document and sets expire_at_dt explicitly.
That
: is working perfectly ok and ducument expires at defined time.
so the delete trigge logic is working correctly...
: But when trying to post with TTL (following query), document does not
: expire after given time.
...which suggests that the TTL-expire_at logic is not being applied
properly.
which is weird.
since your time_to_live_s and expire_at_dt fields are both
stored, can you confirm that a expire_at_dt field is getting popularted by
the update processor by doing as simple query for your doc (ie
q=id:10seconds)
(either way: i can't explain why it's not getting deleted, but it would
help narrow down where the problem is)
-Hoss
http://www.lucidworks.com/
Re: Solr Document expiration with TTL
: If your expire_at_dt field is not populated automatically, let's step : back and recheck a sanity setting. You said it is a managed schema? Is : it a schemaless as well? With an explicit processor chain? If that's : the case, your default chain may not be running AT ALL. yeah ... my only guess here is that even though you posted before that you had this configured in your defaut chain... processor class=solr.processor.DocExpirationUpdateProcessorFactory int name=autoDeletePeriodSeconds30/int str name=ttlFieldNametime_to_live_s/str str name=expirationFieldNameexpire_at_dt/str /processor ...perhaps you have an update.chain=foo type default param configured for your /update handler? * what does your /update requestHandler/ config look like? * are you using the new initParams/ feature of solr? what does it's config look like? : So, recheck your solrconfig.xml. Or add another explicit field : population inside the chain, just like the example did with : TimestampUpdateProcessorFactory : : https://lucidworks.com/blog/document-expiration/ yeah ... that would help as a sanity check as well ... point is: we need to verify which chain you are using when adding the doc. : : : : : Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: : http://www.solr-start.com/ : -Hoss http://www.lucidworks.com/
Re: Solr Document expiration with TTL
Reading https://lucidworks.com/blog/document-expiration/
It seems that your Delete check interval granularity is 30 seconds,
but your TTL is 10 seconds. Have you tried setting
autoDeletePeriodSeconds to something like 2 seconds and seeing if the
problem goes away due to more frequent checking of items to delete?
Also, even with the current setup, you should be observing the record
being deleted if not 10 seconds after than 30 seconds. Are you seeing
it not deleted at all?
Regards,
Alex.
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 25 February 2015 at 01:51, Makailol Charls 4extrama...@gmail.com wrote:
Hello,
We are trying to add documents in solr with ttl defined(document expiration
feature), which is expected to expire at specified time, but it is not.
Following are the settings we have defined in solrconfig.xml and
managed-schema.
solr version : 5.0.0
*solrconfig.xml*
---
updateRequestProcessorChain default=true
processor class=solr.processor.DocExpirationUpdateProcessorFactory
int name=autoDeletePeriodSeconds30/int
str name=ttlFieldNametime_to_live_s/str
str name=expirationFieldNameexpire_at_dt/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
*managed-schema*
---
field name=id type=string indexed=true stored=true
multiValued=false /
field name=time_to_live_s type=string stored=true
multiValued=false /
field name=expire_at_dt type=date stored=true
multiValued=false /
*solr query*
Following query posts a document and sets expire_at_dt explicitly. That
is working perfectly ok and ducument expires at defined time.
curl -X POST -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/update?commit=true' -d '[{
id:10seconds,expire_at_dt:NOW+10SECONDS}]'
But when trying to post with TTL (following query), document does not
expire after given time.
curl -X POST -H 'Content-Type: application/json' '
http://localhost:8983/solr/collection1/update?commit=true' -d '[{
id:10seconds,time_to_live_s:+10SECONDS}]'
Any help would be appreciated.
Thanks,
Makailol
Re: Solr Document expiration with TTL
: Following query posts a document and sets expire_at_dt explicitly. That : is working perfectly ok and ducument expires at defined time. so the delete trigge logic is working correctly... : But when trying to post with TTL (following query), document does not : expire after given time. ...which suggests that the TTL-expire_at logic is not being applied properly. which is weird. since your time_to_live_s and expire_at_dt fields are both stored, can you confirm that a expire_at_dt field is getting popularted by the update processor by doing as simple query for your doc (ie q=id:10seconds) (either way: i can't explain why it's not getting deleted, but it would help narrow down where the problem is) -Hoss http://www.lucidworks.com/
