[jira] [Updated] (SOLR-7372) Limit LRUCache by RAM usage
[ https://issues.apache.org/jira/browse/SOLR-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-7372: Attachment: SOLR-7372.patch # I renamed the maxRamBytes parameter to maxRamMB so that people don't have to convert size in megabytes and gigabytes from bytes. # I also added the description of maxRamMB parameter for LRUCache in solrconfig.xml (used only for query result cache). I think this is ready. Limit LRUCache by RAM usage --- Key: SOLR-7372 URL: https://issues.apache.org/jira/browse/SOLR-7372 Project: Solr Issue Type: Improvement Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: Trunk, 5.2 Attachments: SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch Now that SOLR-7371 has made DocSet impls Accountable, we should add an option to LRUCache to limit itself by RAM. I propose to add a 'maxRamBytes' configuration parameter which it can use to evict items once the total RAM usage of the cache reaches this limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7372) Limit LRUCache by RAM usage
[ https://issues.apache.org/jira/browse/SOLR-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490770#comment-14490770 ] ASF subversion and git services commented on SOLR-7372: --- Commit 1672811 from sha...@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1672811 ] SOLR-7372: Limit memory consumed by LRUCache with a new 'maxRamMB' config parameter Limit LRUCache by RAM usage --- Key: SOLR-7372 URL: https://issues.apache.org/jira/browse/SOLR-7372 Project: Solr Issue Type: Improvement Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: Trunk, 5.2 Attachments: SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch Now that SOLR-7371 has made DocSet impls Accountable, we should add an option to LRUCache to limit itself by RAM. I propose to add a 'maxRamBytes' configuration parameter which it can use to evict items once the total RAM usage of the cache reaches this limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7372) Limit LRUCache by RAM usage
[ https://issues.apache.org/jira/browse/SOLR-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490771#comment-14490771 ] ASF subversion and git services commented on SOLR-7372: --- Commit 1672812 from sha...@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1672812 ] SOLR-7372: Limit memory consumed by LRUCache with a new 'maxRamMB' config parameter Limit LRUCache by RAM usage --- Key: SOLR-7372 URL: https://issues.apache.org/jira/browse/SOLR-7372 Project: Solr Issue Type: Improvement Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: Trunk, 5.2 Attachments: SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch Now that SOLR-7371 has made DocSet impls Accountable, we should add an option to LRUCache to limit itself by RAM. I propose to add a 'maxRamBytes' configuration parameter which it can use to evict items once the total RAM usage of the cache reaches this limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7372) Limit LRUCache by RAM usage
[ https://issues.apache.org/jira/browse/SOLR-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-7372. - Resolution: Fixed Thanks for the review, Yonik! Limit LRUCache by RAM usage --- Key: SOLR-7372 URL: https://issues.apache.org/jira/browse/SOLR-7372 Project: Solr Issue Type: Improvement Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: Trunk, 5.2 Attachments: SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch Now that SOLR-7371 has made DocSet impls Accountable, we should add an option to LRUCache to limit itself by RAM. I propose to add a 'maxRamBytes' configuration parameter which it can use to evict items once the total RAM usage of the cache reaches this limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7372) Limit LRUCache by RAM usage
[ https://issues.apache.org/jira/browse/SOLR-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-7372. - Resolution: Fixed Limit LRUCache by RAM usage --- Key: SOLR-7372 URL: https://issues.apache.org/jira/browse/SOLR-7372 Project: Solr Issue Type: Improvement Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: Trunk, 5.2 Attachments: SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch Now that SOLR-7371 has made DocSet impls Accountable, we should add an option to LRUCache to limit itself by RAM. I propose to add a 'maxRamBytes' configuration parameter which it can use to evict items once the total RAM usage of the cache reaches this limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Reopened] (SOLR-7372) Limit LRUCache by RAM usage
[ https://issues.apache.org/jira/browse/SOLR-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar reopened SOLR-7372: - Re-opening due to Java7 incompatibility on 5x Limit LRUCache by RAM usage --- Key: SOLR-7372 URL: https://issues.apache.org/jira/browse/SOLR-7372 Project: Solr Issue Type: Improvement Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: Trunk, 5.2 Attachments: SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch Now that SOLR-7371 has made DocSet impls Accountable, we should add an option to LRUCache to limit itself by RAM. I propose to add a 'maxRamBytes' configuration parameter which it can use to evict items once the total RAM usage of the cache reaches this limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7372) Limit LRUCache by RAM usage
[ https://issues.apache.org/jira/browse/SOLR-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490777#comment-14490777 ] ASF subversion and git services commented on SOLR-7372: --- Commit 1672815 from sha...@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1672815 ] SOLR-7372: Fix Java7 compatibility Limit LRUCache by RAM usage --- Key: SOLR-7372 URL: https://issues.apache.org/jira/browse/SOLR-7372 Project: Solr Issue Type: Improvement Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: Trunk, 5.2 Attachments: SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch Now that SOLR-7371 has made DocSet impls Accountable, we should add an option to LRUCache to limit itself by RAM. I propose to add a 'maxRamBytes' configuration parameter which it can use to evict items once the total RAM usage of the cache reaches this limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7377) SOLR Streaming Expressions
Dennis Gove created SOLR-7377:
-
Summary: SOLR Streaming Expressions
Key: SOLR-7377
URL: https://issues.apache.org/jira/browse/SOLR-7377
Project: Solr
Issue Type: Improvement
Components: clients - java
Reporter: Dennis Gove
Priority: Minor
Fix For: Trunk
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-6526. Right now that API requires streaming requests to come
in from clients as serialized bytecode of the streaming classes. The suggestion
here is to support string expressions which describe the streaming operations
the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
over=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build their
own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams as
arguments to another stream)
4. Positional parameters
The main goal here is to make streaming as accessible as possible and define a
syntax for running complex queries across large distributed systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5579) Spatial, enhance RPT to differentiate confirmed from non-confirmed hits, then validate with SDV
[ https://issues.apache.org/jira/browse/LUCENE-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490251#comment-14490251 ] ASF subversion and git services commented on LUCENE-5579: - Commit 1672736 from [~dsmiley] in branch 'dev/trunk' [ https://svn.apache.org/r1672736 ] LUCENE-5579: CompositeSpatialStrategy (RPT + SDV) with optimized Intersect Spatial, enhance RPT to differentiate confirmed from non-confirmed hits, then validate with SDV --- Key: LUCENE-5579 URL: https://issues.apache.org/jira/browse/LUCENE-5579 Project: Lucene - Core Issue Type: New Feature Components: modules/spatial Reporter: David Smiley Assignee: David Smiley Attachments: LUCENE-5579_CompositeSpatialStrategy.patch, LUCENE-5579_CompositeSpatialStrategy.patch, LUCENE-5579_SPT_leaf_covered.patch, spatial.alg If a cell is within the query shape (doesn't straddle the edge), then you can be sure that all documents it matches are a confirmed hit. But if some documents are only on the edge cells, then those documents could be validated against SerializedDVStrategy for precise spatial search. This should be *much* faster than using RPT and SerializedDVStrategy independently on the same search, particularly when a lot of documents match. Perhaps this'll be a new RPT subclass, or maybe an optional configuration of RPT. This issue is just for the Intersects predicate, which will apply to Disjoint. Until resolved in other issues, the other predicates can be handled in a naive/slow way by creating a filter that combines RPT's filter and SerializedDVStrategy's filter using BitsFilteredDocIdSet. One thing I'm not sure of is how to expose to Lucene-spatial users the underlying functionality such that they can put other query/filters in-between RPT and the SerializedDVStrategy. Maybe that'll be done by simply ensuring the predicate filters have this capability and are public. It would be ideal to implement this capability _after_ the PrefixTree term encoding is modified to differentiate edge leaf-cells from non-edge leaf cells. This distinction will allow the code here to make more confirmed matches. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7377) SOLR Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-7377:
--
Attachment: SOLR-7377.patch
First-pass patch. Looking for initial feedback.
SOLR Streaming Expressions
--
Key: SOLR-7377
URL: https://issues.apache.org/jira/browse/SOLR-7377
Project: Solr
Issue Type: Improvement
Components: clients - java
Reporter: Dennis Gove
Priority: Minor
Fix For: Trunk
Attachments: SOLR-7377.patch
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-6526. Right now that API requires streaming requests to
come in from clients as serialized bytecode of the streaming classes. The
suggestion here is to support string expressions which describe the streaming
operations the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
over=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build
their own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams as
arguments to another stream)
4. Positional parameters
The main goal here is to make streaming as accessible as possible and define
a syntax for running complex queries across large distributed systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5579) Spatial, enhance RPT to differentiate confirmed from non-confirmed hits, then validate with SDV
[ https://issues.apache.org/jira/browse/LUCENE-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490264#comment-14490264 ] ASF subversion and git services commented on LUCENE-5579: - Commit 1672740 from [~dsmiley] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1672740 ] LUCENE-5579: CompositeSpatialStrategy (RPT + SDV) with optimized Intersect Spatial, enhance RPT to differentiate confirmed from non-confirmed hits, then validate with SDV --- Key: LUCENE-5579 URL: https://issues.apache.org/jira/browse/LUCENE-5579 Project: Lucene - Core Issue Type: New Feature Components: modules/spatial Reporter: David Smiley Assignee: David Smiley Attachments: LUCENE-5579_CompositeSpatialStrategy.patch, LUCENE-5579_CompositeSpatialStrategy.patch, LUCENE-5579_SPT_leaf_covered.patch, spatial.alg If a cell is within the query shape (doesn't straddle the edge), then you can be sure that all documents it matches are a confirmed hit. But if some documents are only on the edge cells, then those documents could be validated against SerializedDVStrategy for precise spatial search. This should be *much* faster than using RPT and SerializedDVStrategy independently on the same search, particularly when a lot of documents match. Perhaps this'll be a new RPT subclass, or maybe an optional configuration of RPT. This issue is just for the Intersects predicate, which will apply to Disjoint. Until resolved in other issues, the other predicates can be handled in a naive/slow way by creating a filter that combines RPT's filter and SerializedDVStrategy's filter using BitsFilteredDocIdSet. One thing I'm not sure of is how to expose to Lucene-spatial users the underlying functionality such that they can put other query/filters in-between RPT and the SerializedDVStrategy. Maybe that'll be done by simply ensuring the predicate filters have this capability and are public. It would be ideal to implement this capability _after_ the PrefixTree term encoding is modified to differentiate edge leaf-cells from non-edge leaf cells. This distinction will allow the code here to make more confirmed matches. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-5579) Spatial, enhance RPT to differentiate confirmed from non-confirmed hits, then validate with SDV
[ https://issues.apache.org/jira/browse/LUCENE-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley resolved LUCENE-5579. -- Resolution: Fixed Fix Version/s: 5.2 Lucene Fields: New,Patch Available (was: New) Spatial, enhance RPT to differentiate confirmed from non-confirmed hits, then validate with SDV --- Key: LUCENE-5579 URL: https://issues.apache.org/jira/browse/LUCENE-5579 Project: Lucene - Core Issue Type: New Feature Components: modules/spatial Reporter: David Smiley Assignee: David Smiley Fix For: 5.2 Attachments: LUCENE-5579_CompositeSpatialStrategy.patch, LUCENE-5579_CompositeSpatialStrategy.patch, LUCENE-5579_SPT_leaf_covered.patch, spatial.alg If a cell is within the query shape (doesn't straddle the edge), then you can be sure that all documents it matches are a confirmed hit. But if some documents are only on the edge cells, then those documents could be validated against SerializedDVStrategy for precise spatial search. This should be *much* faster than using RPT and SerializedDVStrategy independently on the same search, particularly when a lot of documents match. Perhaps this'll be a new RPT subclass, or maybe an optional configuration of RPT. This issue is just for the Intersects predicate, which will apply to Disjoint. Until resolved in other issues, the other predicates can be handled in a naive/slow way by creating a filter that combines RPT's filter and SerializedDVStrategy's filter using BitsFilteredDocIdSet. One thing I'm not sure of is how to expose to Lucene-spatial users the underlying functionality such that they can put other query/filters in-between RPT and the SerializedDVStrategy. Maybe that'll be done by simply ensuring the predicate filters have this capability and are public. It would be ideal to implement this capability _after_ the PrefixTree term encoding is modified to differentiate edge leaf-cells from non-edge leaf cells. This distinction will allow the code here to make more confirmed matches. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6399) Benchmark's AbstractQueryMaker.resetInputs should call setConfig
[ https://issues.apache.org/jira/browse/LUCENE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490277#comment-14490277 ] ASF subversion and git services commented on LUCENE-6399: - Commit 1672742 from [~dsmiley] in branch 'dev/trunk' [ https://svn.apache.org/r1672742 ] LUCENE-6399: Benchmark's QueryMaker.resetInputs should call setConfig Benchmark's AbstractQueryMaker.resetInputs should call setConfig Key: LUCENE-6399 URL: https://issues.apache.org/jira/browse/LUCENE-6399 Project: Lucene - Core Issue Type: Improvement Components: modules/benchmark Reporter: David Smiley Assignee: David Smiley Priority: Minor Fix For: 5.2 Attachments: LUCENE-6399_Benchmark_QueryMaker_resetInput_setConfig.patch DocMaker.resetInput() will call setConfig. QueryMaker should too for the same reason -- so that it can respond to properties that change per round. DocMaker is concrete but QueryMaker is an interface, but this behavior can be put into AbstractQueryMaker. I found this as some benchmarking was driving me crazy as I couldn't get spatial benchmark queries to see changes I had! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6399) Benchmark's AbstractQueryMaker.resetInputs should call setConfig
[ https://issues.apache.org/jira/browse/LUCENE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490283#comment-14490283 ] ASF subversion and git services commented on LUCENE-6399: - Commit 1672744 from [~dsmiley] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1672744 ] LUCENE-6399: Benchmark's QueryMaker.resetInputs should call setConfig Benchmark's AbstractQueryMaker.resetInputs should call setConfig Key: LUCENE-6399 URL: https://issues.apache.org/jira/browse/LUCENE-6399 Project: Lucene - Core Issue Type: Improvement Components: modules/benchmark Reporter: David Smiley Assignee: David Smiley Priority: Minor Fix For: 5.2 Attachments: LUCENE-6399_Benchmark_QueryMaker_resetInput_setConfig.patch DocMaker.resetInput() will call setConfig. QueryMaker should too for the same reason -- so that it can respond to properties that change per round. DocMaker is concrete but QueryMaker is an interface, but this behavior can be put into AbstractQueryMaker. I found this as some benchmarking was driving me crazy as I couldn't get spatial benchmark queries to see changes I had! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7361) Main Jetty thread blocked by core loading delays HTTP listener from binding if core loading is slow
[
https://issues.apache.org/jira/browse/SOLR-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490285#comment-14490285
]
Timothy Potter commented on SOLR-7361:
--
heh - now tons of tests fail because the main thread isn't blocked and finishes
before cores are initialized ... knee-jerk reaction would be to activate this
true async loading method using a System property, e.g.
solr.cloud.loadCoresAsync, which defaults to true but is set to false in most
tests ... other ideas?
Main Jetty thread blocked by core loading delays HTTP listener from binding
if core loading is slow
---
Key: SOLR-7361
URL: https://issues.apache.org/jira/browse/SOLR-7361
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Timothy Potter
Assignee: Timothy Potter
During server startup, the CoreContainer uses an ExecutorService to load
cores in multiple back-ground threads but then blocks until cores are loaded,
see: CoreContainer#load around line 290 on trunk (invokeAll). From the
JavaDoc on that method, we have:
{quote}
Executes the given tasks, returning a list of Futures holding their status
and results when all complete. Future.isDone() is true for each element of
the returned list.
{quote}
In other words, this is a blocking call.
This delays the Jetty HTTP listener from binding and accepting requests until
all cores are loaded. Do we need to block the main thread?
Also, prior to this happening, the node is registered as a live node in ZK,
which makes it a candidate for receiving requests from the Overseer, such as
to service a create collection request. The problem of course is that the
node listed in /live_nodes isn't accepting requests yet. So we either need to
unblock the main thread during server loading or maybe wait longer before we
register as a live node ... not sure which is the better way forward?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5879) Add auto-prefix terms to block tree terms dict
[ https://issues.apache.org/jira/browse/LUCENE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490288#comment-14490288 ] ASF subversion and git services commented on LUCENE-5879: - Commit 1672749 from [~mikemccand] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1672749 ] LUCENE-5879: turn off too-slow term range checking for now Add auto-prefix terms to block tree terms dict -- Key: LUCENE-5879 URL: https://issues.apache.org/jira/browse/LUCENE-5879 Project: Lucene - Core Issue Type: New Feature Components: core/codecs Reporter: Michael McCandless Assignee: Michael McCandless Fix For: Trunk, 5.2 Attachments: LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch This cool idea to generalize numeric/trie fields came from Adrien: Today, when we index a numeric field (LongField, etc.) we pre-compute (via NumericTokenStream) outside of indexer/codec which prefix terms should be indexed. But this can be inefficient: you set a static precisionStep, and always add those prefix terms regardless of how the terms in the field are actually distributed. Yet typically in real world applications the terms have a non-random distribution. So, it should be better if instead the terms dict decides where it makes sense to insert prefix terms, based on how dense the terms are in each region of term space. This way we can speed up query time for both term (e.g. infix suggester) and numeric ranges, and it should let us use less index space and get faster range queries. This would also mean that min/maxTerm for a numeric field would now be correct, vs today where the externally computed prefix terms are placed after the full precision terms, causing hairy code like NumericUtils.getMaxInt/Long. So optos like LUCENE-5860 become feasible. The terms dict can also do tricks not possible if you must live on top of its APIs, e.g. to handle the adversary/over-constrained case when a given prefix has too many terms following it but finer prefixes have too few (what block tree calls floor term blocks). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-6399) Benchmark's AbstractQueryMaker.resetInputs should call setConfig
[ https://issues.apache.org/jira/browse/LUCENE-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley resolved LUCENE-6399. -- Resolution: Fixed Benchmark's AbstractQueryMaker.resetInputs should call setConfig Key: LUCENE-6399 URL: https://issues.apache.org/jira/browse/LUCENE-6399 Project: Lucene - Core Issue Type: Improvement Components: modules/benchmark Reporter: David Smiley Assignee: David Smiley Priority: Minor Fix For: 5.2 Attachments: LUCENE-6399_Benchmark_QueryMaker_resetInput_setConfig.patch DocMaker.resetInput() will call setConfig. QueryMaker should too for the same reason -- so that it can respond to properties that change per round. DocMaker is concrete but QueryMaker is an interface, but this behavior can be put into AbstractQueryMaker. I found this as some benchmarking was driving me crazy as I couldn't get spatial benchmark queries to see changes I had! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6820) The sync on the VersionInfo bucket in DistributedUpdateProcesser#addDocument appears to be a large bottleneck when using replication.
[
https://issues.apache.org/jira/browse/SOLR-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490307#comment-14490307
]
Timothy Potter commented on SOLR-6820:
--
Thanks for the nice description of this Yonik! I think this thread dump shows
the problem nicely:
{code}
recoveryExecutor-7-thread-1 prio=10 tid=0x7f0fe821e800 nid=0xbafc
runnable [0x7f1003c3]
java.lang.Thread.State: RUNNABLE
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:440)
at
org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:111)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:351)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:545)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
- locked 0x0005d9e96768 (a
org.apache.lucene.index.BufferedUpdatesStream)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3238)
- locked 0x0005d9a4f2a8 (a org.apache.solr.update.SolrIndexWriter)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3229)
- locked 0x0005d9a4f2a8 (a org.apache.solr.update.SolrIndexWriter)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:384)
- locked 0x0005d9a4f2a8 (a org.apache.solr.update.SolrIndexWriter)
- locked 0x0005dc1943a8 (a java.lang.Object)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:274)
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1465)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:424)
- locked 0x0005dd011ab8 (a org.apache.solr.update.UpdateLog)
at
org.apache.solr.update.DirectUpdateHandler2.addAndDelete(DirectUpdateHandler2.java:449)
- locked 0x0005dc1943d8 (a java.lang.Object)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:216)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:928)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082)
- locked 0x0005c2b560d0 (a org.apache.solr.update.VersionBucket)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:695)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
at
org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1343)
at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1217)
...
{code}
and
{code}
recoveryExecutor-7-thread-2 prio=10 tid=0x7ec5b4003000 nid=0xc131 waiting
for monitor entry [0x7f1003aab000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:999)
- waiting to lock 0x0005c2b560d0 (a
org.apache.solr.update.VersionBucket)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:695)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
at
org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1343)
at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1217)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
...
{code}
The sync on the VersionInfo bucket in DistributedUpdateProcesser#addDocument
appears to be a large bottleneck when using replication.
-
Key: SOLR-6820
URL:
[jira] [Created] (SOLR-7378) Be more conservative about loading a core when hdfs transaction log could not be recovered
Gregory Chanan created SOLR-7378:
Summary: Be more conservative about loading a core when hdfs
transaction log could not be recovered
Key: SOLR-7378
URL: https://issues.apache.org/jira/browse/SOLR-7378
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 5.0
Reporter: Gregory Chanan
Today, if an HdfsTransactionLog cannot recover its lease, you get the following
warning in the log:
{code}
log.warn(Cannot recoverLease after trying for +
conf.getInt(solr.hdfs.lease.recovery.timeout, 90) +
ms (solr.hdfs.lease.recovery.timeout); continuing, but may be
DATALOSS!!!; +
getLogMessageDetail(nbAttempt, p, startWaiting));
{code}
from:
https://github.com/apache/lucene-solr/blob/a8c24b7f02d4e4c172926d04654bcc007f6c29d2/solr/core/src/java/org/apache/solr/util/FSHDFSUtils.java#L145-L148
But some deployments may not actually want to continue if there is potential
data loss, they may want to investigate what the underlying issue is with HDFS
first. And there's no way outside of looking at the logs to figure out what is
going on.
There's a range of possibilties here, but here's a couple of ideas:
1) config parameter around whether to continue with potential data loss or not
2) load but require special flag to read potentially incorrect data (similar to
shards.tolerant, data.tolerant or something?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7377) SOLR Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-7377:
--
Description:
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-6526. Right now that API requires streaming requests to come
in from clients as serialized bytecode of the streaming classes. The suggestion
here is to support string expressions which describe the streaming operations
the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
sort=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build their
own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams as
arguments to another stream)
4. Positional parameters
The main goal here is to make streaming as accessible as possible and define a
syntax for running complex queries across large distributed systems.
was:
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-6526. Right now that API requires streaming requests to come
in from clients as serialized bytecode of the streaming classes. The suggestion
here is to support string expressions which describe the streaming operations
the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
over=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build their
own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams as
arguments to another stream)
4. Positional parameters
The main goal here is to make streaming as accessible as possible and define a
syntax for running complex queries across large distributed systems.
SOLR Streaming Expressions
--
Key: SOLR-7377
URL: https://issues.apache.org/jira/browse/SOLR-7377
Project: Solr
Issue Type: Improvement
Components: clients - java
Reporter: Dennis Gove
Priority: Minor
Fix For: Trunk
Attachments: SOLR-7377.patch
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-6526. Right now that API requires streaming requests to
come in from clients as serialized bytecode of the streaming classes. The
suggestion here is to support string expressions which describe the streaming
operations the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
sort=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build
their own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams
[JENKINS] Lucene-Solr-5.1-Linux (64bit/jdk1.8.0_60-ea-b06) - Build # 238 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.1-Linux/238/ Java: 64bit/jdk1.8.0_60-ea-b06 -XX:-UseCompressedOops -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test Error Message: There were too many update fails (36 20) - we expect it can happen, but shouldn't easily Stack Trace: java.lang.AssertionError: There were too many update fails (36 20) - we expect it can happen, but shouldn't easily at __randomizedtesting.SeedInfo.seed([9D981D88E966EBE7:15CC2252479A861F]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertFalse(Assert.java:68) at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:230) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at
[jira] [Commented] (SOLR-7370) FSHDFSUtils#recoverFileLease tries to recover the lease every one second after the first four second wait.
[ https://issues.apache.org/jira/browse/SOLR-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490314#comment-14490314 ] Mark Miller commented on SOLR-7370: --- Yeah, it will before I close it. FSHDFSUtils#recoverFileLease tries to recover the lease every one second after the first four second wait. -- Key: SOLR-7370 URL: https://issues.apache.org/jira/browse/SOLR-7370 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Attachments: SOLR-7370.patch It should be trying much less frequently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7379) CompositeSpatialStrategy FieldType
David Smiley created SOLR-7379: -- Summary: CompositeSpatialStrategy FieldType Key: SOLR-7379 URL: https://issues.apache.org/jira/browse/SOLR-7379 Project: Solr Issue Type: New Feature Components: spatial Reporter: David Smiley Assignee: David Smiley Fix For: 5.2 We should add a Solr spatial FieldType that uses the new CompositeSpatialStrategy. Essentially this enabled geometry backed shapes for accuracy combined with the grid indexes of RPT for speed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7379) CompositeSpatialStrategy FieldType
[ https://issues.apache.org/jira/browse/SOLR-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490375#comment-14490375 ] David Smiley commented on SOLR-7379: This one will be a tad more interesting than simply extending the abstract spatial field type as it should have these features: * reference RPT field type via dynamicField pattern or no asterisk for explicitly referencing an existing field. createField should probably route through the other field Type (like a kind of internal copyField) or maybe it's overkill and could simply call the createField on the underlying Strategy. ** Come to think of it, BBoxField PointVector ought to be able to work similarly (reference field completely instead of by pattern) but that's not part of this issue. ** note: SDV (SerializedDVStrategy) probably doesn't deserve it's own field type but we can internally create one. * SerializedDVStrategy could be subclassed to override makeShapeValueSource to be backed by a SolrCache. ** We could cast to JtsGeometry if possible to call index() which uses JTS PreparedGeometry to speedup the checks. When it first puts it into the cache it might not do this but on successfull fetch from the cache it could do this -- which has no effect if it's already cached. It's thread-safe. CompositeSpatialStrategy FieldType -- Key: SOLR-7379 URL: https://issues.apache.org/jira/browse/SOLR-7379 Project: Solr Issue Type: New Feature Components: spatial Reporter: David Smiley Assignee: David Smiley Fix For: 5.2 We should add a Solr spatial FieldType that uses the new CompositeSpatialStrategy. Essentially this enabled geometry backed shapes for accuracy combined with the grid indexes of RPT for speed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7370) FSHDFSUtils#recoverFileLease tries to recover the lease every one second after the first four second wait.
[ https://issues.apache.org/jira/browse/SOLR-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490384#comment-14490384 ] ASF subversion and git services commented on SOLR-7370: --- Commit 1672775 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1672775 ] SOLR-7370: FSHDFSUtils#recoverFileLease tries to recover the lease every one second after the first four second wait. FSHDFSUtils#recoverFileLease tries to recover the lease every one second after the first four second wait. -- Key: SOLR-7370 URL: https://issues.apache.org/jira/browse/SOLR-7370 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Attachments: SOLR-7370.patch It should be trying much less frequently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2925 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2925/ 1 tests failed. REGRESSION: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test Error Message: Captured an uncaught exception in thread: Thread[id=3623, name=Thread-1409, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3623, name=Thread-1409, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest] Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: java.net.SocketTimeoutException: Read timed out at __randomizedtesting.SeedInfo.seed([22BFDC2CAFBF0463]:0) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:639) Caused by: org.apache.solr.client.solrj.SolrServerException: java.net.SocketTimeoutException: Read timed out at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:929) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:782) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:636) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:462) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:233) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:225) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:370) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325) at org.apache.solr.client.solrj.impl.CloudSolrClient$2.call(CloudSolrClient.java:596) at org.apache.solr.client.solrj.impl.CloudSolrClient$2.call(CloudSolrClient.java:593) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 10076 lines...] [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest [junit4] 2 Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 22BFDC2CAFBF0463-001/init-core-data-001 [junit4] 2 1313761 T3112 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /gx_se/ej [junit4] 2 1313773 T3112
[jira] [Commented] (SOLR-7361) Main Jetty thread blocked by core loading delays HTTP listener from binding if core loading is slow
[
https://issues.apache.org/jira/browse/SOLR-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490391#comment-14490391
]
Shawn Heisey commented on SOLR-7361:
I like the idea of having access to the admin UI responsive immediately on
startup ... if nothing else, to be able to see the progress of the startup.
Obviously any requests to cores that are not yet started must fail, but cores
that have started, as well as any functionality that doesn't depend on core
startup, should work.
There's probably more to think about and tweak. During startup, CoreAdmin and
CollectionsAdmin requests will require careful vetting. My thought here is
that about the only operation that should be allowed on things that haven't yet
started is DELETE, if we can find a way to do so safely. If deleting a core
can remove it from the startup queue entirely, it's a slight optimization.
If we aren't already doing so, this will require that we enumerate all the
cores and/or the clusterstate before listening to any kind of request on
CoreAdmin/CollectionsAdmin.
A disabling property for tests is a good interim step, but I think that the
long term goal should be to modify both the async startup and the tests so
everything works right.
Main Jetty thread blocked by core loading delays HTTP listener from binding
if core loading is slow
---
Key: SOLR-7361
URL: https://issues.apache.org/jira/browse/SOLR-7361
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Timothy Potter
Assignee: Timothy Potter
During server startup, the CoreContainer uses an ExecutorService to load
cores in multiple back-ground threads but then blocks until cores are loaded,
see: CoreContainer#load around line 290 on trunk (invokeAll). From the
JavaDoc on that method, we have:
{quote}
Executes the given tasks, returning a list of Futures holding their status
and results when all complete. Future.isDone() is true for each element of
the returned list.
{quote}
In other words, this is a blocking call.
This delays the Jetty HTTP listener from binding and accepting requests until
all cores are loaded. Do we need to block the main thread?
Also, prior to this happening, the node is registered as a live node in ZK,
which makes it a candidate for receiving requests from the Overseer, such as
to service a create collection request. The problem of course is that the
node listed in /live_nodes isn't accepting requests yet. So we either need to
unblock the main thread during server loading or maybe wait longer before we
register as a live node ... not sure which is the better way forward?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7370) FSHDFSUtils#recoverFileLease tries to recover the lease every one second after the first four second wait.
[ https://issues.apache.org/jira/browse/SOLR-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller resolved SOLR-7370. --- Resolution: Fixed Fix Version/s: 5.2 Trunk FSHDFSUtils#recoverFileLease tries to recover the lease every one second after the first four second wait. -- Key: SOLR-7370 URL: https://issues.apache.org/jira/browse/SOLR-7370 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Fix For: Trunk, 5.2 Attachments: SOLR-7370.patch It should be trying much less frequently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7361) Main Jetty thread blocked by core loading delays HTTP listener from binding if core loading is slow
[
https://issues.apache.org/jira/browse/SOLR-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490404#comment-14490404
]
Timothy Potter commented on SOLR-7361:
--
Actually, if we go with the idea that non-cloud mode continues to block the
main thread until cores are loaded, then all tests pass. I think that makes
sense anyway, since there isn't anything like replica state in non-cloud mode
to determine if a core is active, so the listener not being up until cores are
loaded seems like a good thing for non-cloud mode. For cloud mode, it appears
that we don't need a flag for the tests to pass.
Main Jetty thread blocked by core loading delays HTTP listener from binding
if core loading is slow
---
Key: SOLR-7361
URL: https://issues.apache.org/jira/browse/SOLR-7361
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Timothy Potter
Assignee: Timothy Potter
During server startup, the CoreContainer uses an ExecutorService to load
cores in multiple back-ground threads but then blocks until cores are loaded,
see: CoreContainer#load around line 290 on trunk (invokeAll). From the
JavaDoc on that method, we have:
{quote}
Executes the given tasks, returning a list of Futures holding their status
and results when all complete. Future.isDone() is true for each element of
the returned list.
{quote}
In other words, this is a blocking call.
This delays the Jetty HTTP listener from binding and accepting requests until
all cores are loaded. Do we need to block the main thread?
Also, prior to this happening, the node is registered as a live node in ZK,
which makes it a candidate for receiving requests from the Overseer, such as
to service a create collection request. The problem of course is that the
node listed in /live_nodes isn't accepting requests yet. So we either need to
unblock the main thread during server loading or maybe wait longer before we
register as a live node ... not sure which is the better way forward?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7289) Tests should not ignore all leaking threads and instead just ignore the known leaking threads.
[ https://issues.apache.org/jira/browse/SOLR-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490436#comment-14490436 ] ASF subversion and git services commented on SOLR-7289: --- Commit 1672777 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1672777 ] SOLR-7289: Tests should not ignore all leaking threads and instead just ignore the known leaking threads. Tests should not ignore all leaking threads and instead just ignore the known leaking threads. -- Key: SOLR-7289 URL: https://issues.apache.org/jira/browse/SOLR-7289 Project: Solr Issue Type: Test Reporter: Mark Miller Assignee: Mark Miller Fix For: Trunk, 5.1 Attachments: SOLR-7289.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7361) Main Jetty thread blocked by core loading delays HTTP listener from binding if core loading is slow
[
https://issues.apache.org/jira/browse/SOLR-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490437#comment-14490437
]
Shawn Heisey commented on SOLR-7361:
If we can pull the same thing off in non-cloud mode, I would like that ... but
cloud mode is where it's a big problem.
Main Jetty thread blocked by core loading delays HTTP listener from binding
if core loading is slow
---
Key: SOLR-7361
URL: https://issues.apache.org/jira/browse/SOLR-7361
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Timothy Potter
Assignee: Timothy Potter
During server startup, the CoreContainer uses an ExecutorService to load
cores in multiple back-ground threads but then blocks until cores are loaded,
see: CoreContainer#load around line 290 on trunk (invokeAll). From the
JavaDoc on that method, we have:
{quote}
Executes the given tasks, returning a list of Futures holding their status
and results when all complete. Future.isDone() is true for each element of
the returned list.
{quote}
In other words, this is a blocking call.
This delays the Jetty HTTP listener from binding and accepting requests until
all cores are loaded. Do we need to block the main thread?
Also, prior to this happening, the node is registered as a live node in ZK,
which makes it a candidate for receiving requests from the Overseer, such as
to service a create collection request. The problem of course is that the
node listed in /live_nodes isn't accepting requests yet. So we either need to
unblock the main thread during server loading or maybe wait longer before we
register as a live node ... not sure which is the better way forward?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7377) SOLR Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Bernstein updated SOLR-7377:
-
Description:
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-7082. Right now that API requires streaming requests to come
in from clients as serialized bytecode of the streaming classes. The suggestion
here is to support string expressions which describe the streaming operations
the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
sort=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build their
own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams as
arguments to another stream)
4. Positional parameters
The main goal here is to make streaming as accessible as possible and define a
syntax for running complex queries across large distributed systems.
was:
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-6526. Right now that API requires streaming requests to come
in from clients as serialized bytecode of the streaming classes. The suggestion
here is to support string expressions which describe the streaming operations
the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
sort=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build their
own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams as
arguments to another stream)
4. Positional parameters
The main goal here is to make streaming as accessible as possible and define a
syntax for running complex queries across large distributed systems.
SOLR Streaming Expressions
--
Key: SOLR-7377
URL: https://issues.apache.org/jira/browse/SOLR-7377
Project: Solr
Issue Type: Improvement
Components: clients - java
Reporter: Dennis Gove
Priority: Minor
Fix For: Trunk
Attachments: SOLR-7377.patch
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-7082. Right now that API requires streaming requests to
come in from clients as serialized bytecode of the streaming classes. The
suggestion here is to support string expressions which describe the streaming
operations the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
sort=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build
their own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N
[jira] [Commented] (SOLR-7377) SOLR Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490451#comment-14490451
]
Joel Bernstein commented on SOLR-7377:
--
All I can say is awesome!
Just updated the description to reference SOLR-7082.
SOLR Streaming Expressions
--
Key: SOLR-7377
URL: https://issues.apache.org/jira/browse/SOLR-7377
Project: Solr
Issue Type: Improvement
Components: clients - java
Reporter: Dennis Gove
Priority: Minor
Fix For: Trunk
Attachments: SOLR-7377.patch
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-7082. Right now that API requires streaming requests to
come in from clients as serialized bytecode of the streaming classes. The
suggestion here is to support string expressions which describe the streaming
operations the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
sort=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build
their own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams as
arguments to another stream)
4. Positional parameters
The main goal here is to make streaming as accessible as possible and define
a syntax for running complex queries across large distributed systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5989) Add BinaryField, to index a single binary token
[ https://issues.apache.org/jira/browse/LUCENE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490469#comment-14490469 ] ASF subversion and git services commented on LUCENE-5989: - Commit 1672781 from [~mikemccand] in branch 'dev/trunk' [ https://svn.apache.org/r1672781 ] LUCENE-5989: allow passing BytesRef to StringField to make it easier to index arbitrary binary tokens Add BinaryField, to index a single binary token --- Key: LUCENE-5989 URL: https://issues.apache.org/jira/browse/LUCENE-5989 Project: Lucene - Core Issue Type: Improvement Reporter: Michael McCandless Assignee: Michael McCandless Fix For: 5.0, Trunk Attachments: LUCENE-5989.patch, LUCENE-5989.patch 5 years ago (LUCENE-1458) we enabled fully binary terms in the lowest levels of Lucene (the codec APIs) yet today, actually adding an arbitrary byte[] binary term during indexing is far from simple: you must make a custom Field with a custom TokenStream and a custom TermToBytesRefAttribute, as far as I know. This is supremely expert, I wonder if anyone out there has succeeded in doing so? I think we should make indexing a single byte[] as simple as indexing a single String. This is a pre-cursor for issues like LUCENE-5596 (encoding IPv6 address as byte[16]) and LUCENE-5879 (encoding native numeric values in their simple binary form). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7018) bin/solr status error after calling bin/solr stop against a cluster started using bin/solr -e cloud
[
https://issues.apache.org/jira/browse/SOLR-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490519#comment-14490519
]
Oliver Bates commented on SOLR-7018:
I think for people new to solr it may helpful to note why that behavior occurs
(i.e. stopping one node breaks the cluster). And the reason is that `bin/solr
-e cloud` uses an embedded zookeeper. Killing one node takes down zookeeper and
hence the solr cluster stop responding...right?
bin/solr status error after calling bin/solr stop against a cluster
started using bin/solr -e cloud
-
Key: SOLR-7018
URL: https://issues.apache.org/jira/browse/SOLR-7018
Project: Solr
Issue Type: Bug
Reporter: Steve Rowe
Assignee: Timothy Potter
Priority: Blocker
Fix For: 5.0, Trunk
Attachments: SOLR-7018.patch
Start a cluster using {{bin/solr -e cloud -noprompt}}.
Run {{bin/solr stop}}. This will only stop one of the two nodes, using the
default port 8983. (This in itself is a problem.)
Run {{bin/solr status}}. Previously I thought that this process hung because
it took a while to return, but just now I tried it again and got this:
{noformat}
$ bin/solr status
Found 1 Solr nodes:
Solr process 88680 running on port 7574
Failed to get system information from http://localhost:7574/solr/ due to:
org.apache.solr.client.solrj.SolrServerException: KeeperErrorCode =
ConnectionLoss for /overseer/collection-queue-work/qn-
at org.apache.solr.util.SolrCLI.getJson(SolrCLI.java:513)
at org.apache.solr.util.SolrCLI.getJson(SolrCLI.java:456)
at
org.apache.solr.util.SolrCLI$StatusTool.getCloudStatus(SolrCLI.java:697)
at
org.apache.solr.util.SolrCLI$StatusTool.reportStatus(SolrCLI.java:680)
at org.apache.solr.util.SolrCLI$StatusTool.runTool(SolrCLI.java:638)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:203)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40) - Build # 4659 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4659/ Java: 64bit/jdk1.8.0_40 -XX:+UseCompressedOops -XX:+UseG1GC 2 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.embedded.LargeVolumeEmbeddedTest Error Message: Some resources were not closed, shutdown, or released. Stack Trace: java.lang.AssertionError: Some resources were not closed, shutdown, or released. at __randomizedtesting.SeedInfo.seed([4484DB0CD4EA898]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.embedded.LargeVolumeEmbeddedTest Error Message: Could not remove the following files (in the order of attempts): C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.embedded.LargeVolumeEmbeddedTest 4484DB0CD4EA898-001\init-core-data-001\tlog\tlog.002: java.nio.file.FileSystemException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.embedded.LargeVolumeEmbeddedTest 4484DB0CD4EA898-001\init-core-data-001\tlog\tlog.002: The process cannot access the file because it is being used by another process. C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.embedded.LargeVolumeEmbeddedTest 4484DB0CD4EA898-001\init-core-data-001\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.embedded.LargeVolumeEmbeddedTest 4484DB0CD4EA898-001\init-core-data-001\tlog C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.embedded.LargeVolumeEmbeddedTest 4484DB0CD4EA898-001\init-core-data-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.embedded.LargeVolumeEmbeddedTest 4484DB0CD4EA898-001\init-core-data-001 C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.embedded.LargeVolumeEmbeddedTest 4484DB0CD4EA898-001\init-core-data-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.embedded.LargeVolumeEmbeddedTest
[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.9.0-ea-b54) - Build # 12094 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/12094/ Java: 32bit/jdk1.9.0-ea-b54 -server -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test Error Message: Error from server at http://127.0.0.1:55757/compositeid_collection_with_routerfield_shard1_replica1: no servers hosting shard: Stack Trace: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:55757/compositeid_collection_with_routerfield_shard1_replica1: no servers hosting shard: at __randomizedtesting.SeedInfo.seed([D44EB4C4399A16EA:5C1A8B1E97667B12]:0) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:556) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:233) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:225) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDeleteByIdCompositeRouterWithRouterField(FullSolrCloudDistribCmdsTest.java:357) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:502) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[jira] [Commented] (SOLR-7361) Main Jetty thread blocked by core loading delays HTTP listener from binding if core loading is slow
[
https://issues.apache.org/jira/browse/SOLR-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490623#comment-14490623
]
Damien Kamerman commented on SOLR-7361:
---
Regarding loading the cores in the background, I've made a few tweaks to work
with thousands of cores (See SOLR-7191):
1. Load cores in fixed threadPool. Cores are registered in background threads,
so no reason to load all cores at once!
2. Register cores in corename order with a fixed 128 threadPool. This is to not
flood the DistributedQueue.
3. Publish an entire node as 'down' (4.10 branch)
4. Cache ConfigSetService.createIndexSchema() in cloud mode.
So, my questions are:
What threadPool size will be used to load the cores?
What order will cores be loaded in?
Will cores be registered as they are loaded, or offloaded to another threadPool?
My initial thought was to register as a live node after cores are loaded.
Main Jetty thread blocked by core loading delays HTTP listener from binding
if core loading is slow
---
Key: SOLR-7361
URL: https://issues.apache.org/jira/browse/SOLR-7361
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Timothy Potter
Assignee: Timothy Potter
During server startup, the CoreContainer uses an ExecutorService to load
cores in multiple back-ground threads but then blocks until cores are loaded,
see: CoreContainer#load around line 290 on trunk (invokeAll). From the
JavaDoc on that method, we have:
{quote}
Executes the given tasks, returning a list of Futures holding their status
and results when all complete. Future.isDone() is true for each element of
the returned list.
{quote}
In other words, this is a blocking call.
This delays the Jetty HTTP listener from binding and accepting requests until
all cores are loaded. Do we need to block the main thread?
Also, prior to this happening, the node is registered as a live node in ZK,
which makes it a candidate for receiving requests from the Overseer, such as
to service a create collection request. The problem of course is that the
node listed in /live_nodes isn't accepting requests yet. So we either need to
unblock the main thread during server loading or maybe wait longer before we
register as a live node ... not sure which is the better way forward?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6416) BooleanQuery should only extract terms from scoring clauses
[ https://issues.apache.org/jira/browse/LUCENE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490634#comment-14490634 ] Robert Muir commented on LUCENE-6416: - +1 BooleanQuery should only extract terms from scoring clauses --- Key: LUCENE-6416 URL: https://issues.apache.org/jira/browse/LUCENE-6416 Project: Lucene - Core Issue Type: Bug Affects Versions: Trunk, 5.1 Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: 5.2 Attachments: LUCENE-6416.patch BooleanQuery should not extract terms from FILTER clauses. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7372) Limit LRUCache by RAM usage
[ https://issues.apache.org/jira/browse/SOLR-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-7372: Attachment: SOLR-7372.patch # Changed AtomicLong to a simple long because it is always read/written inside synchronized(map) # Changed ramBytesUsed() to also use the same synchronization # Renamed 'evictionsDueToRamBytes' to 'evictionsRamUsage' -- this counter will track number of evictions that happen because we exceeded the maxRamBytes - please note [~erickerickson]. I'll commit this shortly. Limit LRUCache by RAM usage --- Key: SOLR-7372 URL: https://issues.apache.org/jira/browse/SOLR-7372 Project: Solr Issue Type: Improvement Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: Trunk, 5.2 Attachments: SOLR-7372.patch, SOLR-7372.patch, SOLR-7372.patch Now that SOLR-7371 has made DocSet impls Accountable, we should add an option to LRUCache to limit itself by RAM. I propose to add a 'maxRamBytes' configuration parameter which it can use to evict items once the total RAM usage of the cache reaches this limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6319) Delegating OneMerge
[ https://issues.apache.org/jira/browse/LUCENE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Bradshaw updated LUCENE-6319: - Attachment: SOLR-6319.patch Here's a shot at a patch. I added two new classes, AbstractOneMerge and DelegatingOneMerge. I didn't want to overclutter OneMerge with a ton of getters and setters, so I moved most of that logic to AbstractOneMerge. DelegatingOneMerge could extend AbstractOneMerge instead of OneMerge, but that would require clients to change their references from OneMerge to AbstractOneMerge. I figured this was a decent compromise, but any other opinions would be appreciated. SortingOneMerge has been modified to extend DelegatingOneMerge. Almost all of the changes are internal and have been applied to IndexWriter, etc. The only public API changes are the moving of OneMerge.segments, rateLimiter and totalMaxDoc behind public getter methods. Delegating OneMerge --- Key: LUCENE-6319 URL: https://issues.apache.org/jira/browse/LUCENE-6319 Project: Lucene - Core Issue Type: Improvement Components: core/index Reporter: Elliott Bradshaw Attachments: SOLR-6319.patch In trying to integrate SortingMergePolicy into ElasticSearch, I ran into an issue where the custom merge logic was being stripped out by IndexUpgraderMergeSpecification. Related issue here: https://github.com/elasticsearch/elasticsearch/issues/9731 In an endeavor to fix this, I attempted to create a DelegatingOneMerge that could be used to chain the different MergePolicies together. I quickly discovered this to be impossible, due to the direct member variable access of OneMerge by IndexWriter and other classes. It would be great if this variable access could be privatized and the consuming classes modified to use the appropriate getters and setters. Here's an example DelegatingOneMerge and modified OneMerge. https://gist.github.com/ebradshaw/e0b74e9e8d4976ab9e0a https://gist.github.com/ebradshaw/d72116a014f226076303 The downside here is that this would require an API change, as there are three public variables in OneMerge: estimatedMergeBytes, segments and totalDocCount. These would have to be moved behind public getters. Without this change, I'm not sure how we could get the SortingMergePolicy working in ES, but if anyone has any other suggestions I'm all ears! Thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-6319) Delegating OneMerge
[ https://issues.apache.org/jira/browse/LUCENE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490665#comment-14490665 ] Elliott Bradshaw edited comment on LUCENE-6319 at 4/11/15 1:03 AM: --- Ryan, no worries. I figured that this might not be the top of the priority list. Here's my shot at a patch. I added two new classes, AbstractOneMerge and DelegatingOneMerge. I didn't want to overclutter OneMerge with a ton of getters and setters, so I moved most of that logic to AbstractOneMerge. DelegatingOneMerge could extend AbstractOneMerge instead of OneMerge, but that would require clients to change their references from OneMerge to AbstractOneMerge. I figured this was a decent compromise, but any other opinions would be appreciated. SortingOneMerge has been modified to extend DelegatingOneMerge. Almost all of the changes are internal and have been applied to IndexWriter, etc. The only public API changes are the moving of OneMerge.segments, rateLimiter and totalMaxDoc behind public getter methods. was (Author: ebradshaw): Here's a shot at a patch. I added two new classes, AbstractOneMerge and DelegatingOneMerge. I didn't want to overclutter OneMerge with a ton of getters and setters, so I moved most of that logic to AbstractOneMerge. DelegatingOneMerge could extend AbstractOneMerge instead of OneMerge, but that would require clients to change their references from OneMerge to AbstractOneMerge. I figured this was a decent compromise, but any other opinions would be appreciated. SortingOneMerge has been modified to extend DelegatingOneMerge. Almost all of the changes are internal and have been applied to IndexWriter, etc. The only public API changes are the moving of OneMerge.segments, rateLimiter and totalMaxDoc behind public getter methods. Delegating OneMerge --- Key: LUCENE-6319 URL: https://issues.apache.org/jira/browse/LUCENE-6319 Project: Lucene - Core Issue Type: Improvement Components: core/index Reporter: Elliott Bradshaw Attachments: SOLR-6319.patch In trying to integrate SortingMergePolicy into ElasticSearch, I ran into an issue where the custom merge logic was being stripped out by IndexUpgraderMergeSpecification. Related issue here: https://github.com/elasticsearch/elasticsearch/issues/9731 In an endeavor to fix this, I attempted to create a DelegatingOneMerge that could be used to chain the different MergePolicies together. I quickly discovered this to be impossible, due to the direct member variable access of OneMerge by IndexWriter and other classes. It would be great if this variable access could be privatized and the consuming classes modified to use the appropriate getters and setters. Here's an example DelegatingOneMerge and modified OneMerge. https://gist.github.com/ebradshaw/e0b74e9e8d4976ab9e0a https://gist.github.com/ebradshaw/d72116a014f226076303 The downside here is that this would require an API change, as there are three public variables in OneMerge: estimatedMergeBytes, segments and totalDocCount. These would have to be moved behind public getters. Without this change, I'm not sure how we could get the SortingMergePolicy working in ES, but if anyone has any other suggestions I'm all ears! Thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7274) Pluggable authentication module in Solr
[
https://issues.apache.org/jira/browse/SOLR-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490669#comment-14490669
]
Gregory Chanan commented on SOLR-7274:
--
Some initial thoughts...
{code}
--- solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java
(revision 1672548)
+++ solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java
(working copy)
@@ -215,7 +215,7 @@
if (urls.size() = 1) {
String url = urls.get(0);
srsp.setShardAddress(url);
-try (SolrClient client = new HttpSolrClient(url, httpClient)) {
+try (SolrClient client = new HttpSolrClient(url)) {
ssr.nl = client.request(req);
}
} else {
{code}
Why this change?
2. Can we comment on the AuthenticationLayerPlugin? I don't think it's
obvious what needs to be implemented
3. params.put(token.valid, System.getProperty(kerberos.name.rules,
30));
This doesn't look correct
4. Using SolrClient instead of whatever zookeeper uses (default Client,
see the code I linked above) for the jaas configuration app name will cause you
to have to deal with two issues. I'm not against doing something different
here, just pointing out that these problems need to be solved before you can
make this change:
A) kerberos tickets need to be refreshed. The zookeeper client automatically
refreshes kerberos tickets, so if you just use the same configuration and app
name, that's handled for you. If you want to use something different, you'll
have to write code to refresh the tickets. This also means you only refresh
the code when using zookeeper (i.e. SolrCloud)...you might want to pull out
support for specifying the plugin via system props (so you know they have to be
using solrcloud in order to read the zk props), or may want to add some error
checking.
B) assuming you want to use the same properties for talking to zookeeper as
talking to other solr servers, you'll have to specify the jaas entry twice
(once for SolrClient, once for Client or whatever zookeeper is using
(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY). Or you may be able to change how
we handle zookeeper sasl (see SaslACLProvider, I think you'd have to write a
Credentials provider?).
As I said, I'm not against doing this, it just introduces additional issues for
a first version. The code I linked above just uses whatever zookeeper users.
5. In HttpSolrClient.java: postOrPut.setEntity(new
BufferedHttpEntity(postOrPut.getEntity()));
Why are we always buffering the http entities? That seems like something that
should be handled by the authentication plugin, i.e. usually we don't buffer.
If we are using kerberos, we set up a client configurer that is smart enough to
handle the http requests for that authentication scheme (here buffering is
probably fine for the initial version, there are some discussions of an
optimization in SOLR-6625). See this comment in SOLR-6625 for some
implementation ideas
https://issues.apache.org/jira/browse/SOLR-6625?focusedCommentId=14238865page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14238865
6. This doesn't seem to handle forwarding requests in the SolrDispatchFilter.
There's more discussion in SOLR-6625, see here:
https://issues.apache.org/jira/browse/SOLR-6625?focusedCommentId=14239957page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14239957
I don't know how to solve that without implementing SOLR-6625, though.
Pluggable authentication module in Solr
---
Key: SOLR-7274
URL: https://issues.apache.org/jira/browse/SOLR-7274
Project: Solr
Issue Type: Sub-task
Reporter: Anshum Gupta
Attachments: SOLR-7274.patch
It would be good to have Solr support different authentication protocols.
To begin with, it'd be good to have support for kerberos and basic auth.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-6417) Upgrade ANTLR to version 4.5
Jack Conradson created LUCENE-6417: -- Summary: Upgrade ANTLR to version 4.5 Key: LUCENE-6417 URL: https://issues.apache.org/jira/browse/LUCENE-6417 Project: Lucene - Core Issue Type: Improvement Reporter: Jack Conradson I would like to upgrade ANTLR from 3.5 to 4.5. This version adds several features that will improve the existing grammars. The main improvement would be the allowance of left-hand recursion in grammar rules which will reduce the number of rules significantly for expressions. This change will require some code refactoring to the existing expressions work. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-5.x-Windows (32bit/jdk1.8.0_40) - Build # 4542 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Windows/4542/
Java: 32bit/jdk1.8.0_40 -server -XX:+UseSerialGC
2 tests failed.
FAILED: org.apache.solr.handler.TestReplicationHandlerBackup.doTestBackup
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([2A7B86D6D6003D9D]:0)
FAILED:
junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup
Error Message:
Suite timeout exceeded (= 720 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (= 720 msec).
at __randomizedtesting.SeedInfo.seed([2A7B86D6D6003D9D]:0)
Build Log:
[...truncated 10982 lines...]
[junit4] Suite: org.apache.solr.handler.TestReplicationHandlerBackup
[junit4] 2 Creating dataDir:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\init-core-data-001
[junit4] 2 1135797 T6173 oas.SolrTestCaseJ4.setUp ###Starting doTestBackup
[junit4] 2 1135797 T6173 oas.SolrTestCaseJ4.writeCoreProperties Writing
core.properties file to
C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\solr-instance-001\collection1
[junit4] 2 1135813 T6173 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2 1135822 T6173 oejs.AbstractConnector.doStart Started
SelectChannelConnector@127.0.0.1:58471
[junit4] 2 1135823 T6173 oascse.JettySolrRunner$1.lifeCycleStarted Jetty
properties:
{solr.data.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\solr-instance-001\collection1\data, hostContext=/solr,
hostPort=58471}
[junit4] 2 1135823 T6173 oass.SolrDispatchFilter.init
SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
[junit4] 2 1135823 T6173 oasc.SolrResourceLoader.init new
SolrResourceLoader for directory:
'C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\solr-instance-001\'
[junit4] 2 1135847 T6173 oasc.SolrXmlConfig.fromFile Loading container
configuration from
C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\solr-instance-001\solr.xml
[junit4] 2 1135855 T6173 oasc.CorePropertiesLocator.init Config-defined
core root directory:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\solr-instance-001\.
[junit4] 2 1135855 T6173 oasc.CoreContainer.init New CoreContainer
13914897
[junit4] 2 1135855 T6173 oasc.CoreContainer.load Loading cores into
CoreContainer
[instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\solr-instance-001\]
[junit4] 2 1135856 T6173 oasc.CoreContainer.load loading shared library:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\solr-instance-001\lib
[junit4] 2 1135856 T6173 oasc.SolrResourceLoader.addToClassLoader WARN
Can't find (or read) directory to add to classloader: lib (resolved as:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-5.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup
2A7B86D6D6003D9D-001\solr-instance-001\lib).
[junit4] 2 1135865 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting socketTimeout to: 9
[junit4] 2 1135865 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting urlScheme to:
[junit4] 2 1135865 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting connTimeout to: 15000
[junit4] 2 1135865 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting maxConnectionsPerHost to: 20
[junit4] 2 1135866 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting maxConnections to: 1
[junit4] 2 1135866 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting corePoolSize to: 0
[junit4] 2 1135866 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting maximumPoolSize to: 2147483647
[junit4] 2 1135867 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting maxThreadIdleTime to: 5
[junit4] 2 1135867 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting sizeOfQueue to: -1
[junit4] 2 1135867 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting fairnessPolicy to: false
[junit4] 2 1135867 T6173 oashc.HttpShardHandlerFactory.getParameter
Setting useRetries to: false
[junit4] 2 1135870
[jira] [Commented] (SOLR-7275) Pluggable authorization module in Solr
[
https://issues.apache.org/jira/browse/SOLR-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490733#comment-14490733
]
Dennis Gove commented on SOLR-7275:
---
I like this concept but I think the response can be expanded to add a bit more
functionality. It would be nice if the pluggable security layer could respond
in such a way as to not wholly reject a request but to instead restrict what is
returned from a request. It could accomplish this by providing additional
filters to apply to a request.
{code}
public class SolrAuthorizationResponse {
boolean authorized;
String additionalFilterQuery;
...
}
{code}
By adding additionalFilterQuery, this would give the security layer an
opportunity to say, yup, you're authorized but you can't see records matching
this filter or yup, you're authorized but you can only see records also
matching this filter. It provides a way to add fine-grained control of data
access but keep that control completely outside of SOLR (as it would live in
the pluggable security layer).
Additionally, it allows the security layer to add fine-grained control
**without notifying the user they are being restricted** as this lives wholly
in the SOLR --- security layer communication. There are times when telling
the user their request was rejected due to it returning records they're not
privileged to see actually gives the user some information you may not want
them to know - the fact that these restricted records even exist. Instead, by
adding filters and just not returning records the user isn't privileged for,
the user is non-the-wiser that they were restricted at all.
Pluggable authorization module in Solr
--
Key: SOLR-7275
URL: https://issues.apache.org/jira/browse/SOLR-7275
Project: Solr
Issue Type: Sub-task
Reporter: Anshum Gupta
Assignee: Anshum Gupta
Solr needs an interface that makes it easy for different authorization
systems to be plugged into it. Here's what I plan on doing:
Define an interface {{SolrAuthorizationPlugin}} with one single method
{{isAuthorized}}. This would take in a {{SolrRequestContext}} object and
return an {{SolrAuthorizationResponse}} object. The object as of now would
only contain a single boolean value but in the future could contain more
information e.g. ACL for document filtering etc.
The reason why we need a context object is so that the plugin doesn't need to
understand Solr's capabilities e.g. how to extract the name of the collection
or other information from the incoming request as there are multiple ways to
specify the target collection for a request. Similarly request type can be
specified by {{qt}} or {{/handler_name}}.
Flow:
Request - SolrDispatchFilter - isAuthorized(context) - Process/Return.
{code}
public interface SolrAuthorizationPlugin {
public SolrAuthorizationResponse isAuthorized(SolrRequestContext context);
}
{code}
{code}
public class SolrRequestContext {
UserInfo; // Will contain user context from the authentication layer.
HTTPRequest request;
Enum OperationType; // Correlated with user roles.
String[] CollectionsAccessed;
String[] FieldsAccessed;
String Resource;
}
{code}
{code}
public class SolrAuthorizationResponse {
boolean authorized;
public boolean isAuthorized();
}
{code}
User Roles:
* Admin
* Collection Level:
* Query
* Update
* Admin
Using this framework, an implementation could be written for specific
security systems e.g. Apache Ranger or Sentry. It would keep all the security
system specific code out of Solr.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7275) Pluggable authorization module in Solr
[
https://issues.apache.org/jira/browse/SOLR-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490745#comment-14490745
]
Noble Paul commented on SOLR-7275:
--
Yes. That is totally possible
Pluggable authorization module in Solr
--
Key: SOLR-7275
URL: https://issues.apache.org/jira/browse/SOLR-7275
Project: Solr
Issue Type: Sub-task
Reporter: Anshum Gupta
Assignee: Anshum Gupta
Solr needs an interface that makes it easy for different authorization
systems to be plugged into it. Here's what I plan on doing:
Define an interface {{SolrAuthorizationPlugin}} with one single method
{{isAuthorized}}. This would take in a {{SolrRequestContext}} object and
return an {{SolrAuthorizationResponse}} object. The object as of now would
only contain a single boolean value but in the future could contain more
information e.g. ACL for document filtering etc.
The reason why we need a context object is so that the plugin doesn't need to
understand Solr's capabilities e.g. how to extract the name of the collection
or other information from the incoming request as there are multiple ways to
specify the target collection for a request. Similarly request type can be
specified by {{qt}} or {{/handler_name}}.
Flow:
Request - SolrDispatchFilter - isAuthorized(context) - Process/Return.
{code}
public interface SolrAuthorizationPlugin {
public SolrAuthorizationResponse isAuthorized(SolrRequestContext context);
}
{code}
{code}
public class SolrRequestContext {
UserInfo; // Will contain user context from the authentication layer.
HTTPRequest request;
Enum OperationType; // Correlated with user roles.
String[] CollectionsAccessed;
String[] FieldsAccessed;
String Resource;
}
{code}
{code}
public class SolrAuthorizationResponse {
boolean authorized;
public boolean isAuthorized();
}
{code}
User Roles:
* Admin
* Collection Level:
* Query
* Update
* Admin
Using this framework, an implementation could be written for specific
security systems e.g. Apache Ranger or Sentry. It would keep all the security
system specific code out of Solr.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
