[jira] [Commented] (SOLR-3561) Error during deletion of shard/core
[
https://issues.apache.org/jira/browse/SOLR-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546190#comment-15546190
]
Per Steffensen commented on SOLR-3561:
--
I originally created the ticket. I am not against closing it. I do not know if
the problem still exists (in some shape), but a lot of things has changed
since, so someone will have to bring up the problem again if it is still a
problem.
> Error during deletion of shard/core
> ---
>
> Key: SOLR-3561
> URL: https://issues.apache.org/jira/browse/SOLR-3561
> Project: Solr
> Issue Type: Bug
> Components: multicore, replication (java), SolrCloud
>Affects Versions: 4.0-ALPHA
> Environment: Solr trunk (4.0-SNAPSHOT) from 29/2-2012
>Reporter: Per Steffensen
>Assignee: Mark Miller
> Fix For: 4.9, 6.0
>
>
> Running several Solr servers in Cloud-cluster (zkHost set on the Solr
> servers).
> Several collections with several slices and one replica for each slice (each
> slice has two shards)
> Basically we want let our system delete an entire collection. We do this by
> trying to delete each and every shard under the collection. Each shard is
> deleted one by one, by doing CoreAdmin-UNLOAD-requests against the relevant
> Solr
> {code}
> CoreAdminRequest request = new CoreAdminRequest();
> request.setAction(CoreAdminAction.UNLOAD);
> request.setCoreName(shardName);
> CoreAdminResponse resp = request.process(new CommonsHttpSolrServer(solrUrl));
> {code}
> The delete/unload succeeds, but in like 10% of the cases we get errors on
> involved Solr servers, right around the time where shard/cores are deleted,
> and we end up in a situation where ZK still claims (forever) that the deleted
> shard is still present and active.
> Form here the issue is easilier explained by a more concrete example:
> - 7 Solr servers involved
> - Several collection a.o. one called "collection_2012_04", consisting of 28
> slices, 56 shards (remember 1 replica for each slice) named
> "collection_2012_04_sliceX_shardY" for all pairs in {X:1..28}x{Y:1,2}
> - Each Solr server running 8 shards, e.g Solr server #1 is running shard
> "collection_2012_04_slice1_shard1" and Solr server #7 is running shard
> "collection_2012_04_slice1_shard2" belonging to the same slice "slice1".
> When we decide to delete the collection "collection_2012_04" we go through
> all 56 shards and delete/unload them one-by-one - including
> "collection_2012_04_slice1_shard1" and "collection_2012_04_slice1_shard2". At
> some point during or shortly after all this deletion we see the following
> exceptions in solr.log on Solr server #7
> {code}
> Aug 1, 2012 12:02:50 AM org.apache.solr.common.SolrException log
> SEVERE: Error while trying to recover:org.apache.solr.common.SolrException:
> core not found:collection_2012_04_slice1_shard1
> request:
> http://solr_server_1:8983/solr/admin/cores?action=PREPRECOVERY&core=collection_2012_04_slice1_shard1&nodeName=solr_server_7%3A8983_solr&coreNodeName=solr_server_7%3A8983_solr_collection_2012_04_slice1_shard2&state=recovering&checkLive=true&pauseFor=6000&wt=javabin&version=2
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.solr.common.SolrExceptionPropagationHelper.decodeFromMsg(SolrExceptionPropagationHelper.java:29)
> at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:445)
> at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:264)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:188)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:285)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:206)
> Aug 1, 2012 12:02:50 AM org.apache.solr.common.SolrException log
> SEVERE: Recovery failed - trying again...
> Aug 1, 2012 12:02:51 AM org.apache.solr.cloud.LeaderElector$1 process
> WARNING:
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at org.apache.solr.cloud.LeaderElector.checkIfIam
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402138#comment-15402138 ] Per Steffensen commented on SOLR-6810: -- I have not looked much into SOLR-8220, but from the little reading I have done, I guess you would have to doc-value the id-field for SOLR-8220 to help on the SOLR-6810 issue. I guess, most systems do not have id-field doc-valued. I also guess, for most systems it is feasible to do reindex it all, so that id's become doc-value. But not for all systems - e.g. not for our systems. We have 1000 billion documents in one of our systems, and you do not just reindex all of that. It takes weeks and is a very complex operation (we have tried it a few times). I do not know if such an argument counts here, but anyway... > Faster searching limited but high rows across many shards all with many hits > > > Key: SOLR-6810 > URL: https://issues.apache.org/jira/browse/SOLR-6810 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: Per Steffensen >Assignee: Shalin Shekhar Mangar > Labels: distributed_search, performance > Attachments: SOLR-6810-hack-eoe.patch, SOLR-6810-trunk.patch, > SOLR-6810-trunk.patch, SOLR-6810-trunk.patch, branch_5x_rev1642874.patch, > branch_5x_rev1642874.patch, branch_5x_rev1645549.patch > > > Searching "limited but high rows across many shards all with many hits" is > slow > E.g. > * Query from outside client: q=something&rows=1000 > * Resulting in sub-requests to each shard something a-la this > ** 1) q=something&rows=1000&fl=id,score > ** 2) Request the full documents with ids in the global-top-1000 found among > the top-1000 from each shard > What does the subject mean > * "limited but high rows" means 1000 in the example above > * "many shards" means 200-1000 in our case > * "all with many hits" means that each of the shards have a significant > number of hits on the query > The problem grows on all three factors above > Doing such a query on our system takes between 5 min to 1 hour - depending on > a lot of things. It ought to be much faster, so lets make it. > Profiling show that the problem is that it takes lots of time to access the > store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. > Having 1000 shards its up to 1 mio ids that has to be fetched. There is > really no good reason to ever read information from store for more than the > overall top-1000 documents, that has to be returned to the client. > For further detail see mail-thread "Slow searching limited but high rows > across many shards all with high hits" started 13/11-2014 on > dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Using rows=-1 for "give me all"
On 25/05/16 01:11, Chris Hostetter wrote: : > : Back when we used 4.4.0 I believe a query with rows=-1 returned all : > matching : > Nope -- that's never been how rows=-1 behaved. : Believe you are wrong. That was how it behaved in 4.4.0 Nope -- not in a regular search it didn't... OK, thanks. I am confused now, because we have been using the "feature" a lot in the past - also for none-grouping queries. I will investigate more, but cannot promise I will come to a conclusion that is relevant here. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Using rows=-1 for "give me all"
On 23/05/16 23:11, Chris Hostetter wrote: : Back when we used 4.4.0 I believe a query with rows=-1 returned all matching : documents. In 5.1.0 (the one we are using now) rows=-1 will trigger a Nope -- that's never been how rows=-1 behaved. Believe you are wrong. That was how it behaved in 4.4.0 The fact that it didn't return an error is some older versions of solr (and may have behaved similar to rows=0 in specific versions) was a bug of poor validation. but no "fetch all docs" behavior has ever been removed, it never existed in the first place other then in the special case support for the "/export" handler. And I believe that was intended behavior. The javadoc for SortSpec.getCount still says that -1 ought to do a "no cut off". https://github.com/apache/lucene-solr/blob/branch_6_0/solr/core/src/java/org/apache/solr/search/SortSpec.java Also see Yoniks comment around 23/Mar/15 here: https://issues.apache.org/jira/browse/SOLR-7254 Regards, Per
Using rows=-1 for "give me all"
Hi Back when we used 4.4.0 I believe a query with rows=-1 returned all matching documents. In 5.1.0 (the one we are using now) rows=-1 will trigger a validation exception. If I remove the code that throws that exception, it seems like rows=-1 behaves like rows=0. Has the support for rows=-1 (give me all) been reintroduced in a release after 5.1.0? If yes, which JIRA-ticket? If no, any plans to reintroduce it? Any good reason for changing the rows=-1 behavior? Am I the only one that liked it? :-) Regards, Per Steffensen - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249790#comment-15249790 ] Per Steffensen commented on SOLR-8922: -- bq. I don't know how well Per Steffensen tested this, although I got the impression he was using it in production at the time Well, I know I am late, but just for the record. It was tested thoroughly and the gain was significant. It has been running in several production-systems since forever. > DocSetCollector can allocate massive garbage on large indexes > - > > Key: SOLR-8922 > URL: https://issues.apache.org/jira/browse/SOLR-8922 > Project: Solr > Issue Type: Improvement >Reporter: Jeff Wartes >Assignee: Yonik Seeley > Attachments: SOLR-8922.patch, SOLR-8922.patch > > > After reaching a point of diminishing returns tuning the GC collector, I > decided to take a look at where the garbage was coming from. To my surprise, > it turned out that for my index and query set, almost 60% of the garbage was > coming from this single line: > https://github.com/apache/lucene-solr/blob/94c04237cce44cac1e40e1b8b6ee6a6addc001a5/solr/core/src/java/org/apache/solr/search/DocSetCollector.java#L49 > This is due to the simple fact that I have 86M documents in my shards. > Allocating a scratch array big enough to track a result set 1/64th of my > index (1.3M) is also almost certainly excessive, considering my 99.9th > percentile hit count is less than 56k. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5670) _version_ either indexed OR docvalue
[ https://issues.apache.org/jira/browse/SOLR-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190785#comment-15190785 ] Per Steffensen commented on SOLR-5670: -- Well, no, I have not. Our reason for doing this is not necessarily to make things faster. It basically boils down to avoiding OOMs. When you want to get a value for a particular field on a particular document in Solr/Lucene there are several places to find the information. You can find it in the store (if stored=true for that field), which is fairly slow. You can also find it in the FieldCache, which is an in-memory doc-id to field-value map. Problem with FieldCache is that if you load it, you need to load values for ALL the documents in the index (or at least the segment). If you have enormous amounts of documents in your index this can cause OOM, because you simple cannot have all those values in memory. Then doc-value is for the rescue. Doc-value basically holds the same doc-id to field-value data-structure as the FieldCache, but doc-values are maintained continuously in additional files in your index-folder, so you can just go read the particular value you need, hence avoiding OOM. FieldCache is something you "calculate" into memory, based on data form store or index. > _version_ either indexed OR docvalue > > > Key: SOLR-5670 > URL: https://issues.apache.org/jira/browse/SOLR-5670 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: 4.7 >Reporter: Per Steffensen >Assignee: Per Steffensen > Labels: solr, solrcloud, version > Fix For: 4.7, master > > Attachments: SOLR-5670.patch, SOLR-5670.patch > > > As far as I can see there is no good reason to require that "_version_" field > has to be indexed if it is docvalued. So I guess it will be ok with a rule > saying "_version_ has to be either indexed or docvalue (allowed to be both)". -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7890) By default require admin rights to access /security.json in ZK
[
https://issues.apache.org/jira/browse/SOLR-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710861#comment-14710861
]
Per Steffensen commented on SOLR-7890:
--
In general, looks good!
Like the JavaDoc you added to {{VMParamsAllAndReadonlyDigestZkACLProvider}}
Remember update documentation on cwiki, if you find appropriate.
Maybe {{ZkZnodeProtection}} should be closer connected to
{{VMParamsAllAndReadonlyDigestZkACLProvider}} (e.g. inner class). It is only
when you actively decide to use {{VMParamsAllAndReadonlyDigestZkACLProvider}}
as your ACL-provider that {{ZkZnodeProtection}} is used. Default is still
{{DefaultZkACLProvider}} which does not take {{ZkZnodeProtection}} into
consideration. When {{ZkZnodeProtection}} is a separate standalone class, you
might be surprised that it is actually only used when you actively pick
{{VMParamsAllAndReadonlyDigestZkACLProvider}}.
I believe a few versions of Solr have already been released including
SOLR-4580. Therefore you need to think about if Solr behaves appropriate when
upgrading to a release including SOLR-7890. Lets say you have a Solr (version
before SOLR-7890) installation where you have turned on
{{VMParamsAllAndReadonlyDigestZkACLProvider}}, so that all your znodes are
ACL'ed with all/admin/admin and read-only/client/client.
* If you do nothing, but read the release-notes including SOLR-7890
description, you might expect that the ACL's of security.json (already exists
in ZK when you start up your new Solr) is automatically changed to only have
ACL all/admin/admin (and not read-only/client/client anymore). That will not
happen.
* Same problem if you upgrade to the new Solr and provide an explicit
{{-DzkProtectedPaths}} including znodes that already exist
There is a consistency issue here, that we at least need to be explicit about.
You will not end up with the "expected" ACL's on protected-paths if those
znodes already exist when you upgrade to an SOLR-7890 Solr. This problem also
existed before SOLR-7890, but only if you changed your ACL-provider
implementation, and in that case you have a better chance of guessing the issue
as an administrator of the system. So I think this problem becomes more urgent
to document/solve as we introduce SOLR-7890.
Same kind of problem will occur, if we in a later release add to the
default-list
{{VMParamsAllAndReadonlyDigestZkACLProvider.DEFAULT_PROTECTED_PATHS}}
I do know if we should just document this "behavior", or make a solution. In
case we want to make a solution, we might want to have some code that can run
through the entire znode-structure and ensure that the ACL's are as they are
supposed to be according to current code and configuration. Then run that code
when Solr nodes start and/or as part of upgrading Solr-versions.
Sorry, that I do not have time for more than this quick comment.
> By default require admin rights to access /security.json in ZK
> --
>
> Key: SOLR-7890
> URL: https://issues.apache.org/jira/browse/SOLR-7890
> Project: Solr
> Issue Type: Sub-task
> Components: security
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Fix For: Trunk
>
> Attachments: SOLR-7890.patch
>
>
> Perhaps {{VMParamsAllAndReadonlyDigestZkACLProvider}} should by default
> require admin access for read/write of {{/security.json}}, and other
> sensitive paths. Today this is left to the user to implement.
> Also, perhaps factor out the already-known sensitive paths into a separate
> class, so that various {{ACLProvider}} implementations can get a list of
> paths that should be admin-only, read-only etc from one central place. Then
> 3rd party impls pulling ZK creds from elsewhere will still do the right thing
> in the future if we introduce other sensitive Znodes...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661611#comment-14661611 ] Per Steffensen commented on SOLR-4470: -- Hopefully I will soon have the time to see if this other implementation will fulfill our needs. But closing SOLR-4470 for now is fine > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 > Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no > matter what kind of HTTP-request you make to a Solr-node. > It can faily easy be acheived as described on > http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes > also make "internal" request to other Solr-nodes, and for it to work > credentials need to be provided here also. > Ideally we would like to "forward" credentials from a particular request to > all the "internal" sub-requests it triggers. E.g. for search and update > request. > But there are also "internal" requests > * that only indirectly/asynchronously triggered from "outside" requests (e.g. > shard creation/deletion/etc based on calls to the "Collection API") > * that do not in any way have relation to an "outside" "super"-request (e.g. > replica synching stuff) > We would like to aim at a solution where "original" credentials are > "forwarded" when a request directly/synchronously trigger a subrequest, and > fallback to a configured "internal credentials" for the > asynchronous/non-rooted requests. > In our solution we would aim at only supporting basic http auth, but we would > like to make a "framework" around it, so that not to much refactoring is > needed if you later want to make support for other kinds of auth (e.g. digest) > We will work at a solution but create this JIRA issue early in order to get > input/comments from the community as early as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
SOLR-7274 and SOLR-7275 documentation
Hi Happy to see that something seems to have happened around http-interface security on Solr (SOLR-7274 and SOLR-7275). I was wondering if any documentation on how to use/config etc has been produced yet? If yes, where can I find it? If no, any plans? Regards, Per Steffensen - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576732#comment-14576732 ] Per Steffensen commented on SOLR-4470: -- Soon I will be looking into whether SOLR-7274 etc. can be used instead of this SOLR-4470. And to what extend. > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 > Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no > matter what kind of HTTP-request you make to a Solr-node. > It can faily easy be acheived as described on > http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes > also make "internal" request to other Solr-nodes, and for it to work > credentials need to be provided here also. > Ideally we would like to "forward" credentials from a particular request to > all the "internal" sub-requests it triggers. E.g. for search and update > request. > But there are also "internal" requests > * that only indirectly/asynchronously triggered from "outside" requests (e.g. > shard creation/deletion/etc based on calls to the "Collection API") > * that do not in any way have relation to an "outside" "super"-request (e.g. > replica synching stuff) > We would like to aim at a solution where "original" credentials are > "forwarded" when a request directly/synchronously trigger a subrequest, and > fallback to a configured "internal credentials" for the > asynchronous/non-rooted requests. > In our solution we would aim at only supporting basic http auth, but we would > like to make a "framework" around it, so that not to much refactoring is > needed if you later want to make support for other kinds of auth (e.g. digest) > We will work at a solution but create this JIRA issue early in order to get > input/comments from the community as early as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7624) Correct wrong spelling of zkCredentialsProvider
Per Steffensen created SOLR-7624: Summary: Correct wrong spelling of zkCredentialsProvider Key: SOLR-7624 URL: https://issues.apache.org/jira/browse/SOLR-7624 Project: Solr Issue Type: Bug Components: security, SolrCloud Reporter: Per Steffensen Priority: Minor SolrXmlConfig.java contain the string "zkCredientialsProvider". It should be corrected to "zkCredentialsProvider". I believe no other changes are necessary - there are no tests covering this AFAIK -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7543) Create GraphQuery that allows graph traversal as a query operator.
[ https://issues.apache.org/jira/browse/SOLR-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547645#comment-14547645 ] Per Steffensen commented on SOLR-7543: -- bq. "A graph is a filter on top of your data." Yes I completely agree! It is just that, as far a I understand the discussion here, it is not just the very simple basics for doing graphs on top of Solr. Seems like you design for a specific "query" that you can do with a graph database: start at some docs --> recursively follow links/edges (up to) maxDepth times --> then return to me all nodes/vertices you came across (or just the leaves). This is one particular graph traversal you can do, but if you just implemented the graph-basics as a Titan search-backend you could perform all kinds of graph "queries" with the Titan search-tool Gremlin (https://github.com/tinkerpop/gremlin/wiki) - that is more like a full graph-solution on top of Solr. I just realized from the Titan main-page (http://thinkaurelius.github.io/titan/) that there is now some kind of integration with Solr. There wasnt when I did my implementation, but never mind, I mainly did it to understand how Titan works. It is not that I do not like this issue SOLR-7543. I definitely would love to see graphs on top of Solr. I am just trying to take a step back or and look at a more generic approach to all this graph stuff > Create GraphQuery that allows graph traversal as a query operator. > -- > > Key: SOLR-7543 > URL: https://issues.apache.org/jira/browse/SOLR-7543 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Kevin Watters >Priority: Minor > > I have a GraphQuery that I implemented a long time back that allows a user to > specify a "startQuery" to identify which documents to start graph traversal > from. It then gathers up the edge ids for those documents , optionally > applies an additional filter. The query is then re-executed continually > until no new edge ids are identified. I am currently hosting this code up at > https://github.com/kwatters/solrgraph and I would like to work with the > community to get some feedback and ultimately get it committed back in as a > lucene query. > Here's a bit more of a description of the parameters for the query / graph > traversal: > q - the initial start query that identifies the universe of documents to > start traversal from. > fromField - the field name that contains the node id > toField - the name of the field that contains the edge id(s). > traversalFilter - this is an additional query that can be supplied to limit > the scope of graph traversal to just the edges that satisfy the > traversalFilter query. > maxDepth - integer specifying how deep the breadth first search should go. > returnStartNodes - boolean to determine if the documents that matched the > original "q" should be returned as part of the graph. > onlyLeafNodes - boolean that filters the graph query to only return > documents/nodes that have no edges. > We identify a set of documents with "q" as any arbitrary lucene query. It > will collect the values in the fromField, create an OR query with those > values , optionally apply an additional constraint from the "traversalFilter" > and walk the result set until no new edges are detected. Traversal can also > be stopped at N hops away as defined with the maxDepth. This is a BFS > (Breadth First Search) algorithm. Cycle detection is done by not revisiting > the same document for edge extraction. > This query operator does not keep track of how you arrived at the document, > but only that the traversal did arrive at the document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7543) Create GraphQuery that allows graph traversal as a query operator.
[ https://issues.apache.org/jira/browse/SOLR-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545858#comment-14545858 ] Per Steffensen edited comment on SOLR-7543 at 5/15/15 5:52 PM: --- Sounds interesting. I can't help thinking that this will help users do only one particular graph'ish search. But there are millions of other graph'ish searches one might want to do. The solution here might be too specific. A while back I wrote a Solr indexing-backend for the Titan graph database. We can do a storage-backend as well. Putting a full blown graph database on top of Solr (by supporting indexing- and potentially storage-backends for e.g. Titan) might be the way to go instead, so that we will not end up with lots and lots of very specific graph-search query-parsers/resolvers. And this way you will get all the other cool stuff from a full blown graph database - e.g. I liked playing with Titans REPL. Just a thought was (Author: steff1193): Sounds interesting. I can't help thinking that this will help users doing one particular graph'ish search. But there are millions of other graph'ish searches one might want to do. The solution here might be too specific. A while back I wrote a indexing-backend for the Titan graph database. Putting a full blown graph database on top of Solr might be the way to go instead, so that we will not end up with lots and lots of very specific graph-search query-parsers/resolvers. Just a thought > Create GraphQuery that allows graph traversal as a query operator. > -- > > Key: SOLR-7543 > URL: https://issues.apache.org/jira/browse/SOLR-7543 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Kevin Watters >Priority: Minor > > I have a GraphQuery that I implemented a long time back that allows a user to > specify a "startQuery" to identify which documents to start graph traversal > from. It then gathers up the edge ids for those documents , optionally > applies an additional filter. The query is then re-executed continually > until no new edge ids are identified. I am currently hosting this code up at > https://github.com/kwatters/solrgraph and I would like to work with the > community to get some feedback and ultimately get it committed back in as a > lucene query. > Here's a bit more of a description of the parameters for the query / graph > traversal: > q - the initial start query that identifies the universe of documents to > start traversal from. > fromField - the field name that contains the node id > toField - the name of the field that contains the edge id(s). > traversalFilter - this is an additional query that can be supplied to limit > the scope of graph traversal to just the edges that satisfy the > traversalFilter query. > maxDepth - integer specifying how deep the breadth first search should go. > returnStartNodes - boolean to determine if the documents that matched the > original "q" should be returned as part of the graph. > onlyLeafNodes - boolean that filters the graph query to only return > documents/nodes that have no edges. > We identify a set of documents with "q" as any arbitrary lucene query. It > will collect the values in the fromField, create an OR query with those > values , optionally apply an additional constraint from the "traversalFilter" > and walk the result set until no new edges are detected. Traversal can also > be stopped at N hops away as defined with the maxDepth. This is a BFS > (Breadth First Search) algorithm. Cycle detection is done by not revisiting > the same document for edge extraction. > This query operator does not keep track of how you arrived at the document, > but only that the traversal did arrive at the document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7543) Create GraphQuery that allows graph traversal as a query operator.
[ https://issues.apache.org/jira/browse/SOLR-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545858#comment-14545858 ] Per Steffensen commented on SOLR-7543: -- Sounds interesting. I can't help thinking that this will help users doing one particular graph'ish search. But there are millions of other graph'ish searches one might want to do. The solution here might be too specific. A while back I wrote a indexing-backend for the Titan graph database. Putting a full blown graph database on top of Solr might be the way to go instead, so that we will not end up with lots and lots of very specific graph-search query-parsers/resolvers. Just a thought > Create GraphQuery that allows graph traversal as a query operator. > -- > > Key: SOLR-7543 > URL: https://issues.apache.org/jira/browse/SOLR-7543 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Kevin Watters >Priority: Minor > > I have a GraphQuery that I implemented a long time back that allows a user to > specify a "startQuery" to identify which documents to start graph traversal > from. It then gathers up the edge ids for those documents , optionally > applies an additional filter. The query is then re-executed continually > until no new edge ids are identified. I am currently hosting this code up at > https://github.com/kwatters/solrgraph and I would like to work with the > community to get some feedback and ultimately get it committed back in as a > lucene query. > Here's a bit more of a description of the parameters for the query / graph > traversal: > q - the initial start query that identifies the universe of documents to > start traversal from. > fromField - the field name that contains the node id > toField - the name of the field that contains the edge id(s). > traversalFilter - this is an additional query that can be supplied to limit > the scope of graph traversal to just the edges that satisfy the > traversalFilter query. > maxDepth - integer specifying how deep the breadth first search should go. > returnStartNodes - boolean to determine if the documents that matched the > original "q" should be returned as part of the graph. > onlyLeafNodes - boolean that filters the graph query to only return > documents/nodes that have no edges. > We identify a set of documents with "q" as any arbitrary lucene query. It > will collect the values in the fromField, create an OR query with those > values , optionally apply an additional constraint from the "traversalFilter" > and walk the result set until no new edges are detected. Traversal can also > be stopped at N hops away as defined with the maxDepth. This is a BFS > (Breadth First Search) algorithm. Cycle detection is done by not revisiting > the same document for edge extraction. > This query operator does not keep track of how you arrived at the document, > but only that the traversal did arrive at the document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-3428) SolrCmdDistributor flushAdds/flushDeletes problems
[ https://issues.apache.org/jira/browse/SOLR-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen closed SOLR-3428. Resolution: Fixed None of the problems are present in latest code (looking at 5.1.0 release) > SolrCmdDistributor flushAdds/flushDeletes problems > -- > > Key: SOLR-3428 > URL: https://issues.apache.org/jira/browse/SOLR-3428 > Project: Solr > Issue Type: Bug > Components: replication (java), SolrCloud, update >Affects Versions: 4.0-ALPHA > Reporter: Per Steffensen > Assignee: Per Steffensen > Labels: add, delete, replica, solrcloud, update > Original Estimate: 24h > Remaining Estimate: 24h > > A few problems with SolrCmdDistributor.flushAdds/flushDeletes > * If number of AddRequests/DeleteRequests in alist/dlist is below limit for a > specific node the method returns immediately and doesnt flush for subsequent > nodes > * When returning immediately because there is below limit requests for a > given node, then previous nodes that have already been flushed/submitted are > not removed from adds/deletes maps (causing them to be flushed/submitted > again the next time flushAdds/flushDeletes is executed) > * The idea about just combining params does not work for SEEN_LEADER params > (and probably others as well). Since SEEN_LEADER cannot be expressed (unlike > commitWithin and overwrite) for individual operations in the request, you > need to sent two separate submits. One containing requests with > SEEN_LEADER=true and one with SEEN_LEADER=false. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Running 5.1.0 test-suite via maven
On 05/05/15 15:05, Dawid Weiss wrote: I'm part of Apache Lucene and I try to make every release meet quality standards, but Solr tests are quite complex and have had a long history of being very difficult to fix. If you search mailing list archives you'll see at least a few attempts and continuous efforts from various people aiming at stabilizing these test. Yes, I know too well This said, if you're asking whether there was a "green" light before the release then yes Yes, that was what would expect, and therefor kept thinking that it must be something with out build-environment. And therefore likely no need for a JIRA -- anybody who votes for the release to proceed ran the smoketester and this in turn runs all the tests. But machines and environments vary. I don't know anybody who'd run development runs as root, for example (which you apparently did). Yes, that is clearly a mistake The problems you initially experienced were your environment setup problems (different compile JDK compared to runtime JDK), they don't have much to do with Solr tests too. Exactly Dawid Regards, Per Steffensen - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Running 5.1.0 test-suite via maven
I believe a JIRA will be about fixing a problem. But right now it is not so much about fixing a problem, it is about telling if there is a problem in the code to fix, or if it is just us doing something wrong. We want to be able to run the test-suite with the same test-results as you get at Apache CI. So if this particular test also fail at Apache CI for version 5.1.0 we have no problem. It is just that I would expect you to have seen a green test-suite in Apache CI on 5.1.0 code before releasing it. The fact that we do not see a green test-suite on 5.1.0 make us wonder why - do we run the test-suite correctly, or ... ? Can you confirm that you have seen a green test-suite at Apache CI on 5.1.0 code? Because then we have a problem in my organization, and there is nothing to fix on Apache side, and therefore no JIRA to open. If the test is red also on you side, and it is not because we run the test-suite in a wrong way, of course it should be corrected, and that would call for a JIRA. But right now we are not sure. Regards, Per Steffensen On 05/05/15 14:08, Mark Miller wrote: +1 - I'd file JIRAs. - Mark On Tue, May 5, 2015 at 6:13 AM Dawid Weiss mailto:dawid.we...@cs.put.poznan.pl>> wrote: The assertion alone isn't helpful. Solr tests dump tons of logs along the way, if you can copy these to a text file and then create a jira issue (reference it here) then perhaps somebody can look into the cause of the problem (or you can do it -- it's open source after all :). https://issues.apache.org/jira/secure/Dashboard.jspa Dawid On Tue, May 5, 2015 at 11:29 AM, Kåre Brandborg mailto:ka...@liace.dk>> wrote: > I’m a colleague of Per from who I’ve taken over the task to try to get our test environment to build Solr 5.1 and compile and run a test suite with green lights. > > I’ll try to elaborate a little more about our progress. > > We are currently using Teamcity CI and we are running our tests on an Ubuntu 12.04 x64 with jdk7u76 (x64) and ant 1.9.4. > > We have made a single change to the: ./lucene/ivy-settings.xml file (to point it to use our internal repository to resolve artifacts. > > I’ve observed that the following test is failing for us on every run (Have made 5 runs so far on the configuration above): > > 2> 1490839 T6144 oasc.CachingDirectoryFactory.close Closing directory: /opt/buildagent/work/6906da56abce9b00/solr/build/solr-core/test/J1/temp/solr.core.TestCoreDiscovery 93911594584047D7-001/tempDir-001/core2/core2/index > 2> 1490840 T6144 oas.SolrTestCaseJ4.tearDown ###Ending testCoreDirCantRead > 2> NOTE: reproduce with: ant test -Dtestcase=TestCoreDiscovery -Dtests.method=testCoreDirCantRead -Dtests.seed=93911594584047D7 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Australia/West -Dtests.asserts=true -Dtests.file.encoding=US-ASCII > FAILURE 0.72s J1 | TestCoreDiscovery.testCoreDirCantRead <<< >> Throwable #1: java.lang.AssertionError >> at __randomizedtesting.SeedInfo.seed([93911594584047D7:EED5C1C227C789A5]:0) >> at org.apache.solr.core.TestCoreDiscovery.testCoreDirCantRead(TestCoreDiscovery.java:286) >> at java.lang.Thread.run(Thread.java:745) > 2> 1490889 T6144 oas.SolrTestCaseJ4.setUp ###Starting testAlternateCoreDir > > And we are running ant with the following parameter: > > ant -f build.xml -Dtests.haltonfailure=false test > > Looking for any help on what can be causing these tests to fail. We suspect a misconfiguration in our environment but are unsure where to look. > > Thanks > > Kaare Brandborg > > >> On 04 May 2015, at 14:30, Dawid Weiss mailto:dawid.we...@cs.put.poznan.pl>> wrote: >> >>> [junit4] ERROR 3.81s J2 | TestDirectoryTaxonomyWriter.testConcurrency >>> <<< >>> [junit4]> Throwable #1: java.lang.NoSuchMethodError: >>> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; >> >> Sorry about the delay. This indicates your code was compiled with >> JDK1.8 but is executed with Java < 1.8. This method's signature used >> to be an interface, but is a covariant pointing at a specialized >> subclass in 1.8. >> >> You need to compile the code with the version of Java you intend to >> run with. Things will in general work if you compile with an older >> version and try to run with a newer version but not the other way >> around. >> >> You can cr
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520125#comment-14520125 ] Per Steffensen commented on SOLR-6810: -- Happy to see the activity here. Currently I am very busy, so I do not have the time to follow the work closely. Hope I will soon be able to. > Faster searching limited but high rows across many shards all with many hits > > > Key: SOLR-6810 > URL: https://issues.apache.org/jira/browse/SOLR-6810 > Project: Solr > Issue Type: Improvement > Components: search > Reporter: Per Steffensen >Assignee: Shalin Shekhar Mangar > Labels: distributed_search, performance > Attachments: SOLR-6810-trunk.patch, SOLR-6810-trunk.patch, > SOLR-6810-trunk.patch, branch_5x_rev1642874.patch, > branch_5x_rev1642874.patch, branch_5x_rev1645549.patch > > > Searching "limited but high rows across many shards all with many hits" is > slow > E.g. > * Query from outside client: q=something&rows=1000 > * Resulting in sub-requests to each shard something a-la this > ** 1) q=something&rows=1000&fl=id,score > ** 2) Request the full documents with ids in the global-top-1000 found among > the top-1000 from each shard > What does the subject mean > * "limited but high rows" means 1000 in the example above > * "many shards" means 200-1000 in our case > * "all with many hits" means that each of the shards have a significant > number of hits on the query > The problem grows on all three factors above > Doing such a query on our system takes between 5 min to 1 hour - depending on > a lot of things. It ought to be much faster, so lets make it. > Profiling show that the problem is that it takes lots of time to access the > store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. > Having 1000 shards its up to 1 mio ids that has to be fetched. There is > really no good reason to ever read information from store for more than the > overall top-1000 documents, that has to be returned to the client. > For further detail see mail-thread "Slow searching limited but high rows > across many shards all with high hits" started 13/11-2014 on > dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Running 5.1.0 test-suite via maven
Now trying to run ant test with java7. I get the follow failing tests. Hmmm
[junit4] Suite:
org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestDirectoryTaxonomyWriter -Dtests.method=testConcurrency
-Dtests.seed=6A95BF7E6A49187D -Dtests.slow=true -Dtests.locale=en_ZA
-Dtests.timezone=America/St_Vincent -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] ERROR 3.81s J2 |
TestDirectoryTaxonomyWriter.testConcurrency <<<
[junit4]> Throwable #1: java.lang.NoSuchMethodError:
java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
[junit4]> at
__randomizedtesting.SeedInfo.seed([6A95BF7E6A49187D:CC657DE30B1AF8B4]:0)
[junit4]> at
org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter.testConcurrency(TestDirectoryTaxonomyWriter.java:309)
[junit4]> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50):
{$full_path$=PostingsFormat(name=LuceneVarGapDocFreqInterval),
$payloads$=PostingsFormat(name=LuceneVarGapDocFreqInterval),
$facets=PostingsFormat(name=LuceneVarGapFixedInterval)},
docValues:{$facets=DocValuesFormat(name=Lucene50)},
sim=DefaultSimilarity, locale=en_ZA, timezone=America/St_Vincent
[junit4] 2> NOTE: Mac OS X 10.10.2 x86_64/Oracle Corporation
1.7.0_51 (64-bit)/cpus=8,threads=1,free=193456536,total=236453888
[junit4] 2> NOTE: All tests run in this JVM:
[TestTaxonomyCombined, TestDirectoryTaxonomyWriter]
[junit4] Completed on J2 in 6.67s, 15 tests, 1 error <<< FAILURES!
[junit4]
[junit4] Suite: org.apache.lucene.facet.TestMultipleIndexFields
[junit4] Completed on J2 in 1.43s, 5 tests
[junit4]
[junit4] Suite:
org.apache.lucene.facet.taxonomy.TestTaxonomyFacetSumValueSource
[junit4] Completed on J2 in 2.62s, 9 tests
[junit4]
[junit4] Suite: org.apache.lucene.facet.TestDrillSideways
[junit4] Completed on J3 in 10.73s, 6 tests
[junit4]
[junit4] Suite:
org.apache.lucene.facet.taxonomy.directory.TestConcurrentFacetedIndexing
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestConcurrentFacetedIndexing -Dtests.method=testConcurrency
-Dtests.seed=6A95BF7E6A49187D -Dtests.slow=true -Dtests.locale=de_DE
-Dtests.timezone=SST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 1.54s J2 |
TestConcurrentFacetedIndexing.testConcurrency <<<
[junit4]> Throwable #1: java.lang.NoSuchMethodError:
java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
[junit4]> at
__randomizedtesting.SeedInfo.seed([6A95BF7E6A49187D:CC657DE30B1AF8B4]:0)
[junit4]> at
org.apache.lucene.facet.taxonomy.directory.TestConcurrentFacetedIndexing.testConcurrency(TestConcurrentFacetedIndexing.java:142)
[junit4]> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50):
{$full_path$=PostingsFormat(name=Direct),
$payloads$=PostingsFormat(name=Direct)}, docValues:{},
sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=de_DE,
timezone=SST
[junit4] 2> NOTE: Mac OS X 10.10.2 x86_64/Oracle Corporation
1.7.0_51 (64-bit)/cpus=8,threads=1,free=168753144,total=240648192
[junit4] 2> NOTE: All tests run in this JVM:
[TestTaxonomyCombined, TestDirectoryTaxonomyWriter,
TestMultipleIndexFields, TestTaxonomyFacetSumValueSource,
TestConcurrentFacetedIndexing]
[junit4] Completed on J2 in 1.61s, 1 test, 1 error <<< FAILURES!
On 24/04/15 22:11, Dawid Weiss wrote:
1. does it fail when you run it via ant (ant build)?
2. the problem is very likely in dependencies declared in Maven (as
compared to what the ivy dependencies are).
Dawid
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Running 5.1.0 test-suite via maven
I tried running "ant test". I get the test-errors shown below
* "java.lang.NoClassDefFoundError: Could not initialize class
java.lang.UNIXProcess" might be because I am running it on java 8 (I
thought source/target set to 1.7 would be good enough)
* Do not understand why I get "java.lang.NoClassDefFoundError: Could not
initialize class org.apache.hadoop.util.StringUtils".
org.apache.hadoop/hadoop-common/2.6.0 is in ~/.ivy2 (and ~/.m2 for that
matter)
Any help appreciated!
Regards, Per Steffensen
-
[junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
[junit4] 2> Creating dataDir:
/Users/steff/LiaceDisk/lucene_solr_5_1_0/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest
9A1D1C06EE13AC04-001/init-core-data-001
[junit4] 2> 1527744 T5936
oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext
system property: /
[junit4] 2> 1529116 T5936 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50),
sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=hu_HU,
timezone=Pacific/Truk
[junit4] 2> NOTE: Mac OS X 10.10.2 x86_64/Oracle Corporation
1.8.0_25 (64-bit)/cpus=8,threads=1,free=285813160,total=440401920
[junit4] 2> NOTE: All tests run in this JVM: [HighlighterTest,
CoreMergeIndexesAdminHandlerTest, OpenCloseCoreStressTest,
FileUtilsTest, HdfsBasicDistributedZk2Test, TestExactSharedStatsCache,
DistributedQueryComponentCustomSortTest, TestCloudInspectUtil,
TestQuerySenderNoQuery, SimplePostToolTest, TestSchemaNameResource,
TestFieldCollectionResource, BasicFunctionalityTest,
TermVectorComponentDistributedTest, IndexBasedSpellCheckerTest,
LeaderElectionTest, TestRandomDVFaceting, ShowFileRequestHandlerTest,
TestReplicationHandler, InfoHandlerTest, DistributedQueueTest,
TestStressReorder, CursorPagingTest, ScriptEngineTest,
XmlUpdateRequestHandlerTest, ShardRoutingCustomTest,
ClusterStateUpdateTest, TestJsonRequest, TestBM25SimilarityFactory,
MigrateRouteKeyTest, RollingRestartTest, DocValuesTest, PolyFieldTest,
TestSimpleQParserPlugin, TimeZoneUtilsTest, ReplicationFactorTest,
TestJettySolrRunner, DistributedSuggestComponentTest, TestRTGBase,
TestBlobHandler, TestMaxScoreQueryParser, TestHashQParserPlugin,
CSVRequestHandlerTest, TestDocumentBuilder, SolrPluginUtilsTest,
DistributedFacetPivotSmallTest, TestManagedSchema, RecoveryZkTest,
HdfsBasicDistributedZkTest, SimpleCollectionCreateDeleteTest,
TestFastWriter, TestRequestStatusCollectionAPI, DistanceUnitsTest,
TestConfigSets, TestManagedResourceStorage, BadCopyFieldTest,
TestPartialUpdateDeduplication, TestCryptoKeys, MinimalSchemaTest,
SolrInfoMBeanTest, TestCursorMarkWithoutUniqueKey,
TestDistributedGrouping, TestGroupingSearch, TestHighlightDedupGrouping,
TestJoin, TestTolerantSearch, TestTrie,
PathHierarchyTokenizerFactoryTest, TestReversedWildcardFilterFactory,
ActionThrottleTest, AliasIntegrationTest, AssignTest,
BasicDistributedZk2Test, FullSolrCloudDistribCmdsTest,
LeaderInitiatedRecoveryOnCommitTest, OverseerTest,
SharedFSAutoReplicaFailoverUtilsTest, SliceStateTest, SyncSliceTest,
ZkNodePropsTest, HdfsChaosMonkeySafeLeaderTest,
HdfsCollectionsAPIDistributedZkTest, HdfsWriteToMultipleCollectionsTest,
StressHdfsTest]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=StressHdfsTest -Dtests.seed=9A1D1C06EE13AC04
-Dtests.slow=true -Dtests.locale=hu_HU -Dtests.timezone=Pacific/Truk
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J1 | StressHdfsTest (suite) <<<
[junit4]> Throwable #1: java.lang.NoClassDefFoundError: Could
not initialize class java.lang.UNIXProcess
[junit4]> at
__randomizedtesting.SeedInfo.seed([9A1D1C06EE13AC04]:0)
[junit4]> at java.lang.ProcessImpl.start(ProcessImpl.java:130)
[junit4]> at
java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
[junit4]> at
org.apache.hadoop.util.Shell.runCommand(Shell.java:485)
[junit4]> at org.apache.hadoop.util.Shell.run(Shell.java:455)
[junit4]> at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
[junit4]> at
org.apache.hadoop.util.Shell.isSetsidSupported(Shell.java:390)
[junit4]> at
org.apache.hadoop.util.Shell.(Shell.java:380)
[junit4]> at
org.apache.hadoop.util.StringUtils.(StringUtils.java:79)
[junit4]> at
org.apache.hadoop.conf.Configuration.getTrimmedStringCollection(Configuration.java:1747)
[junit4]> at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStorageDirs(FSNamesystem.java:1407)
[junit4]> at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceDirs(FSNamesystem.java:1388)
[junit4]> at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:931)
Running 5.1.0 test-suite via maven
I have checked out tag lucene_solr_5_1_0 and try to run the test-suite using maven ant get-maven-poms cd maven-build/ mvn -N -Pbootstrap -DskipTests install mvn -Dmaven.test.failure.ignore=true test The following tests fail running the test-suite this way using maven * SuggestFieldTest - all test methods fail * hadoop.MorphlineBasicMiniMRTest and hadoop.MorphlineGoLiveMiniMRTest - setupClass fails * TestCoreDiscovery.testCoreDirCantRead - test fails Now trying to run in Eclipse ant eclipse Open Eclipse and import project Running the failing tests * SuggestFieldTest - test passes * hadoop.MorphlineBasicMiniMRTest and hadoop.MorphlineGoLiveMiniMRTest - setupClass fails * TestCoreDiscovery.testCoreDirCantRead - test passes Can anyone explain this? Would expect a green test-suite for a released solr/lucene. What might be wrong with SuggestFieldTest and TestCoreDiscovery tests run through maven (fail), compared to run through Eclipse (pass)? Why does the Morphline tests fail (both maven and Eclipse)? It is the correct way to run tests using maven? It there another way the test-suite is usually run (e.g. using ant)? Thanks in advance. Regards, Per Steffensen - fail stacktraces --- * SuggestFieldTest fails using maven (not Eclipse) like this: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [MockRandom, RAMOnly, LuceneFixedGap, LuceneVarGapFixedInterval, LuceneVarGapDocFreqInterval, TestBloomFilteredLucenePostings, Asserting, Lucene50, BlockTreeOrds, BloomFilter, Direct, FSTOrd50, FST50, Memory, SimpleText] at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:111) at org.apache.lucene.codecs.PostingsFormat.forName(PostingsFormat.java:100) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:255) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:336) at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:104) at org.apache.lucene.index.SegmentReader.(SegmentReader.java:65) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:132) at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:184) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:99) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:429) at org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:343) at org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:280) at org.apache.lucene.search.suggest.document.SuggestFieldTest.testReturnedDocID(SuggestFieldTest.java:459) * The Morphline tests fail (both maven and Eclipse) like this: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryWriter at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createRMApplicationHistoryWriter(ResourceManager.java:357) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:468) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:989) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:255) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.solr.hadoop.hack.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:200) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.solr.hadoop.hack.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:83) at org.apache.solr.hadoop.hack.MiniMRCluster.(MiniMRCluster.java:191) at org.apache.solr.hadoop.hack.MiniMRCluster.(MiniMRCluster.java:1
[jira] [Updated] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Per Steffensen updated SOLR-7176:
-
Attachment: SOLR-7176.patch
Added "Try again" in error-msg, also on {{NodeExistsException}}
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch,
> SOLR-7176.patch, SOLR-7176.patch
>
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506567#comment-14506567
]
Per Steffensen commented on SOLR-7176:
--
bq. This would also be valid for NodeExistsException
Hmmm, believe you are right
bq. I also thought about it. But it seems like when ZkStateReader does not own
the ZK client (e.g. in CollectionsHandler), it shouldn't close it? Any thoughts?
ZkStateReader does not own the ZK client if you use the
{{ZkStateReader(SolrZkClient zkClient)}} constructor - and we do in all the
mentioned caces. And it does not close the ZK client on close, when it does not
own it. Hence in the mentioned cases calling close really does not do much,
except setting closed=true. I just thought it was the right thing to do - call
close when you create and finished using a closable component.
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch,
> SOLR-7176.patch
>
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Per Steffensen updated SOLR-7176:
-
Attachment: SOLR-7176.patch
New patch
bq. I would prefer not to retry automatically
Adding "Try again" to error-msg from ZkCLI if
KeeperException.BadVersionException (to hint the user that it will probably go
well if he just tries again).
Besides that
* Also "// Don't update ZK unless absolutely necessary" in the remove-case
* Sharing constants instead of introducing new similar ones. Because IMHO we
shouldnt keep adding the same constants again and again, but more important: If
HTTP is
{{/admin/collections?action=CLUSTERPROP&name=&val=}},
then the command-line should be {{zkcli.sh … -cmd CLUSTERPROP -name
-val }}, AND if the HTTP ever changes (e.g. using “clusterprop.set”
instead of “CLUSTERPROP”) so should the command-line
* Adding test of remove
* Closing ZkStateReader in CollectionsHandler, ZkCLI and ZkCLITest
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch,
> SOLR-7176.patch
>
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504498#comment-14504498
]
Per Steffensen commented on SOLR-7176:
--
bq. Hence the fix for this use-case can not really depend upon the overseer lock
Sure you can. You can say that you have to own the Overseer-lock, in order to
be able to perform this kind of admin-tasks. The CLI job can try to take the
lock and then perform the operation. If it cannot get the lock (maybe retry for
a period of time), ask the Overseer to do it (either by doing the corresponding
HTTP request or by leaving it directly on the Overseer-queue) and wait
synchronously for the Overseer to handle it (it must be running since it the
lock is taken).
But optimistic locking (compare-and-swap) is probably the best in this case.
Only thing I fear is that if that approach is established as "the way it is
done" it will be repeated in upcoming cases where it might not be sufficient.
Sometime it is worth the effort to establish the best platform to build upon
from the beginning.
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch
>
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503430#comment-14503430
]
Per Steffensen edited comment on SOLR-7176 at 4/20/15 7:02 PM:
---
Believe it is referred to as compare-and-swap, but anyway
{{Overseer.handleProp}} does not use that feature (it always uses version=-1
when calling zk.setData). But actually I just realized that it is not true that
{{CollectionsHandler.handleProp}} does the fetch-update and only leaves update
to Overseer (as I claimed above) - it actually does forward the entire
operation to Overseer so that the Overseer does fetch, update and store (just
at I wanted it to). So you are right, that the problem I sketched above is not
a problem in todays code, but it is not due to usage of compare-and-swap - it
is because two threads will never try to do updates at the same time - they
will ask the Overseer to do the updates (single-threaded = primitive
pessimistic lock).
So we cannot "just eliminate the overseer from the picture completely", if we
still want to avoid the (more or less "theoretical") problem I sketched above.
So the bullet-safe command-line solution should take this into consideration -
e.g.
* 1) Take the overseer-lock before performing the operation
* 2) Use compare-and-swap (and make sure Overseer also does)
I believe I would prefer 1) because it is the most generally useable solution
to the problem. Compare-and-swap (even combined with ZK multi-op feature) will
not always be sufficient for operations that want to update several znodes
atomically - and who knows, maybe some day we also want to that kind of stuff
using command-line. Taking a pessimistic lock (like the Overseer-lock) always
will be sufficient.
was (Author: steff1193):
Believe it is referred to as compare-and-swap, but anyway
{{Overseer.handleProp}} does not use that feature (it always uses version=-1
when calling zk.setData). But actually I just realized that it is not true that
{{CollectionsHandler.handleProp}} does the fetch-update and only leaves update
to Overseer (as I claimed above) - it actually does forward the entire
operation to Overseer so that the Overseer does fetch, update and store (just
at I wanted it to). So you are right, that the problem I sketched above is not
a problem in todays code, but it is not due to usage of compare-and-swap - it
is because two threads will never try to do updates at the same time - they
will ask the Overseer to do the updates (single-threaded = primitive
pessimistic lock).
So we cannot "just eliminate the overseer from the picture completely", if we
still want to avoid the (more or less "theoretical") problem I sketched above.
So the bullet-safe command-line solution should take this into consideration -
e.g.
* 1) Take the overseer-lock before performing the operation
* 2) Use compare-and-swap (and make sure Overseer also does)
I would prefer 1) because it is the most generally useable solution to the
problem. Compare-and-swap (even combined with ZK multi-op feature) will not
always be sufficient for operations that want to update several znodes
atomically - and who knows, maybe some day we also want to that kind of stuff
using command-line. Taking a pessimistic lock (like the Overseer-lock) always
will be sufficient.
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch
>
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503430#comment-14503430
]
Per Steffensen commented on SOLR-7176:
--
Believe it is referred to as compare-and-swap, but anyway
{{Overseer.handleProp}} does not use that feature (it always uses version=-1
when calling zk.setData). But actually I just realized that it is not true that
{{CollectionsHandler.handleProp}} does the fetch-update and only leaves update
to Overseer (as I claimed above) - it actually does forward the entire
operation to Overseer so that the Overseer does fetch, update and store (just
at I wanted it to). So you are right, that the problem I sketched above is not
a problem in todays code, but it is not due to usage of compare-and-swap - it
is because two threads will never try to do updates at the same time - they
will ask the Overseer to do the updates (single-threaded = primitive
pessimistic lock).
So we cannot "just eliminate the overseer from the picture completely", if we
still want to avoid the (more or less "theoretical") problem I sketched above.
So the bullet-safe command-line solution should take this into consideration -
e.g.
* 1) Take the overseer-lock before performing the operation
* 2) Use compare-and-swap (and make sure Overseer also does)
I would prefer 1) because it is the most generally useable solution to the
problem. Compare-and-swap (even combined with ZK multi-op feature) will not
always be sufficient for operations that want to update several znodes
atomically - and who knows, maybe some day we also want to that kind of stuff
using command-line. Taking a pessimistic lock (like the Overseer-lock) always
will be sufficient.
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch
>
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502904#comment-14502904
]
Per Steffensen commented on SOLR-7176:
--
bq. Why can't we just eliminate the overseer from the picture completely?
Not that it is very important in this case, but there is a problem in general
with having several threads doing fetch -> update-locally -> store on state
concurrently without locking (pessimistically or optimistically). Example, two
threads running concurrently
* Thread#1 wants to do the task of setting "urlScheme" to "http":
** fetches {"urlScheme":"https", "autoAddReplicas": "true"}
** changes it to {"urlScheme":"http", "autoAddReplicas": "true"} and stores it
* Thread#2 wants to do the task of setting "autoAddReplicas" to "false":
** fetches {"urlScheme":"https", "autoAddReplicas": "true"}
** changes it to {"urlScheme":"https", "autoAddReplicas": "false"} and stores it
Without locking they can run concurrently and you will end up with a wrong state
* {"urlScheme":"http", "autoAddReplicas": "true"}
* or {"urlScheme":"https", "autoAddReplicas": "false"}
But you actually expected {"urlScheme":"http", "autoAddReplicas": "false"}
I do not know what the initial thought was about the Overseer, but I think of
it as a simple way to get around this locking - making sure that there is never
more than one thread updating state.
When that is said, if the above was the intention with the Overseer, it does
not work today, because CollectionsHandler.handleProp is doing the fetch and
update, and only leaves the storing to Overseer. I would like to see the entire
job handed over to the Overseer, so that it does both fetch, update and store -
so that you can avoid the concurrency scenario above. In general Overseer
should execute entire admin-jobs and not only parts of them.
Anyway, it is a reason not to do this kind of updates without taking locks, and
Overseer is a primitive way of "taking lock", and maybe therefore "do not
eliminate the Overseer". I am not sure it is especially important here.
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-7176.patch, SOLR-7176.patch
>
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500039#comment-14500039
]
Per Steffensen commented on SOLR-7176:
--
bq. With the current setup, you can use "bin/solr" at the commandline on *NIX
and "bin\solr" on Windows, the only difference is the path separator, which
will not be a surprise to most admins
Well I think it might come as a surprise to most *NIX admins that the script is
just called "solr" - and not e.g. solr.sh. But never mind, this JIRA is not
about that. I just had a hard time writing {{solr CLUSTERPROP ...}}, because I
would have to think twice to understand it myself
bq. and it should be handled in a separate issue
Yes, definitely, no one talked about doing the renaming in this issue
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Priority: Minor
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500028#comment-14500028
]
Per Steffensen edited comment on SOLR-7176 at 4/17/15 3:27 PM:
---
I agree, but from time to time I want to add a (async) command to the overseer
while the cluster is not running, expecting the overseer to pick it up and
execute it when I start my cluster. If you would enable this tool to do this
kind of stuff, then suddenly most of the cluster-commands become relevant for
this tool - if it is able to both execute the command directly (if supported -
e.g. the {{CLUSTERPROP}} command) or to leave the command for execution by the
overseer.
And, if you have numerous machines that might or might not currently run a
Solr-node, maybe you actually want to be able to run the {{OVERSEERSTATUS}}
command as a command-line just to get a "not running" response.
was (Author: steff1193):
I agree, but from time to time I want to add a (async) command to the overseer
while the cluster is not running, expecting the overseer to pick it up and
execute it when I start my cluster. If you would enable this tool to do this
kind of stuff, then suddenly most of the cluster-commands become relevant for
this tool - if it is able to both execute the command directly (if supported -
e.g. by {{CLUSTERPROP}} command) or to leave the command for execution by the
overseer.
And, if you have numerous machines that might or might not currently run a
Solr-node, maybe you actually want to be able to run the {{OVERSEERSTATUS}}
command as a command-line just to get an "not running" response.
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Priority: Minor
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500028#comment-14500028
]
Per Steffensen commented on SOLR-7176:
--
I agree, but from time to time I want to add a (async) command to the overseer
while the cluster is not running, expecting the overseer to pick it up and
execute it when I start my cluster. If you would enable this tool to do this
kind of stuff, then suddenly most of the cluster-commands become relevant for
this tool - if it is able to both execute the command directly (if supported -
e.g. by {{CLUSTERPROP}} command) or to leave the command for execution by the
overseer.
And, if you have numerous machines that might or might not currently run a
Solr-node, maybe you actually want to be able to run the {{OVERSEERSTATUS}}
command as a command-line just to get an "not running" response.
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Priority: Minor
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499632#comment-14499632
]
Per Steffensen commented on SOLR-7176:
--
{quote}
{code}
zkcli.sh -zkhost 127.0.0.1:9983 -collection-action CLUSTERPROP -name urlScheme
-val https
{code}
{quote}
I agree, except that it should not be the zkcli.sh tool that is extended. Since
it is the collections API you make a CLI for, so to speak, make a
collectionscli.sh script
{code}
collectionscli.sh -zkhost 127.0.0.1:9983 -action CLUSTERPROP -name urlScheme
-val https
{code}
And later maybe
{code}
collectionscli.sh -zkhost 127.0.0.1:9983 -action ADDROLE -role overseer -val ...
{code}
etc
It think also, that it needs to be considered how and if this is an
extension/modification to the SolrCLI-tool (used from solr/bin/solr and
solr/bin/solr.cmd)
{code}
solr.sh CLUSTERPROP -zkhost 127.0.0.1:9983 -name urlScheme -val https
{code}
Just saying, even though I do not like the current state of it, because of the
enormous amounts of redundant code. But we do not want to end up with a million
different cli-tools either.
BTW, I think solr/bin/solr should be renamed to solr.sh, so I pretended above
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Priority: Minor
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7236) Securing Solr (umbrella issue)
[ https://issues.apache.org/jira/browse/SOLR-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388230#comment-14388230 ] Per Steffensen commented on SOLR-7236: -- Sorry [~janhoy] - will stop now. The discussion is related to security, but probably not enough to be discussed here. Guess this JIRA just deals with the fact that Solr-node is not necessarily web-container in the future - then what to do about security. Great initiative! > Securing Solr (umbrella issue) > -- > > Key: SOLR-7236 > URL: https://issues.apache.org/jira/browse/SOLR-7236 > Project: Solr > Issue Type: New Feature >Reporter: Jan Høydahl > Labels: Security > > This is an umbrella issue for adding security to Solr. The discussion here > should discuss real user needs and high-level strategy, before deciding on > implementation details. All work will be done in sub tasks and linked issues. > Solr has not traditionally concerned itself with security. And It has been a > general view among the committers that it may be better to stay out of it to > avoid "blood on our hands" in this mine-field. Still, Solr has lately seen > SSL support, securing of ZK, and signing of jars, and discussions have begun > about securing operations in Solr. > Some of the topics to address are > * User management (flat file, AD/LDAP etc) > * Authentication (Admin UI, Admin and data/query operations. Tons of auth > protocols: basic, digest, oauth, pki..) > * Authorization (who can do what with what API, collection, doc) > * Pluggability (no user's needs are equal) > * And we could go on and on but this is what we've seen the most demand for -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7236) Securing Solr (umbrella issue)
[ https://issues.apache.org/jira/browse/SOLR-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388227#comment-14388227 ] Per Steffensen commented on SOLR-7236: -- I am ok with embedding Jetty, and you are right, that there are probably lots of things that would be easier. Just make sure that you can still participate in configuring it from the outside - jetty.xml and web.xml. At least until an alternative solution gives the same flexibility. What I fear is that we remove all the flexibility of web-container - because we are using - including its ability to handle security. I checked out 5.0.0 code, but I am not able to see that Solr-node is not still just Jetty on top-level, and that Solr does not control anything before web.xml/SolrDispatchFilter. Can you please point me to some of the more important JIRAs around this "hiding/removing web-container" initiative. Thanks! Just want to understand what has been done/achieved until now. > Securing Solr (umbrella issue) > -- > > Key: SOLR-7236 > URL: https://issues.apache.org/jira/browse/SOLR-7236 > Project: Solr > Issue Type: New Feature >Reporter: Jan Høydahl > Labels: Security > > This is an umbrella issue for adding security to Solr. The discussion here > should discuss real user needs and high-level strategy, before deciding on > implementation details. All work will be done in sub tasks and linked issues. > Solr has not traditionally concerned itself with security. And It has been a > general view among the committers that it may be better to stay out of it to > avoid "blood on our hands" in this mine-field. Still, Solr has lately seen > SSL support, securing of ZK, and signing of jars, and discussions have begun > about securing operations in Solr. > Some of the topics to address are > * User management (flat file, AD/LDAP etc) > * Authentication (Admin UI, Admin and data/query operations. Tons of auth > protocols: basic, digest, oauth, pki..) > * Authorization (who can do what with what API, collection, doc) > * Pluggability (no user's needs are equal) > * And we could go on and on but this is what we've seen the most demand for -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386432#comment-14386432 ] Per Steffensen edited comment on SOLR-6816 at 3/30/15 9:22 AM: --- bq. I think you mis-understood my point. I wasn't talking about retrying documents in the same UpdateRequest. If a Map/Reduce task fails, the HDFS block is retried entirely, meaning a Hadoop-based indexing job may send the same docs that have already been added so using overwite=false is dangerous when doing this type of bulk indexing. The solution proposed in SOLR-3382 would be great to have as well though. Well, we might be mis-understanding each other. Im am not talking about retrying documents in the same UpdateRequest either. What we have: Our indexing client (something not in Solr - think of it as the Map/Reduce job) decides to do 1000 update-doc-commands U1, U2, ... , U1000 (add-doc and delete-doc commands), by sending one bulk-job containing all of those to Solr-node S1. S1 handles some of the Us itself and forwards other Us to the other Solr-nodes - depending or routing. For simplicity lets say that we have three Solr-nodes S1, S2 and S3 and that S1 handles U1-U333 itself, forwards U334-U666 to S2 and U667-U1000 to S3. Now lets say that U100, U200, U400, U500, U700 and U800 fails (two on each Solr-node), and the rest succeeds. S1 gets that information back from S2 and S3 (including reasons for each U that failed), and is able to send a response to our indexing client saying that all was a success, except that U100, U200, U400, U500, U700 and U800 failed, and why they failed. Some might fail due to DocumentAlreadyExists (if U was about creating a new document, assuming that it does not already exist), others might fail due to VersionConflict (if U was about updating an existing document and includes its last known (to the client) version, but the document at server has a higher version-number), others again might fail due to DocumentDoesNotExist (if U was about updating an existing document, but that document does not exist (anylonger) at server). Our indexing client takes note of that combined response from S1, performs the appropriate actions (e.g. version-lookups) and sends a new request to the Solr-cluster now only including U100', U200', U400', U500', U700' and U800'. We have done it like that for a long time, using our solution to EDR-3382 (and our solution to SOLR-3178). I would expect a Map/Reduce-job could do the same, playing the role as the indexing client. Essentially only resending (maybe by issuing a new Map/Reduce-job from the "reduce"-phase of the first Map/Reduce-job) the (modified versions of) update-commands that failed the first time. was (Author: steff1193): bq. I think you mis-understood my point. I wasn't talking about retrying documents in the same UpdateRequest. If a Map/Reduce task fails, the HDFS block is retried entirely, meaning a Hadoop-based indexing job may send the same docs that have already been added so using overwite=false is dangerous when doing this type of bulk indexing. The solution proposed in SOLR-3382 would be great to have as well though. Well, we might be mis-understanding each other. Im am not talking about retrying documents in the same UpdateRequest either. What we have: Our indexing client (something not in Solr - think of it as the Map/Reduce job) decides to do 1000 update-doc-commands U1, U2, ... , U1000 (add-doc and delete-doc commands), by sending one bulk-job containing all of those to Solr-node S1. S1 handles some of the Us itself and forwards other Us to the other Solr-nodes - depending or routing. For simplicity lets say that we have three Solr-nodes S1, S2 and S3 and that S1 handles U1-U333 itself, forwards U334-U666 to S2 and U667-U1000 to S3. Now lets say that U100, U200, U400, U500, U700 and U800 fails (two on each Solr-node), and the rest succeeds. S1 gets that information back from S2 and S3 (including reasons for each U that failed), and is able to send a response to our indexing client saying that all was a success, except that U100, U200, U400, U500, U700 and U800 failed, and why they failed. Some might fail due to DocumentDoNotExist (if U was about creating a new document, assuming that it does not already exist), others might fail due to VersionConflict (if U was about updating an existing document and includes its last known (to the client) version, but the document at server has a higher version-number), other might fail due to DocumentDoesNotExist (if U was about updating an existing document, but that document does not exist (anylonger) at server). Our indexing client takes note of that combined response from S1, perform the appropriate actions (e.g. version-lookups) and sends a new request to the Solr-cluster now
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386432#comment-14386432 ] Per Steffensen commented on SOLR-6816: -- bq. I think you mis-understood my point. I wasn't talking about retrying documents in the same UpdateRequest. If a Map/Reduce task fails, the HDFS block is retried entirely, meaning a Hadoop-based indexing job may send the same docs that have already been added so using overwite=false is dangerous when doing this type of bulk indexing. The solution proposed in SOLR-3382 would be great to have as well though. Well, we might be mis-understanding each other. Im am not talking about retrying documents in the same UpdateRequest either. What we have: Our indexing client (something not in Solr - think of it as the Map/Reduce job) decides to do 1000 update-doc-commands U1, U2, ... , U1000 (add-doc and delete-doc commands), by sending one bulk-job containing all of those to Solr-node S1. S1 handles some of the Us itself and forwards other Us to the other Solr-nodes - depending or routing. For simplicity lets say that we have three Solr-nodes S1, S2 and S3 and that S1 handles U1-U333 itself, forwards U334-U666 to S2 and U667-U1000 to S3. Now lets say that U100, U200, U400, U500, U700 and U800 fails (two on each Solr-node), and the rest succeeds. S1 gets that information back from S2 and S3 (including reasons for each U that failed), and is able to send a response to our indexing client saying that all was a success, except that U100, U200, U400, U500, U700 and U800 failed, and why they failed. Some might fail due to DocumentDoNotExist (if U was about creating a new document, assuming that it does not already exist), others might fail due to VersionConflict (if U was about updating an existing document and includes its last known (to the client) version, but the document at server has a higher version-number), other might fail due to DocumentDoesNotExist (if U was about updating an existing document, but that document does not exist (anylonger) at server). Our indexing client takes note of that combined response from S1, perform the appropriate actions (e.g. version-lookups) and sends a new request to the Solr-cluster now only including U100', U200', U400', U500', U700' and U800'. We have done it like that for a long time, using our solution to EDR-3382 (and our solution to SOLR-3178). I would expect a Map/Reduce-job could do the same, playing the role as the indexing client. Essentially only resending (maybe by issuing a new Map/Reduce-job from the "reduce"-phase of the first Map/Reduce-job) the (modified) update-commands that failed the first time. > Review SolrCloud Indexing Performance. > -- > > Key: SOLR-6816 > URL: https://issues.apache.org/jira/browse/SOLR-6816 > Project: Solr > Issue Type: Task > Components: SolrCloud >Reporter: Mark Miller >Priority: Critical > Attachments: SolrBench.pdf > > > We have never really focused on indexing performance, just correctness and > low hanging fruit. We need to vet the performance and try to address any > holes. > Note: A common report is that adding any replication is very slow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7236) Securing Solr (umbrella issue)
[ https://issues.apache.org/jira/browse/SOLR-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386392#comment-14386392 ] Per Steffensen commented on SOLR-7236: -- Thanks for a constructive answer! bq. Because as long as the network and HTTP are handled by software that is outside of Solr, Solr has absolutely no ability to control it. First of all, that is not true. Second, I am pretty sure that Solr does not have any ambition to deal with all the hard low-level network stuff itself. We will always use a 3rd party component for that - Netty, Spray or whatever. So it will always be a 3rd party component that connects with the network, receives requests, parses them etc before handing over control to Solr. So why not let Jetty (or any other web-container) do that - they do the job pretty well. bq. Ideally, you should be able to drop a configuration right in a handler definition (such as the one for /select) found in solrconfig.xml, listing security credentials (username/password, IP address, perhaps even certificate information) that you are willing to accept for that handler, along with exceptions or credentials that will allow SolrCloud inter-node communication to work. You can do that with a web-container, and I do believe that the way you would do it will not change much whether you want to do it with Jetty or Netty. The HttpServletRequest handed over by the web-container contains everything you need (maybe together with the web-container context), just as it probably would with any other network component. You can plug things into the web-container just as you probably can with any other network component. If you give me an more exact example of what you want to achieve, that you believe cannot be achieved with a web-container, but you believe can be achieved with the other approach, I would love to make the code showing that you are wrong. If I can’t, I will forever keep quiet - and that in itself is quit an achievement. bq. Bringing the servlet container under our control as we did with 5.0 (with initial work in 4.10) allows us to tell people how to configure the servlet container for security in a predictable manner Yes, and if it was a web-container that had control, your could point to web-container documentation for info about how to configure. Even though I think it is an unnecessary move, it is an ok idea to say that Solr IS running in Jetty, making us able to tell exactly how to configure whatever you need to. If you want to use any other web-container, you are on your own. But leaving it a web-app behind the scenes would be great, so that advanced users can still take advantage of that. The problem, though, is that you lock with Jetty, and Jetty becomes an “implementation detail” of Solr. Do not do that if it is not necessary, and I still have not seen very good reasons to do it. But at least I recognize that there might be good reasons. I am not sure about what you actually did in 5.0 (with initial work in 4.10). As far as I can see Solr still starts out by starting Jetty. Can you point me to some of the most important JIRAs for the remove-web-container initiative. Thanks! bq. but it is still not *Solr* and its configuration that's controlling it. But it can be, without removing the web-container. The thing I fear is spending a lot of resources and time removing Jetty and replacing it with lots of other 3rd party components (e.g. including Netty), and at best just reach status quo after a year or two. This entire umbrella-issue (SOLR-7236) is only/mainly necessary because of the moving-out-of-web-container initiative. The fact that Solr is running in a web-container makes it very flexible - e.g. my projects have significant changes to both web.xml and jetty.xml. Other people might have similar setups just with Tomcat or whatever. Establishing the same kind of flexibility without a web-container will take years. In my organization we started out using ElasticSearch, but for political reasons we moved to SolrCloud. The first thing that made me happy about that, was the fact that Solr(Cloud) is a web-app, because I know exactly how they work, they are standardized and flexible - believe a lot of people feel the same way At least, if you insist on removing the web-container, make sure not to do it before all the flexibility it gives can somehow be achieved in another way. If you really wanted to do cool stuff in this area, making Solr(Cloud) based on dependency injection (configuration and/or annotation) would be great - e.g. using Spring or Guice. Both top-level Solr, but also sub-parts of Solr. E.g. the fact that solrconfig.xml is a self-invented configuration-structure that screams to be replaced by (de-facto) standardized dependency injection is a major problem. Sorry for partly highjacking the issue, [~janhoy] - I did not manag
[jira] [Commented] (SOLR-7236) Securing Solr (umbrella issue)
[ https://issues.apache.org/jira/browse/SOLR-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383624#comment-14383624 ] Per Steffensen commented on SOLR-7236: -- I am trying to understand why Solr wants to move away from being a web-app running in a servlet-container. A servlet-container can deal with all of the things you mention in a standardized way bq. User management (flat file, AD/LDAP etc) That should really not be a Solr-concern. If you want your users/credentials in LDAP manage it through an LDAP management-interface, if you want them in flat files, let the admin change the files, etc bq. Authentication (Admin UI, Admin and data/query operations. Tons of auth protocols: basic, digest, oauth, pki..) Implement your own realm authenticating against whatever credentials-store you want with whatever protocol you want - and plug it into your servlet-container. Or use a 3party realm that authenticates against your preferred credentials-store and supporting your auth protocol. It is easy to find realms for Jetty (or Tomcat or ...) authenticating against flat files, LDAP or whatever. We wrote our own realm, authenticating against credentials-store in ZK - easy. bq. Authorization (who can do what with what API, collection, doc) Let your realm list roles that an authenticated user is allowed to play. Setup web.xml mapping roles to the url-paths they are allowed use. If limited flexibility (url-mapping with stupid limitations) of web.xml authentication is not good enough for you, implement it as a Filter yourself, or use some of the Filters already out there. E.g. I provided a reg-exp authorization filter with SOLR-4470 - it is very powerful. bq. Pluggability (no user's needs are equal) It is all pluggable - realms, web.xml, filters etc > Securing Solr (umbrella issue) > -- > > Key: SOLR-7236 > URL: https://issues.apache.org/jira/browse/SOLR-7236 > Project: Solr > Issue Type: New Feature >Reporter: Jan Høydahl > Labels: Security > > This is an umbrella issue for adding security to Solr. The discussion here > should discuss real user needs and high-level strategy, before deciding on > implementation details. All work will be done in sub tasks and linked issues. > Solr has not traditionally concerned itself with security. And It has been a > general view among the committers that it may be better to stay out of it to > avoid "blood on our hands" in this mine-field. Still, Solr has lately seen > SSL support, securing of ZK, and signing of jars, and discussions have begun > about securing operations in Solr. > Some of the topics to address are > * User management (flat file, AD/LDAP etc) > * Authentication (Admin UI, Admin and data/query operations. Tons of auth > protocols: basic, digest, oauth, pki..) > * Authorization (who can do what with what API, collection, doc) > * Pluggability (no user's needs are equal) > * And we could go on and on but this is what we've seen the most demand for -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383567#comment-14383567 ] Per Steffensen commented on SOLR-6816: -- bq. if a task fails, then Hadoop usually re-tries that task a couple of times, meaning all docs in the block that failed will be sent again We do not send all documents again if just a few in a batch (bulk) fails. Lets say you send a batch of 1000 docs for indexing and only 2 fails due to e.g. version-control, we only do another round on those 2 documents - SOLR-3382 > Review SolrCloud Indexing Performance. > -- > > Key: SOLR-6816 > URL: https://issues.apache.org/jira/browse/SOLR-6816 > Project: Solr > Issue Type: Task > Components: SolrCloud >Reporter: Mark Miller >Priority: Critical > Attachments: SolrBench.pdf > > > We have never really focused on indexing performance, just correctness and > low hanging fruit. We need to vet the performance and try to address any > holes. > Note: A common report is that adding any replication is very slow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383556#comment-14383556 ] Per Steffensen commented on SOLR-6816: -- Just to clarify. We did our own implementation of bloom-filter, and did not build on the existing feature, because we did not find it satisfactory. I do not remember exactly why, but maybe it was because the current solution builds a bloom-filter on each segment, and bloom-filters are not really mergeable unless they have the same size and used the same hashing-algorithm (and number of hashes performed). To be efficient you want your bloom-filter to be bigger (and potentially do another number of hashes) the bigger your segment is. We decided to do a one-bloom-filter-per-index/core/replica (or whatever you like to call it) solution, so that you do not need to merge bloom-filters. Besides that you can tell Solr about the maximum expected amount of documents in the index and the FPP you want at that number of documents, and it will calculate the size of the bloom-filter to use. Because such a one-bloom-filter-per-index does not go very well hand-in-hand with Lucenes segmenting strategy, we implemented it as a Solr-feature, that is, it is Solr that maintains the bloom-filter and the feature is not available for people who just uses Lucene. It is not a simple task to do it right, so if the lower-hanging fruits are satisfactory, you should pick them first. To recap, we have recently-updated-or-lookedup cache to be able to efficiently find the version of a document that does actually exist (and was "touched" recently), and bloom-filter to be able to efficiently find out if a document does not exist. > Review SolrCloud Indexing Performance. > -- > > Key: SOLR-6816 > URL: https://issues.apache.org/jira/browse/SOLR-6816 > Project: Solr > Issue Type: Task > Components: SolrCloud >Reporter: Mark Miller >Priority: Critical > Attachments: SolrBench.pdf > > > We have never really focused on indexing performance, just correctness and > low hanging fruit. We need to vet the performance and try to address any > holes. > Note: A common report is that adding any replication is very slow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343271#comment-14343271
]
Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:45 PM:
--
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too lazy doing the
download/upload using ZkCLI yourself), instead of the other way around, where
ZkCLI uses other tools or even just does json-manipulation itself. Please do
not try to implement rules about what can and cannot be in a specific type of
json, what operations you can do on it etc two places in the code-base - use
what we already have.
Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being
consistent, valid etc. You never do changes to clusterstate.json with using
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json.
You never manipulate a clusterstate.json without going through
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so
that it support receiving jobs not coming from overseer queue.
Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handles cluster-props.
That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in above?
was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader, ClusterState
etc. Then it is always those classes that are used to operate on a specific
type of json, and we only have to build consistency, integrity, validity etc
into those classes (separation of concern). The new thing should be that those
classes can be used via an external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343271#comment-14343271
]
Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:48 PM:
--
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too lazy doing the
download/upload using ZkCLI yourself), instead of the other way around, where
ZkCLI uses other tools or even just does json-manipulation itself. Please do
not try to implement rules about what can and cannot be in a specific type of
json, what operations you can do on it etc two places in the code-base - use
what we already have.
Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being
consistent, valid etc. You never do changes to clusterstate.json with using
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json.
You never manipulate a clusterstate.json without going through
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so
that it support receiving jobs not coming from other sources than overseer
queue.
Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handle cluster-props.
That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in above?
was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is,
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343271#comment-14343271
]
Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:45 PM:
--
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too lazy doing the
download/upload using ZkCLI yourself), instead of the other way around, where
ZkCLI uses other tools or even just does json-manipulation itself. Please do
not try to implement rules about what can and cannot be in a specific type of
json, what operations you can do on it etc two places in the code-base - use
what we already have.
Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being
consistent, valid etc. You never do changes to clusterstate.json with using
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json.
You never manipulate a clusterstate.json without going through
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so
that it support receiving jobs not coming from overseer queue.
Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handles cluster-props.
That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in above?
was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that u
[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343271#comment-14343271
]
Per Steffensen commented on SOLR-7176:
--
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader, ClusterState
etc. Then it is always those classes that are used to operate on a specific
type of json, and we only have to build consistency, integrity, validity etc
into those classes (separation of concern). The new thing should be that those
classes can be used via an external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too lazy doing the
download/upload using ZkCLI yourself), instead of the other way around, where
ZkCLI uses other tools or even just does json-manipulation itself. Please do
not try to implement rules about what can and cannot be in a specific type of
json, what operations you can do on it etc two places in the code-base - use
what we already have.
That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in above?
> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
> Issue Type: New Feature
>Reporter: Yonik Seeley
>Priority: Minor
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in
> clusterprops. We should be able to change individual values or perhaps merge
> values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6625) HttpClient callback in HttpSolrServer
[ https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289383#comment-14289383 ] Per Steffensen edited comment on SOLR-6625 at 1/23/15 3:29 PM: --- New patch * Support for multiple callbacks (for nicer separation of concerns) - both for sys-prop-callbacks and for solr.xml-callbacks. preExecuteRequest lets callbacks build on contexts from each other - a logical consequence of introducing support for multiple callbacks * Solr-nodes now use both sol.xml-callbacks AND sys-prop-callbacks * SolrCLI.getJson attempts-variant now actually uses the callbacks given to it (via sys-prop) * Cleaning up some of the code-duplication. Separating more of the code related to this issue in HttpClientRequestCallbackUtil was (Author: steff1193): New patch * Support for multiple callbacks (for nicer separation of concerns) - both for sys-prop-callbacks and for solr.xml-callbacks. preExecuteRequest lets callbacks build on contexts from each other - a logical consequence of introducing support for multiple callbacks * Sold-nodes now use both sol.xml-callbacks AND sys-prop-callbacks * SolrCLI.getJson attempts-variant now actually uses the callbacks given to it (via system param) * Cleaning up some of the code-duplication. Separating more of the code related to this issue in HttpClientRequestCallbackUtil > HttpClient callback in HttpSolrServer > - > > Key: SOLR-6625 > URL: https://issues.apache.org/jira/browse/SOLR-6625 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Attachments: SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, > SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, SOLR-6625_r1654079.patch, > SOLR-6625_r1654079.patch > > > Some of our setups use Solr in a SPNego/kerberos setup (we've done this by > adding our own filters to the web.xml). We have an issue in that SPNego > requires a negotiation step, but some HttpSolrServer requests are not > repeatable, notably the PUT/POST requests. So, what happens is, > HttpSolrServer sends the requests, the server responds with a negotiation > request, and the request fails because the request is not repeatable. We've > modified our code to send a repeatable request beforehand in these cases. > It would be nicer if HttpSolrServer provided a pre/post callback when it was > making an httpclient request. This would allow administrators to make > changes to the request for authentication purposes, and would allow users to > make per-request changes to the httpclient calls (i.e. modify httpclient > requestconfig to modify the timeout on a per-request basis). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6625) HttpClient callback in HttpSolrServer
[ https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-6625: - Attachment: SOLR-6625_r1654079.patch New patch * Support for multiple callbacks (for nicer separation of concerns) - both for sys-prop-callbacks and for solr.xml-callbacks. preExecuteRequest lets callbacks build on contexts from each other - a logical consequence of introducing support for multiple callbacks * Sold-nodes now use both sol.xml-callbacks AND sys-prop-callbacks * SolrCLI.getJson attempts-variant now actually uses the callbacks given to it (via system param) * Cleaning up some of the code-duplication. Separating more of the code related to this issue in HttpClientRequestCallbackUtil > HttpClient callback in HttpSolrServer > - > > Key: SOLR-6625 > URL: https://issues.apache.org/jira/browse/SOLR-6625 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Attachments: SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, > SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, SOLR-6625_r1654079.patch, > SOLR-6625_r1654079.patch > > > Some of our setups use Solr in a SPNego/kerberos setup (we've done this by > adding our own filters to the web.xml). We have an issue in that SPNego > requires a negotiation step, but some HttpSolrServer requests are not > repeatable, notably the PUT/POST requests. So, what happens is, > HttpSolrServer sends the requests, the server responds with a negotiation > request, and the request fails because the request is not repeatable. We've > modified our code to send a repeatable request beforehand in these cases. > It would be nicer if HttpSolrServer provided a pre/post callback when it was > making an httpclient request. This would allow administrators to make > changes to the request for authentication purposes, and would allow users to > make per-request changes to the httpclient calls (i.e. modify httpclient > requestconfig to modify the timeout on a per-request basis). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6625) HttpClient callback in HttpSolrServer
[ https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289100#comment-14289100 ] Per Steffensen edited comment on SOLR-6625 at 1/23/15 11:38 AM: Updated patch to HEAD of branch_x5. Not a completely trivial task. Did not try to make any of the changes I suggested in previous comments. Just tried to make a 1-to-1 upgrade to HEAD of branch_5x. Please someone verify that I did not miss anything. was (Author: steff1193): Updated patch to HEAD of branch_x5. Not a completely trivial task. > HttpClient callback in HttpSolrServer > - > > Key: SOLR-6625 > URL: https://issues.apache.org/jira/browse/SOLR-6625 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Attachments: SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, > SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, SOLR-6625_r1654079.patch > > > Some of our setups use Solr in a SPNego/kerberos setup (we've done this by > adding our own filters to the web.xml). We have an issue in that SPNego > requires a negotiation step, but some HttpSolrServer requests are not > repeatable, notably the PUT/POST requests. So, what happens is, > HttpSolrServer sends the requests, the server responds with a negotiation > request, and the request fails because the request is not repeatable. We've > modified our code to send a repeatable request beforehand in these cases. > It would be nicer if HttpSolrServer provided a pre/post callback when it was > making an httpclient request. This would allow administrators to make > changes to the request for authentication purposes, and would allow users to > make per-request changes to the httpclient calls (i.e. modify httpclient > requestconfig to modify the timeout on a per-request basis). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6625) HttpClient callback in HttpSolrServer
[ https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-6625: - Attachment: SOLR-6625_r1654079.patch Updated patch to HEAD of branch_x5. Not a completely trivial task. > HttpClient callback in HttpSolrServer > - > > Key: SOLR-6625 > URL: https://issues.apache.org/jira/browse/SOLR-6625 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Attachments: SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, > SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, SOLR-6625_r1654079.patch > > > Some of our setups use Solr in a SPNego/kerberos setup (we've done this by > adding our own filters to the web.xml). We have an issue in that SPNego > requires a negotiation step, but some HttpSolrServer requests are not > repeatable, notably the PUT/POST requests. So, what happens is, > HttpSolrServer sends the requests, the server responds with a negotiation > request, and the request fails because the request is not repeatable. We've > modified our code to send a repeatable request beforehand in these cases. > It would be nicer if HttpSolrServer provided a pre/post callback when it was > making an httpclient request. This would allow administrators to make > changes to the request for authentication purposes, and would allow users to > make per-request changes to the httpclient calls (i.e. modify httpclient > requestconfig to modify the timeout on a per-request basis). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264463#comment-14264463 ] Per Steffensen edited comment on SOLR-4470 at 1/5/15 10:29 AM: --- There is a variant of 2nd suggestion, that I just want to mention briefly (I am not good at being brief) - something that we do of other reasons. You might want to divide your set of Solr-nodes into two sub-sets * Search-Solrs: Receiving incoming search requests from outside clients. Not containing any data themselves but orchestrating the distributed searches against the Data-Solrs * Data-Solrs: Not directly accessible (network-wise) to outside clients Setting up web-container security on Search-Solrs only might work, because they do not have to communicate with each other. And it might be enough for you if Data-Solrs are not accessible from outside. We have such a setup but not for security reasons - we have web-container security on both Search-Solrs and Data-Solrs. Just wanted to mention it because it also might make things easier for you, and since I know it is possible this is also an option for you. We do the separation because of different hardware requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but almost no disk-space, Data-Solrs need more disk-space but less memory - in our setup, with the amounts and distribution of data we have and for the searches we perform against the Sor-cluster. Technically what we do is that we make a "search-collection" containing no data. "search-collection" only has shards on the Search-Solrs. Then we have several "data-collections" (containing the data) only having shards on the Data-Solrs. We have custom (not much code) SearchHandler/SearchComponents (Solr concepts) that basically just calculates the "data-collection" you have to search and forwards the search from outside clients to "search-collection" to those "data-collections". We have an advanced algorithm for calculating the "data-collections" that has to be searches given the concrete search to "search-collection", but you can just always calculate "all data-collections". This way Search-Solrs handle the distributed search, while Data-Solrs handle searches against local shard only. This is kinda the same as putting your own custom gateway in front of your Solr-nodes, and make the Solrs-nodes inaccessible directly from outside clients. We just use Solr itself as this front gateway, taking advantage of all the nice features - high availability (with more than one Search-Solr) etc. Clients uses CloudSolrServer, but always make searches against the "search-collection". was (Author: steff1193): There is a variant of 2nd suggestion, that I just want to mention briefly (I am not good at being brief) - something that we do of other reasons. You might want to divide your set of Solr-nodes into two sub-sets * Search-Solrs: Receiving incoming search requests from outside clients. Not containing any data themselves but orchestrating the distributed searches against the Data-Solrs * Data-Solrs: Not directly accessible (network-wise) to outside clients Setting up web-container security on Search-Solrs only might work, because they do not have to communicate with each other. And it might be enough for you if Data-Solrs are not accessible from outside. We have such a setup but not for security reasons - we have web-container security on both Search-Solrs and Data-Solrs. Just wanted to mention it because it also might make things easier for you, and since I know it is possible this is also an option for you. We do the separation because of different hardware requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but almost no disk-space, Data-Solrs need more disk-space but less memory - in our setup, with the amounts and distribution of data we have and for the searches we perform against the Sor-cluster. Technically what we do is that we make a "search-collection" containing no data. "search-collection" only has shards on the Search-Solrs. Then we have several "data-collections" (containing the data) only having shards on the Data-Solrs. We have custom (not much code) SearchHandler/SearchComponents (Solr concepts) that basically just calculates the "data-collection" you have to search and forwards the search from outside clients to "search-collection" to those "data-collections". We have an advanced algorithm for calculating the "data-collections" that has to be searches given the concrete search to "search-collection", but you can just always calculate "all data-collections". This way Search-Solrs handle the distributed search, w
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264463#comment-14264463 ] Per Steffensen commented on SOLR-4470: -- There is a variant of 2nd suggestion, that I just want to mention briefly (I am not at being brief) - something that we do of other reasons. You might want to divide your set of Solr-nodes into two sub-sets * Search-Solrs: Receiving incoming search requests from outside clients. Not containing any data themselves but orchestrating the distributed searches against the Data-Solrs * Data-Solrs: Not directly accessible (network-wise) to outside clients Setting up web-container security on Search-Solrs only might work, because they do not have to communicate with each other. And it might be enough for you if Data-Solrs are not accessible from outside. We have such a setup but not for security reasons - we have web-container security on both Search-Solrs and Data-Solrs. Just wanted to mention it because it also might make things easier for you, and since I know it is possible this is also an option for you. We do the separation because of different hardware requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but almost no disk-space, Data-Solrs need more disk-space but less memory - in our setup, with the amounts and distribution of data we have and for the searches we perform against the Sor-cluster. Technically what we do is that we make a "search-collection" containing no data. "search-collection" only has shards on the Search-Solrs. Then we have several "data-collections" (containing the data) only having shards on the Data-Solrs. We have custom (not much code) SearchHandler/SearchComponents (Solr concepts) that basically just calculates the "data-collection" you have to search and forwards the search from outside clients to "search-collection" to those "data-collections". We have an advanced algorithm for calculating the "data-collections" that has to be searches given the concrete search to "search-collection", but you can just always calculate "all data-collections". This way Search-Solrs handle the distributed search, while Data-Solrs handle searches against local shard only. This is kinda the same as putting your own custom gateway in front of your Solr-nodes, and make the Solrs-nodes inaccessible directly from outside clients. We just use Solr itself as this front gateway, taking advantage of all the nice features - high availability (with more than one Search-Solr) etc. Clients uses CloudSolrServer, but always make searches against the "search-collection". > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 >Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no > matter what kind of HTTP-request you make to a Solr-node. > It can faily easy be acheived as described on > http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes > also make "internal" request to other Solr-nodes, and for it to work > credentials need to be provided here also. > Ideally we would like to "forward" credentials from a particular request to > all the "internal" sub-requests it triggers. E.g. for search and update > request. > But there are also "internal" requests > * that only indirectly/asynchronously triggered from "outside" requests (e.g. > shard creation/deletion/etc based on calls to the "Collection API") > * that do not in any way have relation to an "outside" "super"-request (e.g. > replica synching stuff) > We would like to aim at a solution where "original" credentials are > "forwarded" when a request directly/synchronously trigger a subrequest, and > fallback to a
[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264463#comment-14264463 ] Per Steffensen edited comment on SOLR-4470 at 1/5/15 10:28 AM: --- There is a variant of 2nd suggestion, that I just want to mention briefly (I am not good at being brief) - something that we do of other reasons. You might want to divide your set of Solr-nodes into two sub-sets * Search-Solrs: Receiving incoming search requests from outside clients. Not containing any data themselves but orchestrating the distributed searches against the Data-Solrs * Data-Solrs: Not directly accessible (network-wise) to outside clients Setting up web-container security on Search-Solrs only might work, because they do not have to communicate with each other. And it might be enough for you if Data-Solrs are not accessible from outside. We have such a setup but not for security reasons - we have web-container security on both Search-Solrs and Data-Solrs. Just wanted to mention it because it also might make things easier for you, and since I know it is possible this is also an option for you. We do the separation because of different hardware requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but almost no disk-space, Data-Solrs need more disk-space but less memory - in our setup, with the amounts and distribution of data we have and for the searches we perform against the Sor-cluster. Technically what we do is that we make a "search-collection" containing no data. "search-collection" only has shards on the Search-Solrs. Then we have several "data-collections" (containing the data) only having shards on the Data-Solrs. We have custom (not much code) SearchHandler/SearchComponents (Solr concepts) that basically just calculates the "data-collection" you have to search and forwards the search from outside clients to "search-collection" to those "data-collections". We have an advanced algorithm for calculating the "data-collections" that has to be searches given the concrete search to "search-collection", but you can just always calculate "all data-collections". This way Search-Solrs handle the distributed search, while Data-Solrs handle searches against local shard only. This is kinda the same as putting your own custom gateway in front of your Solr-nodes, and make the Solrs-nodes inaccessible directly from outside clients. We just use Solr itself as this front gateway, taking advantage of all the nice features - high availability (with more than one Search-Solr) etc. Clients uses CloudSolrServer, but always make searches against the "search-collection". was (Author: steff1193): There is a variant of 2nd suggestion, that I just want to mention briefly (I am not at being brief) - something that we do of other reasons. You might want to divide your set of Solr-nodes into two sub-sets * Search-Solrs: Receiving incoming search requests from outside clients. Not containing any data themselves but orchestrating the distributed searches against the Data-Solrs * Data-Solrs: Not directly accessible (network-wise) to outside clients Setting up web-container security on Search-Solrs only might work, because they do not have to communicate with each other. And it might be enough for you if Data-Solrs are not accessible from outside. We have such a setup but not for security reasons - we have web-container security on both Search-Solrs and Data-Solrs. Just wanted to mention it because it also might make things easier for you, and since I know it is possible this is also an option for you. We do the separation because of different hardware requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but almost no disk-space, Data-Solrs need more disk-space but less memory - in our setup, with the amounts and distribution of data we have and for the searches we perform against the Sor-cluster. Technically what we do is that we make a "search-collection" containing no data. "search-collection" only has shards on the Search-Solrs. Then we have several "data-collections" (containing the data) only having shards on the Data-Solrs. We have custom (not much code) SearchHandler/SearchComponents (Solr concepts) that basically just calculates the "data-collection" you have to search and forwards the search from outside clients to "search-collection" to those "data-collections". We have an advanced algorithm for calculating the "data-collections" that has to be searches given the concrete search to "search-collection", but you can just always calculate "all data-collections". This way Search-Solrs handle the distributed search, while
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264372#comment-14264372 ] Per Steffensen commented on SOLR-4470: -- As you noticed. You can configure standard web-container protection in web.xml/jetty.xml, which makes Jetty require username/password get access. This means that the requests going in to Solr-nodes (servers) have username/password added. As long as you send requests from an outside SolrJ client (where you control the code) you can add the credentials, and you will get access. But Solr-nodes themselves send requests to other Solr-nodes, and before SOLR-4470 you have no (easy) way to add credentials to those requests - hence they will not succeed. I guess what happens in you case it that recovery needs to send inter-solr-node requests that fail due to no-access. To get a nice and easy way out of this vote for SOLR-4470 (or some other JIRA issue that will make it possible to add credentials to inter-solr-node requests - e.g. SOLR-6625 will add a more general feature (callback) that will enable you to get around it). It might not be entirely impossible for you to get around the problem. Guess there several more or less hacky options. E.g. * Some 3rd-party tool that intercepts requests between solr-nodes and add credentials * You probably have a well known set of IPs on which the solr-nodes run. You cannot do it in standard web-container, but Jetty might support configuring a set of IPs that do not have to authenticate to get access (if that is acceptable to you). Or maybe you can implement your own bean and plug it into Jetty. * Guess you can discard using standard web-container security, and implement it all yourself. E.g. in a filter that you add in front of SolrDispatchFilter But I believe it is not easy before SOLR-4470. Back when I wrote the code for SOLR-4470, I also modified https://wiki.apache.org/solr/SolrSecurity to show how things will work after SOLR-4470. Carefully (trying) to note that this depends on SOLR-4470 being included in the product. > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 > Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no > matter what kind of HTTP-request you make to a Solr-node. > It can faily easy be acheived as described on > http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes > also make "internal" request to other Solr-nodes, and for it to work > credentials need to be provided here also. > Ideally we would like to "forward" credentials from a particular request to > all the "internal" sub-requests it triggers. E.g. for search and update > request. > But there are also "internal" requests > * that only indirectly/asynchronously triggered from "outside" requests (e.g. > shard creation/deletion/etc based on calls to the "Collection API") > * that do not in any way have relation to an "outside" "super"-request (e.g. > replica synching stuff) > We would like to aim at a solution where "original" credentials are > "forwarded" when a request directly/synchronously trigger a subrequest, and > fallback to a configured "internal credentials" for the > asynchronous/non-rooted requests. > In our solution we would aim at only supporting basic http auth, but we would > like to make a "framework" around it, so that not to much refactoring is > needed if you later want to make support for other kinds of auth (e.g. digest) > We will work at a solution but create this JIRA issue early in order to get > input/comments from the community as early as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260880#comment-14260880
]
Per Steffensen commented on SOLR-6810:
--
{quote}
I think the strategy that Shalin & I were talking about as a potential default
was one that never collected IDs separately, hence no extra round-trip.
step 1: retrieve sort field values (then merge and calculate the range of
ordinals needed for each shard)
step 2: retrieve stored fields by specifying the ordinals from each shard
{quote}
It sounds like third DQA - does not seem to be exactly what my new algorithm
does. But you suggestion still has 2 round-trips. The old/current default-DQA
with dqa.forceSkipGetIds/distrib.singlePass=false (default) has 2 round-trips.
My new DQA with dqa.forceSkipGetIds=true (default) has 2 round-trips, so
choosing that as the new default-DQA will not introduce an extra round-trip
compared to todays default-DQA. But we could choose to select old/current
default-DQA with dqa.forceSkipGetIds/distrib.singlePass=true as the new
default-DQA. It has only 1 round-trip, so compared to that both your DQA
(above) and my new DQA has an extra round-trip. My new DQA will never become
1-round-trip only, because the essence is to make a round-trip first
(inexpensive because you do not access store - not even for ids), collecting
information needed to limit "rows" for the second round-trip where you actually
retrieve the stored fields.
bq. one that never collected IDs separately
My new DQA does not collect IDs separately (as long as you do not explicitly
set dqa.forceSkipGetIds=false)
> Faster searching limited but high rows across many shards all with many hits
>
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
> Issue Type: Improvement
> Components: search
>Reporter: Per Steffensen
>Assignee: Shalin Shekhar Mangar
> Labels: distributed_search, performance
> Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch,
> branch_5x_rev1645549.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is
> slow
> E.g.
> * Query from outside client: q=something&rows=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something&rows=1000&fl=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards have a significant
> number of hits on the query
> The problem grows on all three factors above
> Doing such a query on our system takes between 5 min to 1 hour - depending on
> a lot of things. It ought to be much faster, so lets make it.
> Profiling show that the problem is that it takes lots of time to access the
> store to get id’s for (up to) 1000 docs (value of rows parameter) per shard.
> Having 1000 shards its up to 1 mio ids that has to be fetched. There is
> really no good reason to ever read information from store for more than the
> overall top-1000 documents, that has to be returned to the client.
> For further detail see mail-thread "Slow searching limited but high rows
> across many shards all with high hits" started 13/11-2014 on
> dev@lucene.apache.org
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259447#comment-14259447
]
Per Steffensen edited comment on SOLR-6810 at 12/27/14 6:35 PM:
TestDistributedQueryAlgorithm.testDocReads shows very well exactly how the
number of store accesses is reduced
{code}
// Test the number of documents read from store using
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// vs FIND_ID_RELEVANCE_FETCH_BY_IDS. This demonstrates the advantage of
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// over FIND_ID_RELEVANCE_FETCH_BY_IDS (and vice versa)
private void testDocReads() throws Exception {
for (int startValue = 0; startValue <= MAX_START; startValue++) {
// FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS (assuming skipGetIds
used - default)
// Only reads data (required fields) from store for "rows + (#shards *
start)" documents across all shards
// This can be optimized to become only "rows"
// Only reads the data once
testDQADocReads(ShardParams.DQA.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS,
startValue, ROWS, ROWS + (startValue * jettys.size()), ROWS + (startValue *
jettys.size()));
// DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS (assuming skipGetIds not used -
default)
// Reads data (ids only) from store for "(rows + startValue) * #shards"
documents for each shard
// Besides that reads data (required fields) for "rows" documents across
all shards
testDQADocReads(ShardParams.DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS, startValue,
ROWS, (ROWS + startValue) * jettys.size(), ROWS + ((ROWS + startValue) *
jettys.size()));
}
}
{code}
{code}
testDQADocReads(ShardParams.DQA dqa, int start, int rows, int
expectedUniqueIdCount, int expectedTotalCount) {
...
}
{code}
was (Author: steff1193):
TestDistributedQueryAlgorithm.testDocReads shows very well exactly how the
number of store accesses is reduced
{code}
// Test the number of documents read from store using
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// vs FIND_ID_RELEVANCE_FETCH_BY_IDS. This demonstrates the advantage of
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// over FIND_ID_RELEVANCE_FETCH_BY_IDS (and vice versa)
private void testDocReads() throws Exception {
for (int startValue = 0; startValue <= MAX_START; startValue++) {
// FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS (assuming skipGetIds
used - default)
// Only reads data (required fields) from store for "rows + (#shards *
start)" documents across all shards
// This can be optimized to become only "rows"
// Only reads the data once
testDQADocReads(ShardParams.DQA.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS,
startValue, ROWS, ROWS + (startValue * jettys.size()), ROWS + (startValue *
jettys.size()));
// DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS (assuming skipGetIds not used -
default)
// Reads data (ids only) from store for "(rows + startValue) * #shards"
documents for each shard
// Besides that reads data (required fields) for "rows" documents across
all shards
testDQADocReads(ShardParams.DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS, startValue,
ROWS, (ROWS + startValue) * jettys.size(), ROWS + ((ROWS + startValue) *
jettys.size()));
}
}
{code}
> Faster searching limited but high rows across many shards all with many hits
>
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
> Issue Type: Improvement
> Components: search
>Reporter: Per Steffensen
>Assignee: Shalin Shekhar Mangar
> Labels: distributed_search, performance
> Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch,
> branch_5x_rev1645549.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is
> slow
> E.g.
> * Query from outside client: q=something&rows=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something&rows=1000&fl=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards have a significant
> number of hits on the query
> The problem grows on all three factors above
> Doing such a query on our system takes between 5 min to 1 hour - depending on
> a lot of things. It ought to be much faster, so l
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259447#comment-14259447
]
Per Steffensen commented on SOLR-6810:
--
TestDistributedQueryAlgorithm.testDocReads shows very well exactly how the
number of store accesses is reduced
{code}
// Test the number of documents read from store using
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// vs FIND_ID_RELEVANCE_FETCH_BY_IDS. This demonstrates the advantage of
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// over FIND_ID_RELEVANCE_FETCH_BY_IDS (and vice versa)
private void testDocReads() throws Exception {
for (int startValue = 0; startValue <= MAX_START; startValue++) {
// FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS (assuming skipGetIds
used - default)
// Only reads data (required fields) from store for "rows + (#shards *
start)" documents across all shards
// This can be optimized to become only "rows"
// Only reads the data once
testDQADocReads(ShardParams.DQA.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS,
startValue, ROWS, ROWS + (startValue * jettys.size()), ROWS + (startValue *
jettys.size()));
// DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS (assuming skipGetIds not used -
default)
// Reads data (ids only) from store for "(rows + startValue) * #shards"
documents for each shard
// Besides that reads data (required fields) for "rows" documents across
all shards
testDQADocReads(ShardParams.DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS, startValue,
ROWS, (ROWS + startValue) * jettys.size(), ROWS + ((ROWS + startValue) *
jettys.size()));
}
}
{code}
> Faster searching limited but high rows across many shards all with many hits
>
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
> Issue Type: Improvement
> Components: search
>Reporter: Per Steffensen
>Assignee: Shalin Shekhar Mangar
> Labels: distributed_search, performance
> Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch,
> branch_5x_rev1645549.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is
> slow
> E.g.
> * Query from outside client: q=something&rows=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something&rows=1000&fl=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards have a significant
> number of hits on the query
> The problem grows on all three factors above
> Doing such a query on our system takes between 5 min to 1 hour - depending on
> a lot of things. It ought to be much faster, so lets make it.
> Profiling show that the problem is that it takes lots of time to access the
> store to get id’s for (up to) 1000 docs (value of rows parameter) per shard.
> Having 1000 shards its up to 1 mio ids that has to be fetched. There is
> really no good reason to ever read information from store for more than the
> overall top-1000 documents, that has to be returned to the client.
> For further detail see mail-thread "Slow searching limited but high rows
> across many shards all with high hits" started 13/11-2014 on
> dev@lucene.apache.org
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259445#comment-14259445
]
Per Steffensen commented on SOLR-6810:
--
bq. IMO, one shouldn't have to look at the patch to figure out what it's
trying to do.
Seems reasonable. The way things change is IMHO fairly well documented in
JavaDocs of ShardParams.DQA so I will just steal from there
* Old DQA (FIND_ID_RELEVANCE_FETCH_BY_IDS)
{code}
/**
* Algorithm
* - Shard-queries 1) Ask, by forwarding the outer query, each shard for id
and relevance of the (up to) #rows most relevant matching documents
* - Find among those id/relevances the #rows id's with the highest global
relevances (lets call this set of id's X)
* - Shard-queries 2) Ask, by sending id's, each shard to return the
documents from set X that it holds
* - Return the fetched documents to the client
*/
...
// Default do not force skip get-ids phase
{code}
* New DQA (FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS)
{code}
/**
* Algorithm
* - Shard-queries 1) Ask, by forwarding the outer query, each shard for
relevance of the (up to) #rows most relevant matching documents
* - Find among those relevances the #rows highest global relevances
* Note for each shard (S) how many entries (docs_among_most_relevant(S)) it
has among the #rows globally highest relevances
* - Shard-queries 2) Ask, by forwarding the outer query, each shard S for
id and relevance of the (up to) #docs_among_most_relevant(S) most relevant
matching documents
* - Find among those id/relevances the #rows id's with the highest global
relevances (lets call this set of id's X)
* - Shard-queries 3) Ask, by sending id's, each shard to return the
documents from set X that it holds
* - Return the fetched documents to the client
*
* Advantages
* Asking for data from store (id in shard-queries 1) of
FIND_ID_RELEVANCE_FETCH_BY_IDS) can be expensive, therefore sometimes you want
to ask for data
* from as few documents as possible.
* The main purpose of this algorithm it to limit the rows asked for in
shard-queries 2) compared to shard-queries 1) of FIND_ID_RELEVANCE_FETCH_BY_IDS.
* Lets call the number of rows asked for by the outer request for
"outer-rows"
* shard-queries 2) will never ask for data from more than "outer-rows"
documents total across all involved shards. shard-queries 1) of
FIND_ID_RELEVANCE_FETCH_BY_IDS
* will ask each shard for data from "outer-rows" documents, and in worst
case if each shard contains "outer-rows" matching documents you will
* fetch data for "number of shards involved" * "outer-rows".
* Using FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS will become more
beneficial the more
* - shards are involved
* - and/or the more matching documents each shard holds
*/
...
// Default force skip get-ids phase. In this algorithm there are really
never any reason not to skip it
{code}
* dqa.forceSkipGetIds
{code}
/** Request parameter to force skip get-ids phase of the distributed query?
Value: true or false
* Even if you do not force it, the system might choose to do it anyway
* Skipping the get-ids phase
* - FIND_ID_RELEVANCE_FETCH_BY_IDS: Fetch entire documents in Shard-queries
1) and skip Shard-queries 2)
* - FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS: Fetch entire
documents in Shard-queries 2) and skip Shard-queries 3)
*/
{code}
> Faster searching limited but high rows across many shards all with many hits
>
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
> Issue Type: Improvement
> Components: search
>Reporter: Per Steffensen
>Assignee: Shalin Shekhar Mangar
> Labels: distributed_search, performance
> Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch,
> branch_5x_rev1645549.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is
> slow
> E.g.
> * Query from outside client: q=something&rows=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something&rows=1000&fl=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards hav
[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259438#comment-14259438
]
Per Steffensen edited comment on SOLR-6810 at 12/27/14 6:09 PM:
bq. Since dqa.forceSkipGetIds is always true for this new algorithm then
computing the set X is not necessary and we can just directly fetch all return
fields from individual shards and return the response to the user. Is that
correct?
This is what happens by default with the new algorithm. But dqa.forceSkipGetIds
is not always true. It is true by default, but you can explicitly set it to
false by sending dqa.forceSkipGetIds=false in your request. So basically there
are four options
* old alg without dqa.forceSkipGetIds or with dqa.forceSkipGetIds=false
(default before SOLR-6810, and currently also after SOLR-6810)
* old alg with dqa.forceSkipGetIds=true (same as with distrib.singlePass=true
before SOLR-6810)
* new alg without dqa.forceSkipGetIds or with dqa.forceSkipGetIds=true (does as
you describe above)
* new alg with dqa.forceSkipGetIds=false (does as described in the JavaDoc you
quoted)
The JavaDoc descriptions describe how the alg works WITHOUT dqa.forceSkipGetIds
switched on. But dqa.forceSkipGetIds is switched on for the new alg by default.
The JavaDoc for ShardParams.DQA.FORCE_SKIP_GET_IDS_PARAM describes how the two
algs are altered when running with dqa.forceSkipGetIds=true. The thing is that
you need to know this part as well to understand how the new alg works by
default.
bq. I think the DefaultProvider and DefaultDefaultProvider aren't necessary? We
can just keep a single static ShardParams.getDQA(SolrParams params) method and
modify it if we ever need to change the default.
Well I would prefer to keep ShardParams.DQA.get(params) instead of having a
ShardParams.getDQA(params) - it think it is better "context'ing". But I will
survive if you want to change it.
DefaultProvider in supposed to isolate the default decisions.
DefaultDefaultProvider is an implementation that calculates the out-of-the-box
defaults. It could be done directly in ShardParams.DQA.get, but I like to
structure things. But I have to admit that the main reason I added the
DefaultProvider thing, was that it makes it easier to change the
default-decisions made when running the test-suite. I would like to randomly
select the DQA to be used for every single query fired across the entire
test-suite. This way we will have a very thorough test-coverage of both algs.
Having the option of changing the DefaultProvider made it very easy to achieve
this in SolrTestCaseJ4
{code}
private static DQA.DefaultProvider testDQADefaultProvider =
new DQA.DefaultProvider() {
@Override
public DQA getDefault(SolrParams params) {
// Select randomly the DQA to use
int algNo =
Math.abs(random().nextInt()%(ShardParams.DQA.values().length));
return DQA.values()[algNo];
}
};
{code}
{code}
DQA.setDefaultProvider(testDQADefaultProvider);
{code}
bq. If a user wants to change the default, the dqa can be set in the "defaults"
section of the search handler.
I know it is a matter of opinion but in my mind the best place to deal with
default for DQA is in the code that deals with DQA - not somewhere else. This
makes a much better isolation and it makes code easier to understand. You can
essentially navigate to ShardParams.DQA and read the code and JavaDoc and
understand everything about DQA's. You do not have to know that there is a
decision about default in the SeachHandler. But if you want to change that, it
is ok for me.
bq. Why do we need the switchToTestDQADefaultProvider() and
switchToOriginalDQADefaultProvider() methods? You are already applying the DQA
for each request so why is the switch necessary?
No I am not applying the DQA for each request. I trust you understand why I
want to run with randomized DQA across the entire test-suite - this is why I
invented the testDQADefaultProvider. In tests that explicitly deal with testing
DQA stuff, in some cases I want to switch on the real DefaultProvider because
some of those tests are actually testing out-of-the-box default-behaviour. E.g.
verifyForceSkipGetIds-tests in DistributedQueryComponentOptimizationTest. Also
need it in DistributedExpandComponentTest until SOLR-6813 has been solved.
bq. There's still the ShardParams.purpose field which you added in SOLR-6812
but I removed it. I still think it is unnecessary for purpose to be sent to
shard. Is it necessary for this patch or is it just an artifact from SOLR-6812?
You are right. It is a mistake that I did not remove ShardParams.purpose
bq. Did you benchmark it against the current algorithm for other kinds of
use-cases as well (3-5 shards, small number of rows)? Not asking for id can
speed up responses t
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259438#comment-14259438
]
Per Steffensen commented on SOLR-6810:
--
bq. Since dqa.forceSkipGetIds is always true for this new algorithm then
computing the set X is not necessary and we can just directly fetch all return
fields from individual shards and return the response to the user. Is that
correct?
This is what happens by default with the new algorithm. But dqa.forceSkipGetIds
is not always true. It is true by default, but you can explicitly set it to
false by sending dqa.forceSkipGetIds=false in your request. So basically there
are four options
* old alg without dqa.forceSkipGetIds or with dqa.forceSkipGetIds=false
(default before SOLR-6810, and currently also after SOLR-6810)
* old alg with dqa.forceSkipGetIds=true (same as with distrib.singlePass=true
before SOLR-6810)
* new alg without dqa.forceSkipGetIds or with dqa.forceSkipGetIds=true (does as
you describe above)
* new alg with dqa.forceSkipGetIds=false (does as described in the JavaDoc you
quoted)
The JavaDoc descriptions describe how the alg works WITHOUT dqa.forceSkipGetIds
switched on. But dqa.forceSkipGetIds is switched on for the new alg by default.
The JavaDoc for ShardParams.DQA.FORCE_SKIP_GET_IDS_PARAM describes how the two
algs are altered when running with dqa.forceSkipGetIds=true. The thing is that
you need to know this part as well to understand how the new alg works by
default.
bq. I think the DefaultProvider and DefaultDefaultProvider aren't necessary? We
can just keep a single static ShardParams.getDQA(SolrParams params) method and
modify it if we ever need to change the default.
Well I would prefer to keep ShardParams.DQA.get(params) instead of having a
ShardParams.getDQA(params) - it think it is better "context'ing". But I will
survive if you want to change it.
DefaultProvider in supposed to isolate the default decisions.
DefaultDefaultProvider is an implementation that calculates the out-of-the-box
defaults. It could be done directly in ShardParams.DQA.get, but I like to
structure things. But I have to admit that the main reason I added the
DefaultProvider thing, was that it makes it easier to change the
default-decisions made when running the test-suite. I would like to randomly
select the DQA to be used for every single query fired across the entire
test-suite. This way we will have a very thorough test-coverage of both algs.
Having the option of changing the DefaultProvider made it very easy to achieve
this in SolrTestCaseJ4
{code}
private static DQA.DefaultProvider testDQADefaultProvider =
new DQA.DefaultProvider() {
@Override
public DQA getDefault(SolrParams params) {
// Select randomly the DQA to use
int algNo =
Math.abs(random().nextInt()%(ShardParams.DQA.values().length));
return DQA.values()[algNo];
}
};
{code}
{code}
DQA.setDefaultProvider(testDQADefaultProvider);
{code}
bq. If a user wants to change the default, the dqa can be set in the "defaults"
section of the search handler.
I know it is a matter of opinion but in my mind the best place to deal with
default for DQA is in the code that deals with DQA - not somewhere else. This
makes a much better isolation and it makes code easier to understand. You can
essentially navigate to ShardParams.DQA and read the code and JavaDoc and
understand everything about DQA's. You do not have to know that there is a
decision about default in the SeachHandler. But if you want to change that, it
is ok for me.
bq. Why do we need the switchToTestDQADefaultProvider() and
switchToOriginalDQADefaultProvider() methods? You are already applying the DQA
for each request so why is the switch necessary?
No I am not applying the DQA for each request. I trust you understand why I
want to run with randomized DQA across the entire test-suite - this is why I
invented the testDQADefaultProvider. In tests that explicitly deal with testing
DQA stuff, in some cases I want to switch on the real DefaultProvider because
some of those tests are actually testing out-of-the-box default-behaviour. E.g.
verifyForceSkipGetIds-tests in DistributedQueryComponentOptimizationTest. Also
need it in DistributedExpandComponentTest until SOLR-6813 has been solved.
bq. There's still the ShardParams.purpose field which you added in SOLR-6812
but I removed it. I still think it is unnecessary for purpose to be sent to
shard. Is it necessary for this patch or is it just an artifact from SOLR-6812?
You are right. It is a mistake that I did not remove ShardParams.purpose
bq. Did you benchmark it against the current algorithm for other kinds of
use-cases as well (3-5 shards, small number of rows)? Not asking for id can
speed up responses there too I think.
I did not do any concrete benchmar
[jira] [Comment Edited] (SOLR-6816) Review SolrCloud Indexing Performance.
[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249673#comment-14249673
]
Per Steffensen edited comment on SOLR-6816 at 12/17/14 10:17 AM:
-
Those of you that have been following my comments on misc issues will know that
I like "separation of concerns". So in our version of Solr all this
decision-making on when to do document-version-check, when to delete existing
documents with same id-value etc is isolated in {{enum UpdateSemanticsMode}} -
see
https://issues.apache.org/jira/secure/attachment/12553312/SOLR-3173_3178_3382_3428_plus.patch.
We support different modes that makes slightly different decisions on the
above topics, which is the reason for using an enum. You do not need that,
because you only have "one mode", but that should not prevent you from
separating the decision-making concern.
The patch is not entirely up to date with what we do today, but at least it
illustrates the "separation of concerns". {{DistributedUpdateHandler}} deals
with a million concerns, so maybe you want to adopt that idea and move the code
making the decisions out of {{DistributedUpdateHandler}}.
Only mention this because I sense that at least [~shalinmangar] agrees that
some cleanup (a.o. of {{DistributedUpdateHandler}}) is required:
https://twitter.com/shalinmangar/status/543874893549277184
was (Author: steff1193):
Those of you that have been following my comments on misc issues will know that
I like "separation of concerns". So in our version of Solr all this
decision-making on when to do document-version-check, when to delete existing
documents with same id-value etc is isolated in {{enum UpdateSemanticsMode}} -
see
https://issues.apache.org/jira/secure/attachment/12553312/SOLR-3173_3178_3382_3428_plus.patch.
We support different modes that makes slightly different decisions on the
above topics, which is the reason for using an enum. You do not need that,
because you only have "one mode", but that should not prevent you from
separating the decision-making concern.
The patch is not entirely up to date with what we do today, but at least it
illustrates the "separation of concerns". {{DistributedUpdateHandler}} deals
with a million concerns, so maybe you want to adopt that idea and move the code
making the decisions out of {{DistributedUpdateHandler}}.
> Review SolrCloud Indexing Performance.
> --
>
> Key: SOLR-6816
> URL: https://issues.apache.org/jira/browse/SOLR-6816
> Project: Solr
> Issue Type: Task
> Components: SolrCloud
>Reporter: Mark Miller
>Priority: Critical
> Attachments: SolrBench.pdf
>
>
> We have never really focused on indexing performance, just correctness and
> low hanging fruit. We need to vet the performance and try to address any
> holes.
> Note: A common report is that adding any replication is very slow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249673#comment-14249673
]
Per Steffensen commented on SOLR-6816:
--
Those of you that have been following my comments on misc issues will know that
I like "separation of concerns". So in our version of Solr all this
decision-making on when to do document-version-check, when to delete existing
documents with same id-value etc is isolated in {{enum UpdateSemanticsMode}} -
see
https://issues.apache.org/jira/secure/attachment/12553312/SOLR-3173_3178_3382_3428_plus.patch.
We support different modes that makes slightly different decisions on the
above topics, which is the reason for using an enum. You do not need that,
because you only have "one mode", but that should not prevent you from
separating the decision-making concern.
The patch is not entirely up to date with what we do today, but at least it
illustrates the "separation of concerns". {{DistributedUpdateHandler}} deals
with a million concerns, so maybe you want to adopt that idea and move the code
making the decisions out of {{DistributedUpdateHandler}}.
> Review SolrCloud Indexing Performance.
> --
>
> Key: SOLR-6816
> URL: https://issues.apache.org/jira/browse/SOLR-6816
> Project: Solr
> Issue Type: Task
> Components: SolrCloud
>Reporter: Mark Miller
>Priority: Critical
> Attachments: SolrBench.pdf
>
>
> We have never really focused on indexing performance, just correctness and
> low hanging fruit. We need to vet the performance and try to address any
> holes.
> Note: A common report is that adding any replication is very slow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249639#comment-14249639
]
Per Steffensen commented on SOLR-6816:
--
I believe today overwrite=false will not prevent neither document-version-check
on leader (it will in the Solr we use in my company, but not in Apache Solr)
nor bucket-version-check on non-leaders. As far as I can see
{{DistributedUpdateProcessor.versionAdd}} will do document-version-check if
versionsStored=true, leaderLogic=true and versionOnUpdate != 0. It will do
bucket-version-check if versionsStored=true and leaderLogic=false. This has
nothing to do with overwrite param. This version-check is not only for
add-commands but also for delete-commands.
The overwrite param controls only (in {{DirectUpdateHandler2}}) if you make
sure to delete an existing document with the same id, before you add the new
document. You do that by default, but if overwrite=false you just add the new
document, allowing duplicates (defined to be documents that have the same
id-value).
So as far as I read the code, document-version-check will only be performed on
leaders. Non-leaders will only do bucket-version-check, and I do not think that
is expensive?
As I said our version of Solr does not do document-version-check if
overwrite=false. I think you should introduce that as well. But besides that,
whats left to do in this area?
What did I not understand?
> Review SolrCloud Indexing Performance.
> --
>
> Key: SOLR-6816
> URL: https://issues.apache.org/jira/browse/SOLR-6816
> Project: Solr
> Issue Type: Task
> Components: SolrCloud
>Reporter: Mark Miller
>Priority: Critical
> Attachments: SolrBench.pdf
>
>
> We have never really focused on indexing performance, just correctness and
> low hanging fruit. We need to vet the performance and try to address any
> holes.
> Note: A common report is that adding any replication is very slow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
[ https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248410#comment-14248410 ] Per Steffensen commented on SOLR-6813: -- bq. The distributed deep paging problem will continue to exist But isnt that a performance problem only. Functionality-wise it will work doing deep paging, right? If it is just a performance problem, basically "you should not use distrib.singlePass if you are deep paging", I guess we should not disallow using it together with deep paging. It is just a bad idea performance-wise to use distrib.singlePass together with deep paging. > distrib.singlePass does not work for expand-request - start/rows included > - > > Key: SOLR-6813 > URL: https://issues.apache.org/jira/browse/SOLR-6813 > Project: Solr > Issue Type: Bug > Components: multicore, search >Reporter: Per Steffensen >Assignee: Joel Bernstein > Labels: distributed_search, search > Attachments: test_that_reveals_the_problem.patch > > > Using distrib.singlePass does not work for expand-requests. Even after the > fix provided to SOLR-6812, it does not work for requests where you add start > and/or rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-6810: - Attachment: branch_5x_rev1645549.patch New patch where the my solution to SOLR-6812 has been removed. Patch now matches revision 1645549 where the final solution to SOLR-6812 is included > Faster searching limited but high rows across many shards all with many hits > > > Key: SOLR-6810 > URL: https://issues.apache.org/jira/browse/SOLR-6810 > Project: Solr > Issue Type: Improvement > Components: search > Reporter: Per Steffensen >Assignee: Shalin Shekhar Mangar > Labels: distributed_search, performance > Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, > branch_5x_rev1645549.patch > > > Searching "limited but high rows across many shards all with many hits" is > slow > E.g. > * Query from outside client: q=something&rows=1000 > * Resulting in sub-requests to each shard something a-la this > ** 1) q=something&rows=1000&fl=id,score > ** 2) Request the full documents with ids in the global-top-1000 found among > the top-1000 from each shard > What does the subject mean > * "limited but high rows" means 1000 in the example above > * "many shards" means 200-1000 in our case > * "all with many hits" means that each of the shards have a significant > number of hits on the query > The problem grows on all three factors above > Doing such a query on our system takes between 5 min to 1 hour - depending on > a lot of things. It ought to be much faster, so lets make it. > Profiling show that the problem is that it takes lots of time to access the > store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. > Having 1000 shards its up to 1 mio ids that has to be fetched. There is > really no good reason to ever read information from store for more than the > overall top-1000 documents, that has to be returned to the client. > For further detail see mail-thread "Slow searching limited but high rows > across many shards all with high hits" started 13/11-2014 on > dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
[
https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247963#comment-14247963
]
Per Steffensen commented on SOLR-6813:
--
I would prefer making it work instead of saying that single-pass in not
supported for expand-queries. But I guess that is easy for me to say :-)
If I understand you explanations correctly, expand-queries ought to (almost)
always fail for distributed. Also when rows/start is not set (default start=0
and rows=10). Can you explain why it does not fail in those cases
{code}
query("q", "*:*", "fq", "{!collapse field="+group+"}", "defType",
"edismax", "bf", "field(test_ti)", "expand", "true", "fl","*,score",
ShardParams.DISTRIB_SINGLE_PASS, "true");
query("q", "*:*", "fq", "{!collapse field="+group+"}", "defType",
"edismax", "bf", "field(test_ti)", "expand", "true", "expand.sort", "test_tl
desc", "fl","*,score", ShardParams.DISTRIB_SINGLE_PASS, "true");
query("q", "*:*", "fq", "{!collapse field="+group+"}", "defType",
"edismax", "bf", "field(test_ti)", "expand", "true", "expand.sort", "test_tl
desc", "expand.rows", "1", "fl","*,score", ShardParams.DISTRIB_SINGLE_PASS,
"true");
//Test no expand results
query("q", "test_ti:5", "fq", "{!collapse field="+group+"}", "defType",
"edismax", "bf", "field(test_ti)", "expand", "true", "expand.sort", "test_tl
desc", "expand.rows", "1", "fl","*,score", ShardParams.DISTRIB_SINGLE_PASS,
"true");
//Test zero results
query("q", "test_ti:5434343", "fq", "{!collapse field="+group+"}",
"defType", "edismax", "bf", "field(test_ti)", "expand", "true", "expand.sort",
"test_tl desc", "expand.rows", "1", "fl","*,score",
ShardParams.DISTRIB_SINGLE_PASS, "true");
//Test page 2
query("q", "*:*", "start","1", "rows", "1", "fq", "{!collapse
field="+group+"}", "defType", "edismax", "bf", "field(test_ti)", "expand",
"true", "fl","*,score", ShardParams.DISTRIB_SINGLE_PASS, "true");
{code}
Only the last one above fails?
> distrib.singlePass does not work for expand-request - start/rows included
> -
>
> Key: SOLR-6813
> URL: https://issues.apache.org/jira/browse/SOLR-6813
> Project: Solr
> Issue Type: Bug
> Components: multicore, search
>Reporter: Per Steffensen
>Assignee: Joel Bernstein
> Labels: distributed_search, search
> Attachments: test_that_reveals_the_problem.patch
>
>
> Using distrib.singlePass does not work for expand-requests. Even after the
> fix provided to SOLR-6812, it does not work for requests where you add start
> and/or rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14245266#comment-14245266 ] Per Steffensen commented on SOLR-6810: -- Thanks for taking care of SOLR-6795, SOLR-6796 and SOLR-6812 (and SOLR-6813), [~shalinmangar]!!! > Faster searching limited but high rows across many shards all with many hits > > > Key: SOLR-6810 > URL: https://issues.apache.org/jira/browse/SOLR-6810 > Project: Solr > Issue Type: Improvement > Components: search > Reporter: Per Steffensen >Assignee: Shalin Shekhar Mangar > Labels: distributed_search, performance > Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch > > > Searching "limited but high rows across many shards all with many hits" is > slow > E.g. > * Query from outside client: q=something&rows=1000 > * Resulting in sub-requests to each shard something a-la this > ** 1) q=something&rows=1000&fl=id,score > ** 2) Request the full documents with ids in the global-top-1000 found among > the top-1000 from each shard > What does the subject mean > * "limited but high rows" means 1000 in the example above > * "many shards" means 200-1000 in our case > * "all with many hits" means that each of the shards have a significant > number of hits on the query > The problem grows on all three factors above > Doing such a query on our system takes between 5 min to 1 hour - depending on > a lot of things. It ought to be much faster, so lets make it. > Profiling show that the problem is that it takes lots of time to access the > store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. > Having 1000 shards its up to 1 mio ids that has to be fetched. There is > really no good reason to ever read information from store for more than the > overall top-1000 documents, that has to be returned to the client. > For further detail see mail-thread "Slow searching limited but high rows > across many shards all with high hits" started 13/11-2014 on > dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14245264#comment-14245264 ] Per Steffensen commented on SOLR-6810: -- Now that SOLR-6812 has been solved (in a different way than I suggested), my solution to SOLR-6812 should be removed from the patch. > Faster searching limited but high rows across many shards all with many hits > > > Key: SOLR-6810 > URL: https://issues.apache.org/jira/browse/SOLR-6810 > Project: Solr > Issue Type: Improvement > Components: search > Reporter: Per Steffensen >Assignee: Shalin Shekhar Mangar > Labels: distributed_search, performance > Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch > > > Searching "limited but high rows across many shards all with many hits" is > slow > E.g. > * Query from outside client: q=something&rows=1000 > * Resulting in sub-requests to each shard something a-la this > ** 1) q=something&rows=1000&fl=id,score > ** 2) Request the full documents with ids in the global-top-1000 found among > the top-1000 from each shard > What does the subject mean > * "limited but high rows" means 1000 in the example above > * "many shards" means 200-1000 in our case > * "all with many hits" means that each of the shards have a significant > number of hits on the query > The problem grows on all three factors above > Doing such a query on our system takes between 5 min to 1 hour - depending on > a lot of things. It ought to be much faster, so lets make it. > Profiling show that the problem is that it takes lots of time to access the > store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. > Having 1000 shards its up to 1 mio ids that has to be fetched. There is > really no good reason to ever read information from store for more than the > overall top-1000 documents, that has to be returned to the client. > For further detail see mail-thread "Slow searching limited but high rows > across many shards all with high hits" started 13/11-2014 on > dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244392#comment-14244392 ] Per Steffensen commented on SOLR-4470: -- bq. So maybe one suggestion I would make is - don't give up on this patch. Use this issue as an umbrella issue for adding basic-auth to Solr. Tell us "here's the entire piece of work that I've done, now I want to do it in steps (baby steps preferred)" I am not too happy about this approach in general. I like that only fully done features get committed. If you artificially break a complete feature into small steps AND COMMIT THEM you will temporarily have partly done features. No one says that you will ever get the last parts committed. I see way to much code in Solr that really smells like "we will just do this simple hacky thing here without considering nice code-structure/design and complete features - we will complete, refactor and make it nice later" - but "later" never showed up. One example that I stumbled over just recently is SOLR-5768 - I had to clean it up in SOLR-6795, SOLR-6796, SOLR-6812 and SOLR-6813. Breaking up a complete feature including thorough testing etc. into smaller pieces will have to make us enter that area of partly done stuff. I am afraid that only half of the "subsequent issues" under the "umbrella" will never be committed. That said, I am willing to try. But I want to wait to see what SOLR-6625 brings. It might very well establish a platform for solving this SOLR-4470 much more easily. > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 >Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no > matter what kind of HTTP-request you make to a Solr-node. > It can faily easy be acheived as described on > http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes > also make "internal" request to other Solr-nodes, and for it to work > credentials need to be provided here also. > Ideally we would like to "forward" credentials from a particular request to > all the "internal" sub-requests it triggers. E.g. for search and update > request. > But there are also "internal" requests > * that only indirectly/asynchronously triggered from "outside" requests (e.g. > shard creation/deletion/etc based on calls to the "Collection API") > * that do not in any way have relation to an "outside" "super"-request (e.g. > replica synching stuff) > We would like to aim at a solution where "original" credentials are > "forwarded" when a request directly/synchronously trigger a subrequest, and > fallback to a configured "internal credentials" for the > asynchronous/non-rooted requests. > In our solution we would aim at only supporting basic http auth, but we would > like to make a "framework" around it, so that not to much refactoring is > needed if you later want to make support for other kinds of auth (e.g. digest) > We will work at a solution but create this JIRA issue early in order to get > input/comments from the community as early as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6625) HttpClient callback in HttpSolrServer
[
https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244185#comment-14244185
]
Per Steffensen commented on SOLR-6625:
--
Had a few hours for a look at the latest patch. Not a lot of time so I might
very well have overlooked something, so please bear with me. But from what I
got out of it I have the following comments
It seems you can control the used callbacks in two ways
* 1) Through {{solr.xml}}.
** Request that {{SolrDispatchFilter}} forwards (using {{remoteQuery}})
** Update requests sent through {{UpdateShardHandler}} -
{{DistributedUpdateProcessor}}, {{SnapPuller}}, {{PeerSync}}, {{SyncStrategy}}
...
** {{Overseer}} requests (through {{UpdateShardHandler}} via {{ZkController}})
* 2) Through VM parameter. Used for every request sent - from Solr nodes and
clients or whatever is running in the JVM sending request using
{{HttpSolrServer}} (or {{ConcurrentUpdateSolrServer}}). Believe, for the
requests mentioned under 1) above, both callbacks (the one from {{solr.xml}}
AND the one from VM parameter) will be triggered - is that intentional? (it is
fine for me)
I think, that if you want to support setting the callback through {{solr.xml}}
it should be used for all requests sent out of a Solr node - including e.g.
search requests to shards issued from {{SearchHandler}} (by code inspection it
seems to me that it will not currently be included here). Regarding this
principle of using the {{solr.xml}}-callback for ALL requests going out of a
Solr node, I would really like to see some testing that makes sure this
principle is fulfilled now and that it is not broken in the future (the day
after SOLR-6625 is committed and forgotten about, someone adds a new place in
the code sending requests going out of Solr nodes and forgets to call the
callbacks). I wanted to do the same in SOLR-4470 (making sure that no requests
going out of Solr nodes do not have credentials added - not it the code as it
is today nor in the code as it changes in the future). I achieved it by adding
a protecting layer around every Solr node ever started during testing, and
assuming that the test-suite triggers every type of requests going out of Solr
nodes, that will basically ensure that no one forgets to propagate credentials
when extending the code in the future (those requests will fail and the test
will likely fail). I would really like to see something similar here, verifying
that {{solr.xml}}-callback is triggered for every request going out of a Solr
node.
Maybe you should consider the naming of {{distribCount}} and {{forwardCount}}
in {{BasicDistributedZk2Test.HttpClientRequestCallbackCounter}}. They are very
confusing to me. {{distribCount}} seem to count update-requests (they are kinda
forwarded) and {{forwardCount}} seem to count everything else including request
from e.g. {{Overseer}} (they are among those that can not be characterized as
forwarded)
I know you are not keen on cleaning up while adding a new feature (I was told
that we have to do cleanup in different JIRAs). Personally I am very much
against that, because no one will ever make a JIRA just to clean up (at least,
looking at the code-base I see that it does not happen enough). Assuming we
want to do a little cleanup (in that code that is touched anyway in this JIRA)
I have the following suggestions
* Change {{ConcurrentUpdateSolrServer}} to use {{HttpSolrServer}} or at least
make them share common code. Then we do not have "remember" to add
callback-support in both when we solve this SOLR-6625
* In general we should do some "Separation of Concerns" here and create a
single component X (e.g. {{SolrNodeOutgoingRequestSender}}) concerned with
sending requests out of Solr nodes. Make X use callback from {{solr.xml}}.
** Let {{SolrDispatchFilter}}, {{UpdateShardHandler}} and whatever sends
requests out of Solr nodes use that component X, or something based on or using
it
** It is strange that {{Overseer}} uses {{UpdateShardHandler}} - {{Overseer}}
is not doing updates. Either rename {{UpdateShardHandler}} to something that
does not signal that it is only used for update-requests, or let {{Overseer}}
use something else based on component X
Instead of adding callback-calls numerous places in the code not knowing if
some place was forgotten, polluting the code even more and leaving the same
kind of problems for others doing similar things in the future, IMHO SOLR-6625
should be solved by refactoring the code so that the actual feature of
SOLR-6625 is ridiculously easy to add
* Separating the general concern of sending request from solr-code (server or
client) to one single component Y. ({{HttpSolrServer}} or a class containing
the code shared between it and {{ConcurrentUpdateSolrServer}} is fine for now).
It is easy to make a test that ensures that {{HttpClient.ex
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14241267#comment-14241267 ] Per Steffensen commented on SOLR-6816: -- Thats fine. Just wanted to mention it so that you do not forget when you "nail down a solution" > Review SolrCloud Indexing Performance. > -- > > Key: SOLR-6816 > URL: https://issues.apache.org/jira/browse/SOLR-6816 > Project: Solr > Issue Type: Task > Components: SolrCloud >Reporter: Mark Miller >Priority: Critical > Attachments: SolrBench.pdf > > > We have never really focused on indexing performance, just correctness and > low hanging fruit. We need to vet the performance and try to address any > holes. > Note: A common report is that adding any replication is very slow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6625) HttpClient callback in HttpSolrServer
[ https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14241254#comment-14241254 ] Per Steffensen commented on SOLR-6625: -- bq. Maybe Per Steffensen can chime in with the motivation there rather than always using the static credentials – I'm guessing it is because there is some organization-specific authorization checking expecting the actual user who made the original request. * Every paranoid customers. In general we/they just do not want a user to be able to indirectly (through distributed sub-requests) get a response to a query he is not allowed to fire directly. There is a lot of details to it that I will spare you, but overall we are also 1) using authorization to prevent specific groups of users to do queries (directly or indirectly) that takes certain amounts of resources from the system or 2) to fetch certain fields or any information that can be derived from those fields. Ad 1) By a lot of testing we know which types of queries really takes a lot of resources out of our system, but it is hard to be sure that a query that does not seem to be such a query, does not indirectly fire such queries in distributed sub-requests. Ad 2) Just because an outer request does not specifically ask for a certain field, it is hard to know that the indirect sub-request also does not, even though in practice it is so (except for id an score) * Distributed audit logging * Etc. I realize that the reasons we want the credentials from the outer request forwarded in sub-requests are very thin. I understand if this is not something that you feel is necessary to support in Apache Solr. But if it does not make the solution much more complicated you might just support it. I do not think SOLR-4470 is a complicated solution - most of it, really, is testing. bq. Likewise you can see all of the various other authentication schemes HttpClient supports (SPNego included) SOLR-4470 does not try to do itself what HttpClient can already do. The HttpClient capabilities can just be used outside or through the authentication-framework introduced in SOLR-4470. SOLR-4470 is mostly about joggling the credentials inside Solr-nodes. When they have to be added to requests, of course use HttpClient capabilities. bq. To get around this, we use secure impersonation Interesting! > HttpClient callback in HttpSolrServer > - > > Key: SOLR-6625 > URL: https://issues.apache.org/jira/browse/SOLR-6625 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Attachments: SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, > SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch > > > Some of our setups use Solr in a SPNego/kerberos setup (we've done this by > adding our own filters to the web.xml). We have an issue in that SPNego > requires a negotiation step, but some HttpSolrServer requests are not > repeatable, notably the PUT/POST requests. So, what happens is, > HttpSolrServer sends the requests, the server responds with a negotiation > request, and the request fails because the request is not repeatable. We've > modified our code to send a repeatable request beforehand in these cases. > It would be nicer if HttpSolrServer provided a pre/post callback when it was > making an httpclient request. This would allow administrators to make > changes to the request for authentication purposes, and would allow users to > make per-request changes to the httpclient calls (i.e. modify httpclient > requestconfig to modify the timeout on a per-request basis). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14241199#comment-14241199 ] Per Steffensen commented on SOLR-6816: -- Make sure to document when it is safe to turn off the version-check for replica (overwrite=false). Believe the version-check is also used to detect and prevent old versions that get to the replica last does not overwrite new versions that get to the replica first. E.g leader might get an update-request of document D updating it from state S1 to S2, and send a request U1 to a replica with overwrite=false saying "just write S2 without checking". Immediately after the update of D from S1 to S2, the leader gets another request updating it from S2 to S3. Leader again reacts by sending a "just write S3 without checking"-request U2 to the replica. The replica might receive U2 before U1 (if replica-request is not sent and waited for response while the leader has the bucket-lock for the document) and then you are in trouble. Sure that is an odd case, just do not forget that versions are also used for that on replica (AFAIK) > Review SolrCloud Indexing Performance. > -- > > Key: SOLR-6816 > URL: https://issues.apache.org/jira/browse/SOLR-6816 > Project: Solr > Issue Type: Task > Components: SolrCloud >Reporter: Mark Miller >Priority: Critical > Attachments: SolrBench.pdf > > > We have never really focused on indexing performance, just correctness and > low hanging fruit. We need to vet the performance and try to address any > holes. > Note: A common report is that adding any replication is very slow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14235323#comment-14235323 ] Per Steffensen commented on SOLR-4470: -- Potentially relating to SOLR-6625. We might want to use the more generic callback functionality for adding credentials (in some cases)? > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 > Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no > matter what kind of HTTP-request you make to a Solr-node. > It can faily easy be acheived as described on > http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes > also make "internal" request to other Solr-nodes, and for it to work > credentials need to be provided here also. > Ideally we would like to "forward" credentials from a particular request to > all the "internal" sub-requests it triggers. E.g. for search and update > request. > But there are also "internal" requests > * that only indirectly/asynchronously triggered from "outside" requests (e.g. > shard creation/deletion/etc based on calls to the "Collection API") > * that do not in any way have relation to an "outside" "super"-request (e.g. > replica synching stuff) > We would like to aim at a solution where "original" credentials are > "forwarded" when a request directly/synchronously trigger a subrequest, and > fallback to a configured "internal credentials" for the > asynchronous/non-rooted requests. > In our solution we would aim at only supporting basic http auth, but we would > like to make a "framework" around it, so that not to much refactoring is > needed if you later want to make support for other kinds of auth (e.g. digest) > We will work at a solution but create this JIRA issue early in order to get > input/comments from the community as early as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234548#comment-14234548 ] Per Steffensen edited comment on SOLR-4470 at 12/4/14 7:56 PM: --- Thanks for participating [~zoomieia1980]!!! I am not sure you really understand what my patch is about. It ONLY deals with adding credentials to outgoing requests in order to be able to penetrate a security-layer on the server-side. This is necessary if you choose to somehow put a security-layer between the Solr client and the Solr server. This patch has NOTHING to do with setting up that security-layer around the server. Webcontainers do that very well for you. And if Solr really want to move away from being a webapplication that has to run in a webcontainer, there are numerous other frameworks out there that you can use to set up a security-layer around your server. But this patch has nothing to do with how you add that security-layer around your server. You keep talking about JAAS LoginModules etc. and they really are concepts belonging in this security-layer around the server - thats why I believe you do not completely follow my patch. Changing the SolrJ client so that it adds "hardcoded" credentials to all outgoing requests is really just a few lines around the HTTPClient-usage that you talk about. In this area, my patch really is not more than those few lines, except that I support doing it at several levels * Setting up the credentials once and for all on HTTPClient level, so that it will add those credentials to all requests sent through it * Doing the same on SolrJ XXXSolrServer (clients) level - basically doing it on the HTTPClients of the HTTPSolrServers used directly or underneath more complicated XXXSolrServers like CloudSolrServer * It also supports setting credentials on request-level, allowing you to use different credentials for different requests sent through the same HTTPClient, without having to change the "global" credentials on HTTPClient level But all of this is not that many lines of code The thing is, that in a Cloud setup, the Solr server itself is a Solr client, sending requests to other Solr servers. If there is a security-layer between the Solr servers (and there will be if you use e.g. webcontainer managed security) you need to somehow tell the Solr servers which credentials to add to its outgoing requests targeting other Solr servers. My patch is a lot about this aspect. You can fairly easily make in a way so that it just always uses the same "hardcoded" credentials on those outgoing requests. But you might want to configure those fixed credentials from the outside. What if you actually want to use different credentials depending on the kind of the outgoing request. What if the outgoing request is actually triggered from an incoming request, do you want to use the credentials from the ingoing request on the outgoing request? E.g. if an outside client sends a distributed query R1 to Solr server A, Solr server A has to send sub-requests R2 to Solr server B - should it copy the credentials from R1 to R2 or should it use configured/hardcoded credentials for R2? You might have very different "algorithms" for Solr servers to decide on credentials for the requests it sends to other Solr servers. In some cases the Solr server might need to fetch the credentials from an outside agency, or you might have a handshake/negotiating-step (like SOLR-6625) or something. Most of the non-test-code in my patch is about making "the component that calculates credentials for requests going out of Solr servers" pluggable. But most of the patch is actually test-code. I tend to want to test my stuff thoroughly. In Solr tests we start actual Solr servers sending real requests to each other. So why not test this entire thing by setting up webcontainer managed security in the embedded jetty-containers running the Solr servers in test. Some of the code joggles those things. was (Author: steff1193): Thanks for participating [~zoomieia1980]!!! I am not sure you really understand what my patch is about. It ONLY deals with adding credentials to outgoing requests in order to be able to penetrate a security-layer on the server-side. This is necessary if you choose to somehow put a security-layer between the Solr client and the Solr server. This patch has NOTHING to do with setting up that security-layer around the server. Webcontainers do that very well for you. And if Solr really want to move away from being a webapplication that has to run in a webcontainer, there are numerous other frameworks out there that you can use to set up a security-layer around your server. But this patch has nothing to do with how you add that security-layer around your server. You keep talking about JAAS LoginModule
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234548#comment-14234548 ] Per Steffensen commented on SOLR-4470: -- Thanks for participating [~zoomieia1980]!!! I am not sure you really understand what my patch is about. It ONLY deals with adding credentials to outgoing requests in order to be able to penetrate a security-layer on the server-side. This is necessary if you choose to somehow put a security-layer between the Solr client and the Solr server. This patch has NOTHING to do with setting up that security-layer around the server. Webcontainers do that very well for you. And if Solr really want to move away from being a webapplication that has to run in a webcontainer, there are numerous other frameworks out there that you can use to set up a security-layer around your server. But this patch has nothing to do with how you add that security-layer around your server. You keep talking about JAAS LoginModules etc. and they really are concepts belonging in this security-layer around the server - thats why I believe you do not completely follow my patch. Changing the SolrJ client so that it adds "hardcoded" credentials to all outgoing requests is really just a few lines around the HTTPClient-usage that you talk about. In this area, my patch really is not more than those few lines, except that I support doing it at several levels * Setting up the credentials once and for all on HTTPClient level, so that it will add those credentials to all requests sent through it * Doing the same on SolrJ XXXSolrServer (clients) level - basically doing it on the HTTPClients of the HTTPSolrServers used directly or underneath more complicated XXXSolrServers like CloudSolrServer * It also supports setting credentials on request-level, allowing you to use different credentials for different requests sent through the same HTTPClient, without having to change the "global" credentials on HTTPClient level But all of this is not that many lines of code The thing is, that in a Cloud setup, the Solr server itself is a Solr client, sending requests to other Solr servers. If there is a security-layer between the Solr servers (and there will be if you use e.g. webcontainer managed security) you need to somehow tell the Solr servers which credentials to add to its outgoing requests targeting other Solr servers. My patch is a lot about this aspect. You can fairly easily make in a way so that it just always uses the same "hardcoded" credentials on those outgoing requests. But you might want to configure those fixed credentials from the outside. What if you actually want to use different credentials depending on the kind of the outgoing request. What if the outgoing request is actually triggered from an incoming request, do you want to use the credentials from the ingoing request on the outgoing request? E.g. if an outside client sends a distributed query R1 to Solr server A, Solr server A has to send sub-requests R2 to Solr server B - should it copy the credentials from R1 to R2 or should it use configured/hardcoded credentials from Q2? You might have very different "algorithms" for Solr servers to decide on credentials for the requests it sends to other Solr servers. In some cases the Solr server might need to fetch the credentials from an outside agency, or you might have a handshake/negotiating-step (like SOLR-6625) or something. Most of the non-test-code in my patch is about making "the component that calculates credentials for requests going out of Solr servers" pluggable. But most of the patch is actually test-code. I tend to want to test my stuff thoroughly. In Solr tests we start actual Solr servers sending real requests to each other. So why not test this entire thing by setting up webcontainer managed security in the embedded jetty-containers running the Solr servers in test. Some of the code joggles those things. > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 >Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch,
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234178#comment-14234178 ] Per Steffensen commented on SOLR-4470: -- bq. I noticed that some of the comments were around the size of the patch Yes, unfortunately I have provided a significant number of features/patches that are fairly big that never got committed. It is not that I really want to do it this way. I would much rather, as I sense its most often done, start out with rough half-working patches, and "design the solution" by adding to and discussing the patches as they evolve. Unfortunately that just does not fit very well with the way my customer expects me to work. He tells me (in this case) "I want to protect my Solrs with username/password on HTTP level. You have two weeks to do it". I'd better get started. I make a solution the way I think it should be, and do not have the time to discuss it to much with the rest of you. When I have made a working solution to the customer, I feel obligated to at least try to give it back to Solr - if you want to use Open Source, make sure to (try to) give back whenever you can. This way I end up providing a fully functional complete solution. I understand that this is hard to grasp for you guys. But I also believe that I am often mistaken. I am not saying that you just have to commit this without any change, dialog or whatever. I just want you to pretend that we are in the beginning of solving this ticket, and just look at the full solution from me as a suggestion on how to solve it, that at least can be used starting point for a discussion and "design process". After I have made the complete solution my customer is happy, and then I have the time to discuss/design "the real" solution with you guys. But it usually never gets that far - no committer really want to jump into it. Then I just say to myself that I did my duty and forwarded my solution to the community. If no one wants to participate I cannot do much more. The thing about the discussion/design process is that it is very CALENDAR-time consuming. It is not that I will spend that much actual time on it - usually you just add a few comments, change the patch a little and wait for comments, while you do other stuff (e.g. implementing the next feature for the customer :-) ). So I will have the time to participate in this "after full solution"-phase of making the "right" solution for Solr. bq. Is it correct that every SolrServer can handle authCredentials? And not e.g. only HttpSolrServer Yes bq. Is it mandatory to nearly double the API with this extra parameter? No. Actually I thought [~janhoy] removed the added API methods bq. UpdateRequest.add().doc(doc).commitWithin(-1).authCredentials(creds) Yes the builder-approach is IMHO in general a much better approach compared to having numerous methods with all combinations of parameters or having one methods taking all parameters where "special values" (like null, -1 or something) means that you just want default on that one. But I dont see that done in Solr. When in Rome, I guess... bq. Then we don't put the authCredentials in the face of all users You are probably right bq. I think we ought to be able to break it down into smaller manageable commits Yes, I am sure that can be done. Seems you have a lot of good ideas on how to cut the cake. In regards to this particular SOLR-4470, I think you are focusing too much on the API changes, though. It is a fairly small part of the patch. So if we want to cut SOLR-4470 into smaller patches we need more cutting that just "API stuff" and "the rest". But in general I like all of your ideas in this area. > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 >Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no
[jira] [Comment Edited] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234145#comment-14234145 ] Per Steffensen edited comment on SOLR-6816 at 12/4/14 12:11 PM: Just want to add my 5 cents on this one. It is only regarding indexing when you do version-check/optimistic-locking (SOLR-3178). We have a very different implementation of SOLR-3178, but the performance problems will be the same for "your" implementation. Doing optimistic-locking you typically do a lot of this * 1) real-time-get document D from Solr * 2) update D to D' locally on client * 3) try to replace D with D' in Solr. In case of version-conflict-error go to 1) In step 1) you get-by-id document D, and I step 3) you UpdateLog.lookupVersion on the same id. In our system it is most likely that two processes, both wanting to update document D, run at the same time or fairly shortly after each other. It is rare that the same document gets updated a long time apart. In order to speed up on those aspects, we have introduced a "recently looked-up or updated" cache, where we store documents that has recently been fetch by real-time-get or updated. It has improved our indexing speed significantly. We have a mature solution that is running in production. In the scenarios above you most often discover that the document you try to real-time-get or lookup-version for does NOT exist, but it is relatively time-consuming to realize that (looking in index). We have a PoC of introducing a bloom-filter that can help say one of "document definitely does not exist" (you do not have to search the index) or "document may exist" (you will have to search the index to see if it exists). Our PoC shows that this will speed up our indexing-speed tremendously (like 60-80% reduction), but we havnt prioritized to mature and put it into production yet. The PoC was using a modified version of Guava bloom-filter - modified to be able work in a memory-mapped file, so that we do not lose bloom-filter information when shutting down Solr (it will take some time building it from scratch every time you start Solr). Guava bloom-filter currently is memory only - you can save it to file and load it again, but it will not go on continuously, and it is not efficient to store it completely to disk at every update :-) Hence the "work in memory-mapped file" modification. Of course, let me know if any of this sounds interesting to you. was (Author: steff1193): Just want to add my 5 cents on this one. It is only regarding indexing when you do version-check/optimistic-locking (SOLR-3178). We have a very different implementation of SOLR-3178, but the performance problems will be the same for "your" implementation. Doing optimistic-locking you typically do a lot of this * 1) real-time-get document D from Solr * 2) update D to D' locally on client * 3) try to replace D with D' in Solr. In case of version-conflict-error go to 1) In step 1) you get-by-id document D, and I step 3) you UpdateLog.lookupVersion on the same id. In our system it is most likely that two processes, both wanting to update document D, run at the same time or fairly shortly after each other. It is rare that the same document gets updated a long time apart. In order to speed up on those aspects, we have introduced a "recently looked-up or updated" cache, where we store documents that has recently been fetch by real-time-get or updated. It has improved our indexing speed significantly. We have a mature solution that is running in production. In the scenarios above you most often discover that the document you try to real-time-get or lookup-version for does NOT exist, but it is relatively time-consuming to realize that (looking in index). We have a PoC of introducing a bloom-filter that can help say one of "document definitely does not exist" (you do not have to search the index) or "document may exist" (you will have to search the index to see if it exists). Our PoC shows that this will speed up our indexing-speed tremendously (like 60-80% reduction), but we havnt prioritized to mature and put it into production yet. The PoC was using a modified version of Guava bloom-filter - modified to be able work in a memory-mapped file, so that we do not lose bloom-filter information when shutting down Solr (it will take some time building it from scratch every time you start Solr). Guava bloom-filter currently is memory only - you can save it to file and load it again, but it will not go on continuously, and it is not efficient to store it completely to disk at every update :-) Hence the "work in memory-mapped file" modification. > Review SolrCloud Indexing Performance. >
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234145#comment-14234145 ] Per Steffensen commented on SOLR-6816: -- Just want to add my 5 cents on this one. It is only regarding indexing when you do version-check/optimistic-locking (SOLR-3178). We have a very different implementation of SOLR-3178, but the performance problems will be the same for "your" implementation. Doing optimistic-locking you typically do a lot of this * 1) real-time-get document D from Solr * 2) update D to D' locally on client * 3) try to replace D with D' in Solr. In case of version-conflict-error go to 1) In step 1) you get-by-id document D, and I step 3) you UpdateLog.lookupVersion on the same id. In our system it is most likely that two processes, both wanting to update document D, run at the same time or fairly shortly after each other. It is rare that the same document gets updated a long time apart. In order to speed up on those aspects, we have introduced a "recently looked-up or updated" cache, where we store documents that has recently been fetch by real-time-get or updated. It has improved our indexing speed significantly. We have a mature solution that is running in production. In the scenarios above you most often discover that the document you try to real-time-get or lookup-version for does NOT exist, but it is relatively time-consuming to realize that (looking in index). We have a PoC of introducing a bloom-filter that can help say one of "document definitely does not exist" (you do not have to search the index) or "document may exist" (you will have to search the index to see if it exists). Our PoC shows that this will speed up our indexing-speed tremendously (like 60-80% reduction), but we havnt prioritized to mature and put it into production yet. The PoC was using a modified version of Guava bloom-filter - modified to be able work in a memory-mapped file, so that we do not lose bloom-filter information when shutting down Solr (it will take some time building it from scratch every time you start Solr). Guava bloom-filter currently is memory only - you can save it to file and load it again, but it will not go on continuously, and it is not efficient to store it completely to disk at every update :-) Hence the "work in memory-mapped file" modification. > Review SolrCloud Indexing Performance. > -- > > Key: SOLR-6816 > URL: https://issues.apache.org/jira/browse/SOLR-6816 > Project: Solr > Issue Type: Task > Components: SolrCloud >Reporter: Mark Miller >Priority: Critical > Attachments: SolrBench.pdf > > > We have never really focused on indexing performance, just correctness and > low hanging fruit. We need to vet the performance and try to address any > holes. > Note: A common report is that adding any replication is very slow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232772#comment-14232772
]
Per Steffensen edited comment on SOLR-6810 at 12/4/14 10:59 AM:
We have solved the problem (reducing response-time by a factor of 60 on our
particular system/data/distribution) the following way
Introduced the concept of "distributed query algorithm" (DQA) controlled by
request parameter {{dqa}}. Naming the existing (default) distributed query
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and
introducing a new alternative distributed query algorithm called
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias
{{frfilrfbi}})
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids}} does it in a different
way - see JavaDoc of
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}
Believe “distributed query algorithm” is a pendant to elasticsearch's “search
type”, but just with much better naming that say something about what it is
actually controlling :-)
Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}}
that becomes single-pass (going from 2 to 1 pass) with this flag.
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes.
{{dqa.forceSkipGetIds=true}} is default for
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}.
Attaching patch corresponding to our solution - going into production as we
speak to reduce our response-times by a factor of 60. You do not necessarily
need to just adopt it. But lets at least consider it a starting-point for a
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score.
This is important for the optimization
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s.
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test
asserts that you only go to store X times across the cluster and not (up to)
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help
collecting information about how {{LeafReader}} s are used in different
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around
DQA’s. And made more systematic in the way the tests are performed. Do not want
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired
running the test-suite - when you do not specify which DQA you want explicitly
in the request. With helper-methods for fixing the DQA for tests that focus on
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the
test-suite green. But should probably be committed as part of SOLR-6812, and
left out of this SOLR-6810. New DQA
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default.
And since we run tests randomly selecting the DQA for every query, we are also
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will
likely fail if skip-get-ids does not work for all kinds of requests. This is
actually also a good way to have {{dqa.forceSkipGetIds}} (old
{{distrib.singlePass}}) tested, so that we will not have a partially-working
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have
been removed again, because the problems (along with any other problems with
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it
will not make the test-suite fail, by forcing the particular query in
{{DistributedExpandComponentTest
[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Per Steffensen updated SOLR-6810:
-
Attachment: branch_5x_rev1642874.patch
Got in doubt how exactly the store-read-improvements would be when start-param
is > 0. Now {{TestDistributedQueryAlgorithm.testDocReads}} clearly show that.
Besides that improved test-coverage for sort + start > 0 queries in
{{TestDistributedQueryAlgorithm}}. And better assert messages.
> Faster searching limited but high rows across many shards all with many hits
>
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Per Steffensen
> Labels: distributed_search, performance
> Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is
> slow
> E.g.
> * Query from outside client: q=something&rows=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something&rows=1000&fl=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards have a significant
> number of hits on the query
> The problem grows on all three factors above
> Doing such a query on our system takes between 5 min to 1 hour - depending on
> a lot of things. It ought to be much faster, so lets make it.
> Profiling show that the problem is that it takes lots of time to access the
> store to get id’s for (up to) 1000 docs (value of rows parameter) per shard.
> Having 1000 shards its up to 1 mio ids that has to be fetched. There is
> really no good reason to ever read information from store for more than the
> overall top-1000 documents, that has to be returned to the client.
> For further detail see mail-thread "Slow searching limited but high rows
> across many shards all with high hits" started 13/11-2014 on
> dev@lucene.apache.org
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234103#comment-14234103 ] Per Steffensen commented on SOLR-4470: -- But voting for it might help :-) > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 > Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no > matter what kind of HTTP-request you make to a Solr-node. > It can faily easy be acheived as described on > http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes > also make "internal" request to other Solr-nodes, and for it to work > credentials need to be provided here also. > Ideally we would like to "forward" credentials from a particular request to > all the "internal" sub-requests it triggers. E.g. for search and update > request. > But there are also "internal" requests > * that only indirectly/asynchronously triggered from "outside" requests (e.g. > shard creation/deletion/etc based on calls to the "Collection API") > * that do not in any way have relation to an "outside" "super"-request (e.g. > replica synching stuff) > We would like to aim at a solution where "original" credentials are > "forwarded" when a request directly/synchronously trigger a subrequest, and > fallback to a configured "internal credentials" for the > asynchronous/non-rooted requests. > In our solution we would aim at only supporting basic http auth, but we would > like to make a "framework" around it, so that not to much refactoring is > needed if you later want to make support for other kinds of auth (e.g. digest) > We will work at a solution but create this JIRA issue early in order to get > input/comments from the community as early as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests
[ https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234089#comment-14234089 ] Per Steffensen commented on SOLR-4470: -- bq. Does anyone have an idea when this will be released? Thanks! My best guess is never, unfortunately. But you can build your own version of Solr including it. We do that. I can give you a few hints on how to build your own Solr, if you want. Do not have the time to give you a thorough description. But a few hints I will manage. Let me know. Do not know if it is actually described in details somewhere? > Support for basic http auth in internal solr requests > - > > Key: SOLR-4470 > URL: https://issues.apache.org/jira/browse/SOLR-4470 > Project: Solr > Issue Type: New Feature > Components: clients - java, multicore, replication (java), SolrCloud >Affects Versions: 4.0 > Reporter: Per Steffensen >Assignee: Jan Høydahl > Labels: authentication, https, solrclient, solrcloud, ssl > Fix For: Trunk > > Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, > SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, > SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, > SOLR-4470_trunk_r1568857.patch > > > We want to protect any HTTP-resource (url). We want to require credentials no > matter what kind of HTTP-request you make to a Solr-node. > It can faily easy be acheived as described on > http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes > also make "internal" request to other Solr-nodes, and for it to work > credentials need to be provided here also. > Ideally we would like to "forward" credentials from a particular request to > all the "internal" sub-requests it triggers. E.g. for search and update > request. > But there are also "internal" requests > * that only indirectly/asynchronously triggered from "outside" requests (e.g. > shard creation/deletion/etc based on calls to the "Collection API") > * that do not in any way have relation to an "outside" "super"-request (e.g. > replica synching stuff) > We would like to aim at a solution where "original" credentials are > "forwarded" when a request directly/synchronously trigger a subrequest, and > fallback to a configured "internal credentials" for the > asynchronous/non-rooted requests. > In our solution we would aim at only supporting basic http auth, but we would > like to make a "framework" around it, so that not to much refactoring is > needed if you later want to make support for other kinds of auth (e.g. digest) > We will work at a solution but create this JIRA issue early in order to get > input/comments from the community as early as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232772#comment-14232772
]
Per Steffensen edited comment on SOLR-6810 at 12/3/14 3:53 PM:
---
We have solved the problem (reducing response-time by a factor of 60 on our
particular system/data/distribution) the following way
Introduced the concept of "distributed query algorithm" (DQA) controlled by
request parameter {{dqa}}. Naming the existing (default) distributed query
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and
introducing a new alternative distributed query algorithm called
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias
{{frfilrfbi}})
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids}} does it in a different
way - see JavaDoc of
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}
Believe “distributed query algorithm” is a pendant to elasticsearch's “search
type”, but just with much better naming that say something about what it is
actually controlling :-)
Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}}
that becomes single-pass (going from 2 to 1 pass) with this flag.
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes.
{{dqa.forceSkipGetIds=true}} is default for
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}. There are really no need
to ever run with {{dqa.forceSkipGetIds=false}} for this DQA, but it is
supported.
Attaching patch corresponding to our solution - going into production as we
speak to reduce our response-times by a factor of 60. You do not necessarily
need to just adopt it. But lets at least consider it a starting-point for a
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score.
This is important for the optimization
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s.
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test
asserts that you only go to store X times across the cluster and not (up to)
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help
collecting information about how {{LeafReader}} s are used in different
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around
DQA’s. And made more systematic in the way the tests are performed. Do not want
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired
running the test-suite - when you do not specify which DQA you want explicitly
in the request. With helper-methods for fixing the DQA for tests that focus on
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the
test-suite green. But should probably be committed as part of SOLR-6812, and
left out of this SOLR-6810. New DQA
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default.
And since we run tests randomly selecting the DQA for every query, we are also
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will
likely fail if skip-get-ids does not work for all kinds of requests. This is
actually also a good way to have {{dqa.forceSkipGetIds}} (old
{{distrib.singlePass}}) tested, so that we will not have a partially-working
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have
been removed again, because the problems (along with any other problems with
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it
will not
[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232772#comment-14232772
]
Per Steffensen edited comment on SOLR-6810 at 12/3/14 3:48 PM:
---
We have solved the problem (reducing response-time by a factor of 60 on our
particular system/data/distribution) the following way
Introduced the concept of "distributed query algorithm" (DQA) controlled by
request parameter {{dqa}}. Naming the existing (default) distributed query
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and
introducing a new alternative distributed query algorithm called
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias
{{frfilrfbi}})
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids}} does it in a different
way - see JavaDoc of
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}
Believe “distributed query algorithm” is a pendant to elasticsearch's “search
type”, but just with much better naming that say something about what it is
actually controlling :-)
Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}}
that becomes single-pass (going from 2 to 1 pass) with this flag.
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes.
{{dqa.forceSkipGetIds=true}} is default for
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}. There are really no need
to ever run with {{dqa.forceSkipGetIds=false}} for this DQA, but it is
supported.
Attaching patch corresponding to our solution - going into production as we
speak to reduce our response-times by a factor of 60. You do not necessarily
need to just adopt it. But lets at least consider it a starting-point for a
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score.
This is important for the optimization
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s.
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test
asserts that you only go to store X times across the cluster and not (up to)
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help
collecting information about how {{LeafReader}} s are used in different
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around
DQA’s. And made more systematic in the way the tests are performed. Do not want
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired
running the test-suite - when you do not specify which DQA you want explicitly
in the request. With helper-methods for fixing the DQA for tests that focus on
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the
test-suite green. But should probably be committed as part of SOLR-6812, and
left out of this SOLR-6810. New DQA
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default.
And since we run tests randomly selecting the DQA for every query, we are also
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will
likely fail if skip-get-ids does not work for all kinds of requests. This is
actually also a good way to have {{dqa.forceSkipGetIds}} (old
{{distrib.singlePass}}) tested, so that we will not have a partially-working
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have
been removed again, because the problems (along with any other problems with
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it
will not
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232781#comment-14232781
]
Per Steffensen commented on SOLR-6810:
--
SOLR-6795, SOLR-6796, SOLR-6812 and SOLR-6813 was all just preparation for this
patch. It makes {{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) work for
all queries (ever fires during the test-suite). So we have a complete feature
now (when those issues are corrected).
> Faster searching limited but high rows across many shards all with many hits
>
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Per Steffensen
> Labels: distributed_search, performance
> Attachments: branch_5x_rev1642874.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is
> slow
> E.g.
> * Query from outside client: q=something&rows=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something&rows=1000&fl=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards have a significant
> number of hits on the query
> The problem grows on all three factors above
> Doing such a query on our system takes between 5 min to 1 hour - depending on
> a lot of things. It ought to be much faster, so lets make it.
> Profiling show that the problem is that it takes lots of time to access the
> store to get id’s for (up to) 1000 docs (value of rows parameter) per shard.
> Having 1000 shards its up to 1 mio ids that has to be fetched. There is
> really no good reason to ever read information from store for more than the
> overall top-1000 documents, that has to be returned to the client.
> For further detail see mail-thread "Slow searching limited but high rows
> across many shards all with high hits" started 13/11-2014 on
> dev@lucene.apache.org
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Per Steffensen updated SOLR-6810:
-
Attachment: branch_5x_rev1642874.patch
We have solved the problem (reducing response-time by a factor of 60 on our
particular system/data/distribution) the following way
Introduced the concept of "distributed query algorithm" (DQA) controlled by
request parameter {{dqa}}. Naming the existing (default) distributed query
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and
introducing a new alternative distributed query algorithm called
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias
{{frfilrfbi}})
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids does}} it in a different
way - see JavaDoc of
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}
Believe “distributed query algorithm” is a pendant to elasticsearch's “search
type”, but just with much better naming that say something about what it is
actually controlling :-)
Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}}
that becomes single-pass (going from 2 to 1 pass) with this flag.
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes.
{{dqa.forceSkipGetIds=true}} is default for
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}. There are really no need
to ever run with {{dqa.forceSkipGetIds=false}} for this DQA, but it is
supported.
Attaching patch corresponding to our solution - going into production as we
speak to reduce our response-times by a factor of 60. You do not necessarily
need to just adopt it. But lets at least consider it a starting-point for a
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score.
This is important for the optimization
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s.
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test
asserts that you only go to store X times across the cluster and not (up to)
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help
collecting information about how {{LeafReader}} s are used in different
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around
DQA’s. And made more systematic in the way the tests are performed. Do not want
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired
running the test-suite - when you do not specify which DQA you want explicitly
in the request. With helper-methods for fixing the DQA for tests that focus on
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the
test-suite green. But should probably be committed as part of SOLR-6812, and
left out of this SOLR-6810. New DQA
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default.
And since we run tests randomly selecting the DQA for every query, we are also
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will
likely fail if skip-get-ids does not work for all kinds of requests. This is
actually also a good way to have {{dqa.forceSkipGetIds}} (old
{{distrib.singlePass}}) tested, so that we will not have a partially-working
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have
been removed again, because the problems (along with any other problems with
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it
will not make the test-suite fail, by forcing the part
[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232781#comment-14232781
]
Per Steffensen edited comment on SOLR-6810 at 12/3/14 8:44 AM:
---
SOLR-6795, SOLR-6796, SOLR-6812 and SOLR-6813 where all just preparation for
this patch. It makes {{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) work
for all queries (ever fired during the test-suite). So we have a complete
feature now (when those issues are corrected).
was (Author: steff1193):
SOLR-6795, SOLR-6796, SOLR-6812 and SOLR-6813 was all just preparation for this
patch. It makes {{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) work for
all queries (ever fires during the test-suite). So we have a complete feature
now (when those issues are corrected).
> Faster searching limited but high rows across many shards all with many hits
>
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Per Steffensen
> Labels: distributed_search, performance
> Attachments: branch_5x_rev1642874.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is
> slow
> E.g.
> * Query from outside client: q=something&rows=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something&rows=1000&fl=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards have a significant
> number of hits on the query
> The problem grows on all three factors above
> Doing such a query on our system takes between 5 min to 1 hour - depending on
> a lot of things. It ought to be much faster, so lets make it.
> Profiling show that the problem is that it takes lots of time to access the
> store to get id’s for (up to) 1000 docs (value of rows parameter) per shard.
> Having 1000 shards its up to 1 mio ids that has to be fetched. There is
> really no good reason to ever read information from store for more than the
> overall top-1000 documents, that has to be returned to the client.
> For further detail see mail-thread "Slow searching limited but high rows
> across many shards all with high hits" started 13/11-2014 on
> dev@lucene.apache.org
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232772#comment-14232772
]
Per Steffensen edited comment on SOLR-6810 at 12/3/14 8:50 AM:
---
We have solved the problem (reducing response-time by a factor of 60 on our
particular system/data/distribution) the following way
Introduced the concept of "distributed query algorithm" (DQA) controlled by
request parameter {{dqa}}. Naming the existing (default) distributed query
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and
introducing a new alternative distributed query algorithm called
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias
{{frfilrfbi}})
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids does}} it in a different
way - see JavaDoc of
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}
Believe “distributed query algorithm” is a pendant to elasticsearch's “search
type”, but just with much better naming that say something about what it is
actually controlling :-)
Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}}
that becomes single-pass (going from 2 to 1 pass) with this flag.
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes.
{{dqa.forceSkipGetIds=true}} is default for
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}. There are really no need
to ever run with {{dqa.forceSkipGetIds=false}} for this DQA, but it is
supported.
Attaching patch corresponding to our solution - going into production as we
speak to reduce our response-times by a factor of 60. You do not necessarily
need to just adopt it. But lets at least consider it a starting-point for a
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score.
This is important for the optimization
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s.
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test
asserts that you only go to store X times across the cluster and not (up to)
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help
collecting information about how {{LeafReader}} s are used in different
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around
DQA’s. And made more systematic in the way the tests are performed. Do not want
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired
running the test-suite - when you do not specify which DQA you want explicitly
in the request. With helper-methods for fixing the DQA for tests that focus on
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the
test-suite green. But should probably be committed as part of SOLR-6812, and
left out of this SOLR-6810. New DQA
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default.
And since we run tests randomly selecting the DQA for every query, we are also
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will
likely fail if skip-get-ids does not work for all kinds of requests. This is
actually also a good way to have {{dqa.forceSkipGetIds}} (old
{{distrib.singlePass}}) tested, so that we will not have a partially-working
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have
been removed again, because the problems (along with any other problems with
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it
will not
[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-6810: - Description: Searching "limited but high rows across many shards all with many hits" is slow E.g. * Query from outside client: q=something&rows=1000 * Resulting in sub-requests to each shard something a-la this ** 1) q=something&rows=1000&fl=id,score ** 2) Request the full documents with ids in the global-top-1000 found among the top-1000 from each shard What does the subject mean * "limited but high rows" means 1000 in the example above * "many shards" means 200-1000 in our case * "all with many hits" means that each of the shards have a significant number of hits on the query The problem grows on all three factors above Doing such a query on our system takes between 5 min to 1 hour - depending on a lot of things. It ought to be much faster, so lets make it. Profiling show that the problem is that it takes lots of time to access the store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. Having 1000 shards its up to 1 mio ids that has to be fetched. There is really no good reason to ever read information from store for more than the overall top-1000 documents, that has to be returned to the client. For further detail see mail-thread "Slow searching limited but high rows across many shards all with high hits" started 13/11-2014 on dev@lucene.apache.org was: Searching "limited but high rows across many shards all with many hits" is slow E.g. * Query from outside client: q=something&rows=1000 * Resulting in sub-requests to each shard something a-la this ** 1) q=something&rows=1000&fl=id,score ** 2) Request the full documents with ids in the global-top-1000 found among the top-1000 from each shard What does the subject mean * "limited but high rows" means 1000 in the example above * "many shards" means 200-1000 in our case * "all with many hits" means that each of the shards have a significant number of hits on the query The problem grows on all three factors above Doing such a query on our system takes between 5 min to 1 hour - depending on a lot of things. It ought to be much faster, so lets make it. For further detail see mail-thread "Slow searching limited but high rows across many shards all with high hits" started 13/11-2014 on dev@lucene.apache.org > Faster searching limited but high rows across many shards all with many hits > > > Key: SOLR-6810 > URL: https://issues.apache.org/jira/browse/SOLR-6810 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: Per Steffensen > Labels: distributed_search, performance > > Searching "limited but high rows across many shards all with many hits" is > slow > E.g. > * Query from outside client: q=something&rows=1000 > * Resulting in sub-requests to each shard something a-la this > ** 1) q=something&rows=1000&fl=id,score > ** 2) Request the full documents with ids in the global-top-1000 found among > the top-1000 from each shard > What does the subject mean > * "limited but high rows" means 1000 in the example above > * "many shards" means 200-1000 in our case > * "all with many hits" means that each of the shards have a significant > number of hits on the query > The problem grows on all three factors above > Doing such a query on our system takes between 5 min to 1 hour - depending on > a lot of things. It ought to be much faster, so lets make it. > Profiling show that the problem is that it takes lots of time to access the > store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. > Having 1000 shards its up to 1 mio ids that has to be fetched. There is > really no good reason to ever read information from store for more than the > overall top-1000 documents, that has to be returned to the client. > For further detail see mail-thread "Slow searching limited but high rows > across many shards all with high hits" started 13/11-2014 on > dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6812) distrib.singlePass does not work for expand-request
[
https://issues.apache.org/jira/browse/SOLR-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231323#comment-14231323
]
Per Steffensen edited comment on SOLR-6812 at 12/2/14 3:09 PM:
---
Attached a fix to the problem. Problem is that {{ExpandComponent}} kinda
"detects" that it is supposed to do its thing for shard-searches, by seeing if
ids-param is there. It is not in case of {{distrib.singlePass}}.
Patch works on purpose instead, because that is really whats controlling the
things in {{QueryComponent}}. So now purpose is distributed in the
shard-requests.
was (Author: steff1193):
Attached a fix to the problem. Problem is that {{ExpandComponent}} kinda
"detects" that it is supposed to do its thing for shard-searches, by seeing if
ids-param is there. It is not in case of {{distrib.singlePass}}.
Patch works on purpose instead, because that is really whats controlling the
things in {{QueryComponent}}. So not purpose is distributed in the
shard-requests.
> distrib.singlePass does not work for expand-request
> ---
>
> Key: SOLR-6812
> URL: https://issues.apache.org/jira/browse/SOLR-6812
> Project: Solr
> Issue Type: Bug
> Components: multicore, search
>Reporter: Per Steffensen
> Labels: distributed_search, search
> Attachments: fix.patch, test_that_reveals_the_problem.patch
>
>
> Using distrib.singlePass does not work for expand-requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests
[ https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231325#comment-14231325 ] Per Steffensen edited comment on SOLR-6796 at 12/2/14 12:55 PM: Maybe I can convince you to take a look at SOLR-6812 also, [~shalinmangar]? And maybe SOLR-6813? :-) I know I am pushing it now :-) was (Author: steff1193): Maybe I can convince you to take a look at SOLR-6812 also, [~shalinmangar]? > distrib.singlePass does not return correct set of fields for > multi-fl-parameter requests > > > Key: SOLR-6796 > URL: https://issues.apache.org/jira/browse/SOLR-6796 > Project: Solr > Issue Type: Bug > Components: multicore, search >Affects Versions: 5.0 >Reporter: Per Steffensen >Assignee: Shalin Shekhar Mangar > Labels: distributed_search, search > Attachments: fix.patch, fix.patch, fix.patch, fix.patch, > test_that_reveals_the_problem.patch > > > If I pass distrib.singlePass in a request that also has two fl-parameters, in > some cases, I will not get the expected set of fields back for the returned > documents -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
[
https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231428#comment-14231428
]
Per Steffensen commented on SOLR-6813:
--
Unlike for SOLR-6812 (and SOLR-6795 and SOLR-6796) I am not going to provide a
fix here, because I really do not know enough about expand-requests to be able
to say which response is actually the correct one - {{controlRsp}} or {{rsp}}
in {{BaseDistributedSearchTestCase.query}} when doing the modified request from
{{test_that_reveals_the_problem.patch}}
> distrib.singlePass does not work for expand-request - start/rows included
> -
>
> Key: SOLR-6813
> URL: https://issues.apache.org/jira/browse/SOLR-6813
> Project: Solr
> Issue Type: Bug
> Components: multicore, search
> Reporter: Per Steffensen
> Labels: distributed_search, search
> Attachments: test_that_reveals_the_problem.patch
>
>
> Using distrib.singlePass does not work for expand-requests. Even after the
> fix provided to SOLR-6812, it does not work for requests where you add start
> and/or rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
[ https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-6813: - Attachment: test_that_reveals_the_problem.patch Attached patch (branch_5x) that change an existing test to reveal the problem. Apply the patch and run DistributedExpandComponentTest Probably not just want to commit the changed test after a fix of the problem, but maybe we want to add the test somewhere else. > distrib.singlePass does not work for expand-request - start/rows included > - > > Key: SOLR-6813 > URL: https://issues.apache.org/jira/browse/SOLR-6813 > Project: Solr > Issue Type: Bug > Components: multicore, search > Reporter: Per Steffensen > Labels: distributed_search, search > Attachments: test_that_reveals_the_problem.patch > > > Using distrib.singlePass does not work for expand-requests. Even after the > fix provided to SOLR-6812, it does not work for requests where you add start > and/or rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
Per Steffensen created SOLR-6813: Summary: distrib.singlePass does not work for expand-request - start/rows included Key: SOLR-6813 URL: https://issues.apache.org/jira/browse/SOLR-6813 Project: Solr Issue Type: Bug Components: multicore, search Reporter: Per Steffensen Using distrib.singlePass does not work for expand-requests. Even after the fix provided to SOLR-6812, it does not work for requests where you add start and/or rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests
[ https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231325#comment-14231325 ] Per Steffensen commented on SOLR-6796: -- Maybe I can convince you to take a look at SOLR-6812 also, [~shalinmangar]? > distrib.singlePass does not return correct set of fields for > multi-fl-parameter requests > > > Key: SOLR-6796 > URL: https://issues.apache.org/jira/browse/SOLR-6796 > Project: Solr > Issue Type: Bug > Components: multicore, search >Affects Versions: 5.0 >Reporter: Per Steffensen >Assignee: Shalin Shekhar Mangar > Labels: distributed_search, search > Attachments: fix.patch, fix.patch, fix.patch, fix.patch, > test_that_reveals_the_problem.patch > > > If I pass distrib.singlePass in a request that also has two fl-parameters, in > some cases, I will not get the expected set of fields back for the returned > documents -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6812) distrib.singlePass does not work for expand-request
[
https://issues.apache.org/jira/browse/SOLR-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Per Steffensen updated SOLR-6812:
-
Attachment: fix.patch
Attached a fix to the problem. Problem is that {{ExpandComponent}} kinda
"detects" that it is supposed to do its thing for shard-searches, by seeing if
ids-param is there. It is not in case of {{distrib.singlePass}}.
Patch works on purpose instead, because that is really whats controlling the
things in {{QueryComponent}}. So not purpose is distributed in the
shard-requests.
> distrib.singlePass does not work for expand-request
> ---
>
> Key: SOLR-6812
> URL: https://issues.apache.org/jira/browse/SOLR-6812
> Project: Solr
> Issue Type: Bug
> Components: multicore, search
>Reporter: Per Steffensen
> Labels: distributed_search, search
> Attachments: fix.patch, test_that_reveals_the_problem.patch
>
>
> Using distrib.singlePass does not work for expand-requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6812) distrib.singlePass does not work for expand-request
[
https://issues.apache.org/jira/browse/SOLR-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Per Steffensen updated SOLR-6812:
-
Attachment: test_that_reveals_the_problem.patch
Attached patch (branch_5x) that change an existing test to reveal the problem.
Apply the patch and run {{DistributedExpandComponentTest}}
Probably not just want to commit the changed test after a fix of the problem,
but maybe we want to add the test somewhere else.
> distrib.singlePass does not work for expand-request
> ---
>
> Key: SOLR-6812
> URL: https://issues.apache.org/jira/browse/SOLR-6812
> Project: Solr
> Issue Type: Bug
> Components: multicore, search
> Reporter: Per Steffensen
> Labels: distributed_search, search
> Attachments: test_that_reveals_the_problem.patch
>
>
> Using distrib.singlePass does not work for expand-requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6812) distrib.singlePass does not work for expand-request
Per Steffensen created SOLR-6812: Summary: distrib.singlePass does not work for expand-request Key: SOLR-6812 URL: https://issues.apache.org/jira/browse/SOLR-6812 Project: Solr Issue Type: Bug Components: multicore, search Reporter: Per Steffensen Using distrib.singlePass does not work for expand-requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Slow searching limited but high rows across many shards all with high hits
On 18/11/14 14:41, Toke Eskildsen wrote: Your solution would help others. The obvious next step would be a JIRA. https://issues.apache.org/jira/browse/SOLR-6810. I will provide our solution tomorrow (hopefully) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
Per Steffensen created SOLR-6810: Summary: Faster searching limited but high rows across many shards all with many hits Key: SOLR-6810 URL: https://issues.apache.org/jira/browse/SOLR-6810 Project: Solr Issue Type: Improvement Components: search Reporter: Per Steffensen Searching "limited but high rows across many shards all with many hits" is slow E.g. * Query from outside client: q=something&rows=1000 * Resulting in sub-requests to each shard something a-la this ** 1) q=something&rows=1000&fl=id,score ** 2) Request the full documents with ids in the global-top-1000 found among the top-1000 from each shard What does the subject mean * "limited but high rows" means 1000 in the example above * "many shards" means 200-1000 in our case * "all with many hits" means that each of the shards have a significant number of hits on the query The problem grows on all three factors above Doing such a query on our system takes between 5 min to 1 hour - depending on a lot of things. It ought to be much faster, so lets make it. For further detail see mail-thread "Slow searching limited but high rows across many shards all with high hits" started 13/11-2014 on dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
