[jira] [Commented] (SOLR-13593) Allow to specify analyzer components by their SPI names in schema definition
[
https://issues.apache.org/jira/browse/SOLR-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876372#comment-16876372
]
Varun Thacker commented on SOLR-13593:
--
{quote}I agree with this idea! Meanwhile I think the field type resolution
should be treated in another issue.
{quote}
Make sense!
> Allow to specify analyzer components by their SPI names in schema definition
>
>
> Key: SOLR-13593
> URL: https://issues.apache.org/jira/browse/SOLR-13593
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Schema and Analysis
>Reporter: Tomoko Uchida
>Priority: Major
>
> Now each analysis factory has explicitely documented SPI name which is stored
> in the static "NAME" field (LUCENE-8778).
> Solr uses factories' simple class name in schema definition (like
> class="solr.WhitespaceTokenizerFactory"), but we should be able to also use
> more concise SPI names (like name="whitespace").
> e.g.:
> {code:xml}
>
>
>
> />
>
>
>
> {code}
> would be
> {code:xml}
>
>
>
>
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13593) Allow to specify analyzer components by their SPI names in schema definition
[
https://issues.apache.org/jira/browse/SOLR-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876223#comment-16876223
]
Varun Thacker edited comment on SOLR-13593 at 7/1/19 2:33 PM:
--
+1 for having the ability to say "whitespace" etc
-Is the attribute called name or class? in your example the tokenizer has
{{name=whitespace}} but the filter has- {{-class=keywordMarker.- Maybe
}}{{type}} sounds better?
Also should {{solr.TextField}} be just {{text}} ?
was (Author: varunthacker):
+1 for having the ability to say "whitespace" etc
Is the attribute called name or class? in your example the tokenizer has
{{name=whitespace}} but the filter has {{class=keywordMarker . Maybe }}{{type}}
sounds better?
Also should {{solr.TextField}} be just {{text}} ?
> Allow to specify analyzer components by their SPI names in schema definition
>
>
> Key: SOLR-13593
> URL: https://issues.apache.org/jira/browse/SOLR-13593
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Schema and Analysis
>Reporter: Tomoko Uchida
>Priority: Major
>
> Now each analysis factory has explicitely documented SPI name which is stored
> in the static "NAME" field (LUCENE-8778).
> Solr uses factories' simple class name in schema definition (like
> class="solr.WhitespaceTokenizerFactory"), but we should be able to also use
> more concise SPI names (like name="whitespace").
> e.g.:
> {code:xml}
>
>
>
> />
>
>
>
> {code}
> would be
> {code:xml}
>
>
>
>
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13593) Allow to specify analyzer components by their SPI names in schema definition
[
https://issues.apache.org/jira/browse/SOLR-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876223#comment-16876223
]
Varun Thacker commented on SOLR-13593:
--
+1 for having the ability to say "whitespace" etc
Is the attribute called name or class? in your example the tokenizer has
{{name=whitespace}} but the filter has {{class=keywordMarker . Maybe }}{{type}}
sounds better?
Also should {{solr.TextField}} be just {{text}} ?
> Allow to specify analyzer components by their SPI names in schema definition
>
>
> Key: SOLR-13593
> URL: https://issues.apache.org/jira/browse/SOLR-13593
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Schema and Analysis
>Reporter: Tomoko Uchida
>Priority: Major
>
> Now each analysis factory has explicitely documented SPI name which is stored
> in the static "NAME" field (LUCENE-8778).
> Solr uses factories' simple class name in schema definition (like
> class="solr.WhitespaceTokenizerFactory"), but we should be able to also use
> more concise SPI names (like name="whitespace").
> e.g.:
> {code:xml}
>
>
>
> />
>
>
>
> {code}
> would be
> {code:xml}
>
>
>
>
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8847) Code Cleanup: Rewrite StringBuilder.append with concatted strings
[ https://issues.apache.org/jira/browse/LUCENE-8847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860130#comment-16860130 ] Varun Thacker commented on LUCENE-8847: --- Thanks Uwe! > Code Cleanup: Rewrite StringBuilder.append with concatted strings > - > > Key: LUCENE-8847 > URL: https://issues.apache.org/jira/browse/LUCENE-8847 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Koen De Groote >Assignee: Uwe Schindler >Priority: Trivial > Fix For: master (9.0) > > > EDIT: Again, to clarify, please don't bother yourself with this ticket on > company time, on personal time you could be working on something that makes > you money or improves the product for your feature personally. > > This entire ticket is an afterthough. A look back at the code base that most > people don't have the time for. > - > > Code cleanup as suggested by static analysis tools. Will be done in my spare > time. > If someone reviews this, please also do not take up actual time from your > work to do that. I do not wish to take away from your working hours. > > These are simple, trivial things, that were probably overlooked or not even > considered(which isn't an accusation or something negative). But also stuff > that the Java compiler/JIT won't optimize on its own. > > That's what static analysis tool are good for: picking stuff like that up. > > I'm talking about Intellij's static code analysis. Facebook's "Infer" for > Java. Google's "errorprone", etc... > These are the kinds of things that, frankly, for the people actually working > on real features, are very time consuming, not even part of the feature, and > have a very low chance of actually turning up a real performance issue. > So I'm opting to have a look at the results of these tools and implementing > the sensible stuff and if something bigger pops up I'll make a separate > ticket for those things individually. > > Creating this ticket so I can name a branch after it. > > The only questions I have are: since the code base is so large, do I apply > each subject to all parts of it? Or only core? How do I split it up? > Do I make multiple PRs with this one ticket? Or do I make multiple tickets > and give each their own PR? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13533) Code Cleanup - Performance
[
https://issues.apache.org/jira/browse/SOLR-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859687#comment-16859687
]
Varun Thacker commented on SOLR-13533:
--
{quote}bq. The name's "Koen" by the way :P
{quote}
I'm going to blame myself for typing without my morning coffee :) Sorry about
that!
{quote}Which defeats the very purpose of using a StringBuilder.
{quote}
My personal take on this though is that unless they are showing up in a
profiler under real load fixing these isn't worth it. However looking at your
PR the changes make the code more readable as well . I'll leave comments on the
PR any changes are required
> Code Cleanup - Performance
> --
>
> Key: SOLR-13533
> URL: https://issues.apache.org/jira/browse/SOLR-13533
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Koen De Groote
>Priority: Trivial
> Labels: performance
>
> Code cleanup as suggested by static analysis tools. Will be done in my spare
> time.
> If someone reviews this, please also do not take up actual time from your
> work to do that. I do not wish to take away from your working hours.
>
> These are simple, trivial things, that were probably overlooked or not even
> considered(which isn't an accusation or something negative). But also stuff
> that the Java compiler/JIT won't optimize on its own.
>
> That's what static analysis tool are good for: picking stuff like that up.
>
> I'm talking about Intellij's static code analysis. Facebook's "Infer" for
> Java. Google's "errorprone", etc...
> These are the kinds of things that, frankly, for the people actually working
> on real features, are very time consuming, not even part of the feature, and
> have a very low chance of actually turning up a real performance issue.
> So I'm opting to have a look at the results of these tools and implementing
> the sensible stuff and if something bigger pops up I'll make a separate
> ticket for those things individually.
>
> Creating this ticket so I can name a branch after it.
>
> The only questions I have are: since the code base is so large, do I apply
> each subject to all parts of it? Or only core? How do I split it up?
> Do I make multiple PRs with this one ticket? Or do I make multiple tickets
> and give each their own PR?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13533) Code Cleanup - Performance
[ https://issues.apache.org/jira/browse/SOLR-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859503#comment-16859503 ] Varun Thacker commented on SOLR-13533: -- Hi Kevin, Splitting them up will always be easier for folks to review and the PRs won't get out of sync very quickly. You could split it up anyways you feel is best while working on it honestly - per error type that the tool catches or go module by module ( and split core by packages or something ) I think you can just create PRs and they get posted on the mailing list so we'll know about it. Additionally you could always link all the PRs here for reference as well > Code Cleanup - Performance > -- > > Key: SOLR-13533 > URL: https://issues.apache.org/jira/browse/SOLR-13533 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Koen De Groote >Priority: Trivial > Labels: performance > > Code cleanup as suggested by static analysis tools. Will be done in my spare > time. > If someone reviews this, please also do not take up actual time from your > work to do that. > > These are simple, trivial things, that were probably overlooked or not even > considered(which isn't an accusation or something negative). But also stuff > that the Java compiler/JIT won't optimize on its own. > > That's what static analysis tool are good for: picking stuff like that up. > > I'm talking about Intellij's static code analysis. Facebook's "Infer" for > Java. Google's "errorprone", etc... > These are the kinds of things that, frankly, for the people actually working > on real features, are very time consuming and have a very low chance of > actually turning up a real performance issue. > So I'm opting to have a look at the results of these tools and implementing > the sensible stuff and if something bigger pops up I'll make a separate > ticket for those things individually. > > Creating this ticket so I can name a branch after it. > > The only questions I have are: since the code base is so large, do I apply > each subject to all parts of it? Or only core? How do I split it up? > Do I make multiple PRs with this one ticket? Or do I make multiple tickets > and give each their own PR? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853404#comment-16853404
]
Varun Thacker commented on SOLR-9952:
-
{quote}Ok let me take my statement back. The problem is .
{quote}
I missed this statement so I think I understand how running HDFS and backing up
to S3 will work
What do you think the scope of this Jira should be ? Closed because the
functionality exists or should we still make an S3 backup repository which
takes the fs.s3a params and instead of using HdfsBackupRepository and
specifying ( Dsolr.hdfs.confdir=/etc/hadoop/conf ) and a core-site.xml where
fs.s3a params are specified
> S3BackupRepository
> --
>
> Key: SOLR-9952
> URL: https://issues.apache.org/jira/browse/SOLR-9952
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments:
> 0001-SOLR-9952-Added-dependencies-for-hadoop-amazon-integ.patch,
> 0002-SOLR-9952-Added-integration-test-for-checking-backup.patch, Running Solr
> on S3.pdf, core-site.xml.template
>
>
> I'd like to have a backup repository implementation allows to snapshot to AWS
> S3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852465#comment-16852465
]
Varun Thacker commented on SOLR-9952:
-
So the only namespace conflict for a user running hadoop and wanting to use S3
backups is the {{solr.hdfs.home}} param?
> S3BackupRepository
> --
>
> Key: SOLR-9952
> URL: https://issues.apache.org/jira/browse/SOLR-9952
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments:
> 0001-SOLR-9952-Added-dependencies-for-hadoop-amazon-integ.patch,
> 0002-SOLR-9952-Added-integration-test-for-checking-backup.patch, Running Solr
> on S3.pdf, core-site.xml.template
>
>
> I'd like to have a backup repository implementation allows to snapshot to AWS
> S3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852425#comment-16852425
]
Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:40 PM:
---
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
s3a://my-bucket/solr_811_backups
s3a://my-bucket/solr_811_backups
{code}
Manually added
[https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375]
and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP&name=testS3A&collection=test&repository=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://my-bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n", "code":500}} {code}
EDIT : I was able to get it to work when I ran this command before the backup -
{code:java}
aws s3api put-object --bucket my-bucket --key solr_811_backups{code}
Output of the backup
{code:java}
aws s3 ls s3:/my-bucket/solr_811_backups/testS3A/
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852425#comment-16852425
]
Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:39 PM:
---
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
s3a://my-bucket/solr_811_backups
s3a://my-bucket/solr_811_backups
{code}
Manually added
[https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375]
and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP&name=testS3A&collection=test&repository=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://my-bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n", "code":500}}{code}
Related code -
[https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L1063
]
EDIT : I was able to get it to work when I ran this command before the backup -
{code:java}
aws s3api put-objec
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852425#comment-16852425
]
Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:39 PM:
---
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
s3a://my-bucket/solr_811_backups
s3a://my-bucket/solr_811_backups
{code}
Manually added
[https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375]
and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP&name=testS3A&collection=test&repository=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://my-bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n", "code":500}}{code}
Related code -
[https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L1063
EDIT : I was able to get it to work when I ran this command before the backup -
{code:java}
aws s3api put-object --bu
[jira] [Commented] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852425#comment-16852425
]
Varun Thacker commented on SOLR-9952:
-
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
s3a://bucket/solr_811_backups
s3a://bucket/solr_811_backups
{code}
Manually added
https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375
and https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0 to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP&name=testS3A&collection=test&repository=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n",
"code":500}}{code}
Related code -
[https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L1063]
> S3BackupRepository
> --
>
> Key: SOLR-9952
> URL: https://issues.apache.org/jira/browse/SOLR-9952
> Project: Solr
>
[jira] [Comment Edited] (SOLR-13452) Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
[
https://issues.apache.org/jira/browse/SOLR-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845411#comment-16845411
]
Varun Thacker edited comment on SOLR-13452 at 5/22/19 1:39 AM:
---
When i run {{./gradlew runjflex}} StandardTokenizerImpl has a compile time
error - I believe we need to port this hack
[https://github.com/apache/lucene-solr/blob/c756b50ae4324b5692b2956c7a5d484122ac3016/lucene/common-build.xml#L2575]
over? I'll try to address this and post back.
I knew about this specifically because I recently ran into this while trying to
build a custom tokenizer
Just changing the skeleton.default file and removing the final modifier from
ZZ_BUFFERSIZE didn't work . It hadn't worked for me previously as well. I'll
look at it when I'm back in a few hours
was (Author: varunthacker):
When i run {{./gradlew runjflex}} StandardTokenizerImpl has a compile time
error - I believe we need to port this hack
[https://github.com/apache/lucene-solr/blob/c756b50ae4324b5692b2956c7a5d484122ac3016/lucene/common-build.xml#L2575]
over? I'll try to address this and post back.
I knew about this specifically because I recently ran into this while trying to
build a custom tokenizer
> Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
> -
>
> Key: SOLR-13452
> URL: https://issues.apache.org/jira/browse/SOLR-13452
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build
>Reporter: Mark Miller
>Priority: Major
>
> I took some things from the great work that Dat did in
> [https://github.com/apache/lucene-solr/tree/jira/gradle] and took the ball a
> little further.
>
> When working with gradle in sub modules directly, I recommend
> [https://github.com/dougborg/gdub]
> This gradle branch uses the following plugin for version locking, version
> configuration and version consistency across modules:
> [https://github.com/palantir/gradle-consistent-versions]
> By default, dependencies are not transitive, but there is a special
> Configuration for adding dependencies on other project internal modules that
> are transitive to their direct external dependencies (their jar libs).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13452) Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
[
https://issues.apache.org/jira/browse/SOLR-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845411#comment-16845411
]
Varun Thacker edited comment on SOLR-13452 at 5/22/19 1:31 AM:
---
When i run {{./gradlew runjflex}} StandardTokenizerImpl has a compile time
error - I believe we need to port this hack
[https://github.com/apache/lucene-solr/blob/c756b50ae4324b5692b2956c7a5d484122ac3016/lucene/common-build.xml#L2575]
over? I'll try to address this and post back.
I knew about this specifically because I recently ran into this while trying to
build a custom tokenizer
was (Author: varunthacker):
When i run {{./gradlew runjflex}} this the StandardTokenizerImpl has a compile
time error - I believe we need to port this hack
[https://github.com/apache/lucene-solr/blob/c756b50ae4324b5692b2956c7a5d484122ac3016/lucene/common-build.xml#L2575]
over? I'll try to address this and post back.
I knew about this specifically because I recently ran into this while trying to
build a custom tokenizer
> Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
> -
>
> Key: SOLR-13452
> URL: https://issues.apache.org/jira/browse/SOLR-13452
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build
>Reporter: Mark Miller
>Priority: Major
>
> I took some things from the great work that Dat did in
> [https://github.com/apache/lucene-solr/tree/jira/gradle] and took the ball a
> little further.
>
> When working with gradle in sub modules directly, I recommend
> [https://github.com/dougborg/gdub]
> This gradle branch uses the following plugin for version locking, version
> configuration and version consistency across modules:
> [https://github.com/palantir/gradle-consistent-versions]
> By default, dependencies are not transitive, but there is a special
> Configuration for adding dependencies on other project internal modules that
> are transitive to their direct external dependencies (their jar libs).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13452) Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
[
https://issues.apache.org/jira/browse/SOLR-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845411#comment-16845411
]
Varun Thacker commented on SOLR-13452:
--
When i run {{./gradlew runjflex}} this the StandardTokenizerImpl has a compile
time error - I believe we need to port this hack
[https://github.com/apache/lucene-solr/blob/c756b50ae4324b5692b2956c7a5d484122ac3016/lucene/common-build.xml#L2575]
over? I'll try to address this and post back.
I knew about this specifically because I recently ran into this while trying to
build a custom tokenizer
> Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
> -
>
> Key: SOLR-13452
> URL: https://issues.apache.org/jira/browse/SOLR-13452
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build
>Reporter: Mark Miller
>Priority: Major
>
> I took some things from the great work that Dat did in
> [https://github.com/apache/lucene-solr/tree/jira/gradle] and took the ball a
> little further.
>
> When working with gradle in sub modules directly, I recommend
> [https://github.com/dougborg/gdub]
> This gradle branch uses the following plugin for version locking, version
> configuration and version consistency across modules:
> [https://github.com/palantir/gradle-consistent-versions]
> By default, dependencies are not transitive, but there is a special
> Configuration for adding dependencies on other project internal modules that
> are transitive to their direct external dependencies (their jar libs).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12488) Rewrite exists field value query to leverage DocValuesFieldExistsQuery and NormsFieldExistsQuery
[ https://issues.apache.org/jira/browse/SOLR-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker resolved SOLR-12488. -- Resolution: Duplicate Indeed! I'll close this one out. > Rewrite exists field value query to leverage DocValuesFieldExistsQuery and > NormsFieldExistsQuery > > > Key: SOLR-12488 > URL: https://issues.apache.org/jira/browse/SOLR-12488 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: query parsers >Reporter: Varun Thacker >Priority: Major > > A few of us were discussing at Buzzwords on how a common use case requirement > is "match documents which have values for a field" > To do this we need to query "-fq:brand:*" . This query can be slow and can be > optimized > We can take advantage of NormsFieldExistsQuery and DocValuesFieldExistsQuery > to speed up this use-case and not have to resort to WildcardQuery > Today Solr's schema has doc-values enabled for fields that support it and for > text fields norms are enabled by default so most users would already have the > necessary indexed structures -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12823) remove clusterstate.json in Lucene/Solr 8.0
[ https://issues.apache.org/jira/browse/SOLR-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818656#comment-16818656 ] Varun Thacker commented on SOLR-12823: -- Hi Jan, Unfortunately I don't see myself working on this in the short term. > remove clusterstate.json in Lucene/Solr 8.0 > --- > > Key: SOLR-12823 > URL: https://issues.apache.org/jira/browse/SOLR-12823 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > > clusterstate.json is an artifact of a pre 5.0 Solr release. We should remove > that in 8.0 > It stays empty unless you explicitly ask to create the collection with the > old "stateFormat" and there is no reason for one to create a collection with > the old stateFormat. > We should also remove the "stateFormat" argument in create collection > We should also remove MIGRATESTATEVERSION as well > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12823) remove clusterstate.json in Lucene/Solr 8.0
[ https://issues.apache.org/jira/browse/SOLR-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-12823: - Priority: Major (was: Blocker) > remove clusterstate.json in Lucene/Solr 8.0 > --- > > Key: SOLR-12823 > URL: https://issues.apache.org/jira/browse/SOLR-12823 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > > clusterstate.json is an artifact of a pre 5.0 Solr release. We should remove > that in 8.0 > It stays empty unless you explicitly ask to create the collection with the > old "stateFormat" and there is no reason for one to create a collection with > the old stateFormat. > We should also remove the "stateFormat" argument in create collection > We should also remove MIGRATESTATEVERSION as well > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13288) Async logging max length should only apply to the message
Varun Thacker created SOLR-13288:
Summary: Async logging max length should only apply to the message
Key: SOLR-13288
URL: https://issues.apache.org/jira/browse/SOLR-13288
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Varun Thacker
Assignee: Varun Thacker
After SOLR-12753 messages are limited to 10240 chars. +1 to the limit, we even
hit this issue internally recently.
Sample log line
{code:java}
2019-03-03 19:04:51.293 INFO (qtp776700275-57) [c:gettingstarted s:shard2
r:core_node7 x:gettingstarted_shard2_replica_n4] o.a.s.c.S.Request
[gettingstarted_shard2_replica_n4] webapp=/solr path=/select
params={q=*:*&_=1551639889792} hits=0 status=0 QTime=206 } {code}
The way it's implemented currently though it picks the first 10240 chars from
the start. So let's say it was reduced to 10 the log line will look like
{code:java}
2019-03-03{code}
If we wrap the {{maxLen}} around the message part then we ensure some parts
are always captured. So with this pattern
{code:java}
%d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica}
%X{core}] %c{1.} %maxLen{%m %notEmpty{=>%ex{short}}}{10}} %n{code}
the message will now look like
{code:java}
2019-03-03 19:07:24.901 INFO (qtp776700275-57) [c:gettingstarted s:shard2
r:core_node7 x:gettingstarted_shard2_replica_n4] o.a.s.c.S.Request [gettingst}
{code}
This is still not perfect as ideally we'd want to capture the hits/status/QTime
part even if the message get's shortened. I'm not sure if the log4j2 Pattern
Layout syntax support it?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13288) Async logging max length should only apply to the message
[
https://issues.apache.org/jira/browse/SOLR-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-13288:
-
Description:
After SOLR-12753 messages are limited to 10240 chars. +1 for having a limit, we
even hit this issue internally recently.
Sample log line
{code:java}
2019-03-03 19:04:51.293 INFO (qtp776700275-57) [c:gettingstarted s:shard2
r:core_node7 x:gettingstarted_shard2_replica_n4] o.a.s.c.S.Request
[gettingstarted_shard2_replica_n4] webapp=/solr path=/select
params={q=*:*&_=1551639889792} hits=0 status=0 QTime=206 } {code}
The way it's implemented currently though it picks the first 10240 chars from
the start. So let's say it was reduced to 10 the log line will look like
{code:java}
2019-03-03{code}
If we wrap the {{maxLen}} around the message part then we ensure some parts
are always captured. So with this pattern
{code:java}
%d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica}
%X{core}] %c{1.} %maxLen{%m %notEmpty{=>%ex{short}}}{10}} %n{code}
the message will now look like
{code:java}
2019-03-03 19:07:24.901 INFO (qtp776700275-57) [c:gettingstarted s:shard2
r:core_node7 x:gettingstarted_shard2_replica_n4] o.a.s.c.S.Request [gettingst}
{code}
This is still not perfect as ideally we'd want to capture the hits/status/QTime
part even if the message get's shortened. I'm not sure if the log4j2 Pattern
Layout syntax support it?
was:
After SOLR-12753 messages are limited to 10240 chars. +1 to the limit, we even
hit this issue internally recently.
Sample log line
{code:java}
2019-03-03 19:04:51.293 INFO (qtp776700275-57) [c:gettingstarted s:shard2
r:core_node7 x:gettingstarted_shard2_replica_n4] o.a.s.c.S.Request
[gettingstarted_shard2_replica_n4] webapp=/solr path=/select
params={q=*:*&_=1551639889792} hits=0 status=0 QTime=206 } {code}
The way it's implemented currently though it picks the first 10240 chars from
the start. So let's say it was reduced to 10 the log line will look like
{code:java}
2019-03-03{code}
If we wrap the {{maxLen}} around the message part then we ensure some parts
are always captured. So with this pattern
{code:java}
%d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica}
%X{core}] %c{1.} %maxLen{%m %notEmpty{=>%ex{short}}}{10}} %n{code}
the message will now look like
{code:java}
2019-03-03 19:07:24.901 INFO (qtp776700275-57) [c:gettingstarted s:shard2
r:core_node7 x:gettingstarted_shard2_replica_n4] o.a.s.c.S.Request [gettingst}
{code}
This is still not perfect as ideally we'd want to capture the hits/status/QTime
part even if the message get's shortened. I'm not sure if the log4j2 Pattern
Layout syntax support it?
> Async logging max length should only apply to the message
> -
>
> Key: SOLR-13288
> URL: https://issues.apache.org/jira/browse/SOLR-13288
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
>
> After SOLR-12753 messages are limited to 10240 chars. +1 for having a limit,
> we even hit this issue internally recently.
>
> Sample log line
> {code:java}
> 2019-03-03 19:04:51.293 INFO (qtp776700275-57) [c:gettingstarted s:shard2
> r:core_node7 x:gettingstarted_shard2_replica_n4] o.a.s.c.S.Request
> [gettingstarted_shard2_replica_n4] webapp=/solr path=/select
> params={q=*:*&_=1551639889792} hits=0 status=0 QTime=206 } {code}
> The way it's implemented currently though it picks the first 10240 chars from
> the start. So let's say it was reduced to 10 the log line will look like
> {code:java}
> 2019-03-03{code}
> If we wrap the {{maxLen}} around the message part then we ensure some parts
> are always captured. So with this pattern
> {code:java}
> %d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica}
> %X{core}] %c{1.} %maxLen{%m %notEmpty{=>%ex{short}}}{10}} %n{code}
> the message will now look like
> {code:java}
> 2019-03-03 19:07:24.901 INFO (qtp776700275-57) [c:gettingstarted s:shard2
> r:core_node7 x:gettingstarted_shard2_replica_n4] o.a.s.c.S.Request
> [gettingst} {code}
> This is still not perfect as ideally we'd want to capture the
> hits/status/QTime part even if the message get's shortened. I'm not sure if
> the log4j2 Pattern Layout syntax support it?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13021) You cannot safely create and delete the same collection.
[ https://issues.apache.org/jira/browse/SOLR-13021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16747528#comment-16747528 ] Varun Thacker commented on SOLR-13021: -- I haven't tested SOLR-12818 since it was reported but maybe that still reproduces ? > You cannot safely create and delete the same collection. > > > Key: SOLR-13021 > URL: https://issues.apache.org/jira/browse/SOLR-13021 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12544) ZkStateReader can cache deleted collections and never refresh it
[
https://issues.apache.org/jira/browse/SOLR-12544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker resolved SOLR-12544.
--
Resolution: Fixed
I don't think so?I believe the user upgraded to a more recent version and
didn't run into this.
> ZkStateReader can cache deleted collections and never refresh it
>
>
> Key: SOLR-12544
> URL: https://issues.apache.org/jira/browse/SOLR-12544
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.2.1
>Reporter: Varun Thacker
>Priority: Major
>
> After a delete collection call, CLUSTERSTATUS starts breaking with this error
> permanently with this error
> {code:java}
> org.apache.solr.common.SolrException: Error loading config name for
> collection my_collection
> at
> org.apache.solr.common.cloud.ZkStateReader.readConfigName(ZkStateReader.java:198)
> at
> org.apache.solr.handler.admin.ClusterStatus.getClusterStatus(ClusterStatus.java:141)
> ...
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /collections/my_collection
> ...{code}
> SOLR-10720 addresses the problem by skipping over the collection as it was
> aimed to fix the race condition between delete collection and CLUSTERSTATUS
> being called.
>
> The fact that we see the error never go away , means there is another bug
> lingering which will make the state never refresh and thus other calls list
> LIST will always show the collection.
>
> This happened with Solr 7.2.1 and doesn't happen very often. But when it does
> the only solution is to restart the node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13000) log4j-slf4j-impl jar (inadvertently?) missing in solrj/lib
[
https://issues.apache.org/jira/browse/SOLR-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16696186#comment-16696186
]
Varun Thacker commented on SOLR-13000:
--
{quote}but for convenience an actual logging implementation is shipped with it?
{quote}
After SOLR-7887 we made it to "compile"
> log4j-slf4j-impl jar (inadvertently?) missing in solrj/lib
> --
>
> Key: SOLR-13000
> URL: https://issues.apache.org/jira/browse/SOLR-13000
> Project: Solr
> Issue Type: Bug
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-13000.patch
>
>
> My colleague [~ajhalani] noticed that declaring a {{solr-solrj}} dependency
> alone when configuring {{-Dlog4j.configurationFile}} for a project is not
> sufficient.
> I looked into further and it seems that the {{log4j-slf4j-impl}} jar is
> (inadvertently?) missing in {{solrj/lib}} in the release?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12632) Completely remove Trie fields
[
https://issues.apache.org/jira/browse/SOLR-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692678#comment-16692678
]
Varun Thacker commented on SOLR-12632:
--
I did some basic benchmarking on my laptop against branch 7_6. Index has 25M
documents in a two shard collection.
Both top_level_pi/one_level_pi have a 1M cardinality.
{code:java}
SolrInputDocument document = new SolrInputDocument();
document.addField("id", x*batchSize+j);
document.addField("top_level_pi", TestUtil.nextInt(r,0, 1000*1000));
document.addField("one_level_pi", TestUtil.nextInt(r,0, 1000*1000));
document.addField("top_level_ti", TestUtil.nextInt(r,0, 1000*1000));
document.addField("one_level_ti", TestUtil.nextInt(r,0, 1000*1000));
{code}
{code:java}
{code}
There were two types of queries that I ran against : one with counts and one
with count and a sum aggregation
The totalRows is 11
*Simple Count*
{code:java}
1.
time : 38-41s
filterCache inserts : 54303
facet(gettingstarted,q="id:123*",buckets="top_level_pi,one_level_pi",
bucketSorts="count(*) desc", bucketSizeLimit=-1, count(*))
2.
time : 6-9s
filterCache inserts : 54280
facet(gettingstarted,q="id:123*",buckets="top_level_ti,one_level_ti",
bucketSorts="count(*) desc", bucketSizeLimit=-1, count(*))
Underlying Facet Query that the stream expression forms
{
"top_level_ti": {
"type": "terms",
"field": "top_level_ti",
"limit": 2147483647,
"sort": {
"count": "desc"
},
"facet": {
"one_level_ti": {
"type": "terms",
"field": "one_level_ti",
"limit": 2147483647,
"sort": {
"count": "desc"
},
"facet": {}
}
}
}
{code}
*Count + Sum Aggregation*
{code:java}
3.
time : 110s
filterCache inserts : 110044
facet(gettingstarted,q="id:123*",buckets="top_level_pi,one_level_pi",
bucketSorts="count(*) desc", bucketSizeLimit=-1, count(*), sum(top_level_pi))
4.
time : 11-13s
filterCache inserts ; 110021
facet(gettingstarted,q="id:123*",buckets="top_level_ti,one_level_ti",
bucketSorts="count(*) desc", bucketSizeLimit=-1, count(*), sum(top_level_ti))
Underlying Facet Query that the stream expression forms
{
"top_level_pi": {
"type": "terms",
"field": "top_level_pi",
"limit": 2147483647,
"sort": {
"count": "desc"
},
"facet": {
"one_level_pi": {
"type": "terms",
"field": "one_level_pi",
"limit": 2147483647,
"sort": {
"count": "desc"
},
"facet": {
"facet_0": "sum(top_level_pi)"
}
}
}
}
}{code}
Few observations
# For the same query trie is a lot faster than point fields
# When you add a sum aggregation to the same facet query the time doubles
# The filter cache inserts are exteremly high. For nested facets with high
cardinality this will simply nuke the filer cache without much reuse
# For the same queries when I ran without the filter cache, only query 3 was
faster by 30-50% the rest were roughly the same time.
For reference , I ran the points query with java flight recorder and here's the
main stack trace from method profiling
{code:java}
org.apache.lucene.util.bkd.BKDReader$PackedIndexTree.pushLeft(BKDReader.java:410)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:786)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:797)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:797)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:797)
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
org.apache.lucene.search.PointRangeQuery$1$4.get(PointRangeQuery.java:299)
org.apache.lucene.search.PointRangeQuery$1.scorer(PointRangeQuery.java:323)
org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
org.apache.lucene.search.IndexOrDocValuesQuery$1.bulkScorer(IndexOrDocValuesQuery.java:138)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:667)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:151)
org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:140)
org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1178)
org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:1213)
org.apache.solr.
[jira] [Assigned] (SOLR-12632) Completely remove Trie fields
[ https://issues.apache.org/jira/browse/SOLR-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker reassigned SOLR-12632: Assignee: (was: Varun Thacker) > Completely remove Trie fields > - > > Key: SOLR-12632 > URL: https://issues.apache.org/jira/browse/SOLR-12632 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Steve Rowe >Priority: Blocker > Labels: numeric-tries-to-points > Fix For: master (8.0) > > > Trie fields were deprecated in Solr 7.0. We should remove them completely > before we release Solr 8.0. > Unresolved points-related issues: > [https://jira.apache.org/jira/issues/?jql=project=SOLR+AND+labels=numeric-tries-to-points+AND+resolution=unresolved] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-12632) Completely remove Trie fields
[ https://issues.apache.org/jira/browse/SOLR-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker reassigned SOLR-12632: Assignee: Varun Thacker > Completely remove Trie fields > - > > Key: SOLR-12632 > URL: https://issues.apache.org/jira/browse/SOLR-12632 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Steve Rowe >Assignee: Varun Thacker >Priority: Blocker > Labels: numeric-tries-to-points > Fix For: master (8.0) > > > Trie fields were deprecated in Solr 7.0. We should remove them completely > before we release Solr 8.0. > Unresolved points-related issues: > [https://jira.apache.org/jira/issues/?jql=project=SOLR+AND+labels=numeric-tries-to-points+AND+resolution=unresolved] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12998) Race condition between requery recovery and the core load completing
[
https://issues.apache.org/jira/browse/SOLR-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-12998:
-
Description:
I started Solr ( 7_6 branch ) using {{bin/solr start -e cloud -noprompt}} and
indexed 25M documents which amounts to roughly 1GB per replica in raw disk
space.
Whenever I restart the cluster, node2 ( 7574 ) get's these errors in the logs
which seems to indicate that request recovery is trying to find a core which
isn't completely loaded yet.
{code:java}
2018-11-18 21:06:09.845 INFO
(coreLoadExecutor-9-thread-2-processing-n:192.168.0.7:7574_solr)
[c:gettingstarted s:shard2 r:core_node8 x:gettingstarted_shard2_replica_n6]
o.a.s.c.SolrCore [[gettingstarted_shard2_replica_n6] ] Opening new SolrCore at
[/Users/varunthacker/apache-work/lucene-solr/solr/example/cloud/node2/solr/gettingstarted_shard2_replica_n6],
dataDir=[/Users/varunthacker/apache-work/lucene-solr/solr/example/cloud/node2/solr/gettingstarted_shard2_replica_n6/data/]
...
2018-11-18 21:06:10.501 INFO (qtp690521419-148) [
x:gettingstarted_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation It has been
requested that we recover: core=gettingstarted_shard2_replica_n6
...
2018-11-18 21:06:10.503 ERROR (qtp690521419-141) [
x:gettingstarted_shard1_replica_n1] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Unable to locate core
gettingstarted_shard1_replica_n1
at
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$5(CoreAdminOperation.java:167)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
...
2018-11-18 21:06:10.503 INFO (qtp690521419-148) [
x:gettingstarted_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null
path=/admin/cores
params={core=gettingstarted_shard2_replica_n6&action=REQUESTRECOVERY&wt=javabin&version=2}
status=400 QTime=3
... still the core is loading
2018-11-18 21:06:10.876 INFO
(coreLoadExecutor-9-thread-2-processing-n:192.168.0.7:7574_solr)
[c:gettingstarted s:shard2 r:core_node8 x:gettingstarted_shard2_replica_n6]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@2d17168a[gettingstarted_shard2_replica_n6] main]
... now done
2018-11-18 21:06:10.953 INFO
(searcherExecutor-10-thread-1-processing-n:192.168.0.7:7574_solr
x:gettingstarted_shard2_replica_n6 c:gettingstarted s:shard2 r:core_node8)
[c:gettingstarted s:shard2 r:core_node8 x:gettingstarted_shard2_replica_n6]
o.a.s.c.SolrCore [gettingstarted_shard2_replica_n6] Registered new searcher
Searcher@2d17168a[gettingstarted_shard2_replica_n6]
main{ExitableDirectoryReader(UninvertingDirectoryReader(_49(7.6.0):C2416916/4913:delGen=2
_a4(7.6.0):C2474322 _71(7.6.0):C2723220 _4k(7.6.0):c111317 _d7(7.6.0):C2501101
_bt(7.6.0):c91922 _di(7.6.0):c339286 _eb(7.6.0):c163244 _ds(7.6.0):c92780
_e1(7.6.0):c337329 _em(7.6.0):c360003 _ew(7.6.0):c263533 _fg(7.6.0):c226118
_f6(7.6.0):c266132 _es(7.6.0):C2361 _f8(7.6.0):C2251 _ff(7.6.0):C1821
_fh(7.6.0):C20079 _fi(7.6.0):C35733 _fj(7.6.0):C37203 _fk(7.6.0):C34780
_fl(7.6.0):C262 _fm(7.6.0):C2148 _fn(7.6.0):C2503))}{code}
was:
I started Solr ( 7_6 branch ) using {{bin/solr start -e cloud -noprompt}} and
indexed 25M documents which amounts to roughly 1GB per replica in raw disk
space.
Whenever I restart the cluster node2 ( 7574 ) get's these errors in the logs
which seem to indicate that request recovery is trying to find a core which
isn't completely loaded yet.
{code:java}
2018-11-18 21:06:09.845 INFO
(coreLoadExecutor-9-thread-2-processing-n:192.168.0.7:7574_solr)
[c:gettingstarted s:shard2 r:core_node8 x:gettingstarted_shard2_replica_n6]
o.a.s.c.SolrCore [[gettingstarted_shard2_replica_n6] ] Opening new SolrCore at
[/Users/varunthacker/apache-work/lucene-solr/solr/example/cloud/node2/solr/gettingstarted_shard2_replica_n6],
dataDir=[/Users/varunthacker/apache-work/lucene-solr/solr/example/cloud/node2/solr/gettingstarted_shard2_replica_n6/data/]
...
2018-11-18 21:06:10.501 INFO (qtp690521419-148) [
x:gettingstarted_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation It has been
requested that we recover: core=gettingstarted_shard2_replica_n6
...
2018-11-18 21:06:10.503 ERROR (qtp690521419-141) [
x:gettingstarted_shard1_replica_n1] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Unable to locate core
gettingstarted_shard1_replica_n1
at
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$5(CoreAdminOperation.java:167)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
at
org.apache.solr.handler.a
[jira] [Created] (SOLR-12998) Race condition between requery recovery and the core load completing
Varun Thacker created SOLR-12998:
Summary: Race condition between requery recovery and the core load
completing
Key: SOLR-12998
URL: https://issues.apache.org/jira/browse/SOLR-12998
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Varun Thacker
I started Solr ( 7_6 branch ) using {{bin/solr start -e cloud -noprompt}} and
indexed 25M documents which amounts to roughly 1GB per replica in raw disk
space.
Whenever I restart the cluster node2 ( 7574 ) get's these errors in the logs
which seem to indicate that request recovery is trying to find a core which
isn't completely loaded yet.
{code:java}
2018-11-18 21:06:09.845 INFO
(coreLoadExecutor-9-thread-2-processing-n:192.168.0.7:7574_solr)
[c:gettingstarted s:shard2 r:core_node8 x:gettingstarted_shard2_replica_n6]
o.a.s.c.SolrCore [[gettingstarted_shard2_replica_n6] ] Opening new SolrCore at
[/Users/varunthacker/apache-work/lucene-solr/solr/example/cloud/node2/solr/gettingstarted_shard2_replica_n6],
dataDir=[/Users/varunthacker/apache-work/lucene-solr/solr/example/cloud/node2/solr/gettingstarted_shard2_replica_n6/data/]
...
2018-11-18 21:06:10.501 INFO (qtp690521419-148) [
x:gettingstarted_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation It has been
requested that we recover: core=gettingstarted_shard2_replica_n6
...
2018-11-18 21:06:10.503 ERROR (qtp690521419-141) [
x:gettingstarted_shard1_replica_n1] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Unable to locate core
gettingstarted_shard1_replica_n1
at
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$5(CoreAdminOperation.java:167)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
...
2018-11-18 21:06:10.503 INFO (qtp690521419-148) [
x:gettingstarted_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null
path=/admin/cores
params={core=gettingstarted_shard2_replica_n6&action=REQUESTRECOVERY&wt=javabin&version=2}
status=400 QTime=3
... still the core is loading
2018-11-18 21:06:10.876 INFO
(coreLoadExecutor-9-thread-2-processing-n:192.168.0.7:7574_solr)
[c:gettingstarted s:shard2 r:core_node8 x:gettingstarted_shard2_replica_n6]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@2d17168a[gettingstarted_shard2_replica_n6] main]
... now done
2018-11-18 21:06:10.953 INFO
(searcherExecutor-10-thread-1-processing-n:192.168.0.7:7574_solr
x:gettingstarted_shard2_replica_n6 c:gettingstarted s:shard2 r:core_node8)
[c:gettingstarted s:shard2 r:core_node8 x:gettingstarted_shard2_replica_n6]
o.a.s.c.SolrCore [gettingstarted_shard2_replica_n6] Registered new searcher
Searcher@2d17168a[gettingstarted_shard2_replica_n6]
main{ExitableDirectoryReader(UninvertingDirectoryReader(_49(7.6.0):C2416916/4913:delGen=2
_a4(7.6.0):C2474322 _71(7.6.0):C2723220 _4k(7.6.0):c111317 _d7(7.6.0):C2501101
_bt(7.6.0):c91922 _di(7.6.0):c339286 _eb(7.6.0):c163244 _ds(7.6.0):c92780
_e1(7.6.0):c337329 _em(7.6.0):c360003 _ew(7.6.0):c263533 _fg(7.6.0):c226118
_f6(7.6.0):c266132 _es(7.6.0):C2361 _f8(7.6.0):C2251 _ff(7.6.0):C1821
_fh(7.6.0):C20079 _fi(7.6.0):C35733 _fj(7.6.0):C37203 _fk(7.6.0):C34780
_fl(7.6.0):C262 _fm(7.6.0):C2148 _fn(7.6.0):C2503))}{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12991) RecoveryStrategy should mention the root cause
Varun Thacker created SOLR-12991:
Summary: RecoveryStrategy should mention the root cause
Key: SOLR-12991
URL: https://issues.apache.org/jira/browse/SOLR-12991
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 7.5
Reporter: Varun Thacker
Here is a log line which could be more informative to help debug the root cause.
{code:java}
date time INFO (recoveryExecutor-4-thread-6-processing-n:host:port_solr
x:coll_shard1_replica_n203 c:coll s:shard1 r:core_node204) [c:coll s:shard1
r:core_node204 x:coll_shard1_replica_n203] o.a.s.c.RecoveryStrategy Failed to
connect leader https://host:8985/solr on recovery, try again{code}
In these code paths we should make sure the root cause is mentioned. Also these
should be WARNs or ERRORs
{code:java}
catch (IOException e) {
log.info("Failed to connect leader {} on recovery, try again",
leaderReplica.getBaseUrl());
Thread.sleep(500);
} catch (Exception e) {
if (e.getCause() instanceof IOException) {
log.info("Failed to connect leader {} on recovery, try again",
leaderReplica.getBaseUrl());
Thread.sleep(500);
} else {
return leaderReplica;
}
}{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12820) Auto pick method:dvhash based on thresholds
[
https://issues.apache.org/jira/browse/SOLR-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685527#comment-16685527
]
Varun Thacker commented on SOLR-12820:
--
In SOLR-12795 the method param is exposed so the user can at-least explicitly
chose dvhash. I think this Jira is still valid as we could try to pick the
default based on some heurestics
> Auto pick method:dvhash based on thresholds
> ---
>
> Key: SOLR-12820
> URL: https://issues.apache.org/jira/browse/SOLR-12820
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Facet Module
>Reporter: Varun Thacker
>Priority: Major
>
> I've worked with two users last week where explicitly using method:dvhash
> improved the faceting speeds drastically.
> The common theme in both the use-cases were: One collection hosting data for
> multiple users. We always filter documents for one user ( therby limiting
> the number of documents drastically ) and then perfoming a complex nested
> JSON facet.
> Both use-cases fit perfectly in this criteria that [~yo...@apache.org]
> mentioed on SOLR-9142
> {quote}faceting on a string field with a high cardinality compared to it's
> domain is less efficient than it could be.
> {quote}
> And DVHASH was the perfect optimization for these use-cases.
> We are using the facet stream expression in one of the use-cases which
> doesn't expose the method param. We could expose the method param to facet
> stream but I feel the better approach to solve this problem would be to
> address this TODO in the code withing the JSON Facet Module
> {code:java}
> if (mincount > 0 && prefix == null && (ntype != null || method ==
> FacetMethod.DVHASH)) {
> // TODO can we auto-pick for strings when term cardinality is much
> greater than DocSet cardinality?
> // or if we don't know cardinality but DocSet size is very small
> return new FacetFieldProcessorByHashDV(fcontext, this, sf);{code}
> I thought about this a little and this was the approach I am thinking
> currently to tackle this problem
> {code:java}
> int matchingDocs = fcontext.base.size();
> int totalDocs = fcontext.searcher.getIndexReader().maxDoc();
> //if matchingDocs is close to the totalDocs then we aren't filtering many
> documents.
> //that means the array approach would probably be better than the dvhash
> approach
> //Trying to find the cardinality for the matchingDocs would be expensive.
> //Also for totalDocs we don't have a global cardinality present at index time
> but we have a per segment cardinality
> //So using the number of matches as an alternate heuristic would do the job
> here?{code}
> Any thoughts if this approach makes sense? it could be I'm thinking of this
> approach just because both the users I worked with last week fell in this
> cateogory.
>
> cc [~dsmiley] [~joel.bernstein]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12911) Include "refine" param (refinement SOLR-9432) to the FacetStream
[
https://issues.apache.org/jira/browse/SOLR-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker resolved SOLR-12911.
--
Resolution: Fixed
Fix Version/s: master (8.0)
7.6
The work done for refine was merged as part of SOLR-12795 . Joel I'll mark this
issue as resolved
> Include "refine" param (refinement SOLR-9432) to the FacetStream
> -
>
> Key: SOLR-12911
> URL: https://issues.apache.org/jira/browse/SOLR-12911
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Amrit Sarkar
>Assignee: Joel Bernstein
>Priority: Major
> Fix For: 7.6, master (8.0)
>
> Attachments: SOLR-12911.patch, SOLR-12911.patch
>
>
> SOLR-9432 introduces refinement in JSON Faceting which makes sure all the
> respective buckets are fetched from each shard of the collection. This
> ensures authentic mathematical bucket counts.
> FacetStream should have refinement as an optional parameter like below, with
> default being "false":
> {code}
> facet(collection1,
> q="*:*",
> refine="true",
> buckets="a_s",
> bucketSorts="sum(a_i) desc",
> bucketSizeLimit=100,
> sum(a_i),
> count(*))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12795) Introduce 'rows' and 'offset' parameter in FacetStream
[ https://issues.apache.org/jira/browse/SOLR-12795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker resolved SOLR-12795. -- Resolution: Fixed Hi Joel, Marking this issue as fixed. > Introduce 'rows' and 'offset' parameter in FacetStream > -- > > Key: SOLR-12795 > URL: https://issues.apache.org/jira/browse/SOLR-12795 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: streaming expressions >Reporter: Amrit Sarkar >Assignee: Joel Bernstein >Priority: Major > Fix For: 7.6, master (8.0) > > Attachments: SOLR-12795.patch, SOLR-12795.patch, SOLR-12795.patch, > SOLR-12795.patch, SOLR-12795.patch, SOLR-12795.patch, SOLR-12795.patch, > SOLR-12795.patch, SOLR-12795.patch, SOLR-12795.patch, SOLR-12795.patch, > SOLR-12795.patch > > > Today facetStream takes a "bucketSizeLimit" parameter. Here is what the doc > says about this parameter - The number of buckets to include. This value is > applied to each dimension. > Now let's say we create a facet stream with 3 nested facets. For example > "year_i,month_i,day_i" and provide 10 as the bucketSizeLimit. > FacetStream would return 10 results to us for this facet expression while the > total number of unqiue values are 1000 (10*10*10 ) > The API should have a separate parameter "limit" which limits the number of > tuples (say 500) while bucketSizeLimit should be used to specify the size of > each bucket in the JSON Facet API. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12945) Cluster endpoint doesn't work
[
https://issues.apache.org/jira/browse/SOLR-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-12945:
-
Component/s: v2 API
> Cluster endpoint doesn't work
> -
>
> Key: SOLR-12945
> URL: https://issues.apache.org/jira/browse/SOLR-12945
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: v2 API
>Reporter: Varun Thacker
>Priority: Major
>
> I have a 2 node cluster and then when I hit this URL (
> [http://localhost:8983/api/cluster/collections] ) with Solr 7.5 I run into
> this error
> {code:java}
> 2018-10-31 00:03:42.565 ERROR (qtp690521419-111) [ ] o.a.s.a.V2HttpCall >>
> path: '/cluster/collections/test_empty'
> 2018-10-31 00:03:42.565 ERROR (qtp690521419-111) [ ] o.a.s.a.V2HttpCall
> Error in init()
> org.apache.solr.common.SolrException: no core retrieved for null
> at org.apache.solr.api.V2HttpCall.init(V2HttpCall.java:141)
> [solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
> - 2018-09-18 13:07:55]
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:469)
> [solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
> - 2018-09-18 13:07:55]
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
> [solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
> - 2018-09-18 13:07:55]
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
> [solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
> - 2018-09-18 13:07:55]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
> [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> [jetty-security-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> [jetty-rewrite-9.4.11.v20180605.jar:9.4.11.v20180605]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at org.eclipse.jetty.server.Server.handle(Server.java:531)
> [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
> at org.eclipse.jetty.server.HttpChannel.handle(HttpCha
[jira] [Created] (SOLR-12945) Cluster endpoint doesn't work
Varun Thacker created SOLR-12945:
Summary: Cluster endpoint doesn't work
Key: SOLR-12945
URL: https://issues.apache.org/jira/browse/SOLR-12945
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Varun Thacker
I have a 2 node cluster and then when I hit this URL (
[http://localhost:8983/api/cluster/collections] ) with Solr 7.5 I run into
this error
{code:java}
2018-10-31 00:03:42.565 ERROR (qtp690521419-111) [ ] o.a.s.a.V2HttpCall >>
path: '/cluster/collections/test_empty'
2018-10-31 00:03:42.565 ERROR (qtp690521419-111) [ ] o.a.s.a.V2HttpCall Error
in init()
org.apache.solr.common.SolrException: no core retrieved for null
at org.apache.solr.api.V2HttpCall.init(V2HttpCall.java:141)
[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
2018-09-18 13:07:55]
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:469)
[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
2018-09-18 13:07:55]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
2018-09-18 13:07:55]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
2018-09-18 13:07:55]
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
[jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
[jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
[jetty-security-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
[jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
[jetty-rewrite-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at org.eclipse.jetty.server.Server.handle(Server.java:531)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
[jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
[jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]
at org.eclipse.jetty.io.FillInterest.fil
[jira] [Comment Edited] (SOLR-12866) Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore failures
[
https://issues.apache.org/jira/browse/SOLR-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669415#comment-16669415
]
Varun Thacker edited comment on SOLR-12866 at 10/30/18 11:23 PM:
-
Here is a patch which tries to write a mock testing testing replica asssingment
under the following conditions -
# We have created a collection with createNodeSet=EMPTY , so clusterstate has
already been created but no replicas are present ( testEmptyCollection.json )
# Create one replica for all 3 shards of the collection on a 2 node cluster
I want to test if the assignment engine places all of them on only one node.
This is what I am seeing in RestoreCmd as posted on my earlier comment so I
want to see where are we going wrong.
Unfortunately as of now, I'm stuck in getting the mock to work. When i run the
test I get this error
{code:java}
16:06:45.053
[TEST-TestPolicy.testPolicyForEmptyCollection-seed#[33EAB712F76AB404]] ERROR
org.apache.solr.client.solrj.cloud.autoscaling.Policy - Exception! prefs = [{
"minimize":"cores",
"precision":1}, {"maximize":"freedisk"}], recent r1 = node2, r2 = node1,
matrix = 2
...
Caused by: org.apache.solr.common.SolrException
at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.setApproxValuesAndSortNodes(Policy.java:314)
at
org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.applyRules(Policy.java:606)
...{code}
The test doesn't like the fact that I have "replicaInfo" as empty. So the
preferences sort algorithm runs into a NullPointerException and throws the
RuntimeException.
was (Author: varunthacker):
Here is a patch which tries to write a mock testing testing replica asssingment
under the following conditions -
# We have created a collection with createNodeSet=EMPTY , so clusterstate has
already been created but no replicas are present ( testEmptyCollection.json )
# Create one replica for all 3 shards of the collection on a 2 node cluster
I want to test if the assignment engine places all of them on only one node.
This is what I am seeing in RestoreCmd as posted on my earlier comment so I
want to see where are we going wrong.
Unfortunately as of now, I'm stuck in getting the mock to work. When i run the
test I get this error
{code:java}
16:06:45.053
[TEST-TestPolicy.testPolicyForEmptyCollection-seed#[33EAB712F76AB404]] ERROR
org.apache.solr.client.solrj.cloud.autoscaling.Policy - Exception! prefs = [{
"minimize":"cores",
"precision":1}, {"maximize":"freedisk"}], recent r1 = node2, r2 = node1,
matrix = 2
...
Caused by: org.apache.solr.common.SolrException
at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.setApproxValuesAndSortNodes(Policy.java:314)
at
org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.applyRules(Policy.java:606)
...{code}
> Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore
> failures
> -
>
> Key: SOLR-12866
> URL: https://issues.apache.org/jira/browse/SOLR-12866
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Steve Rowe
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12866.patch
>
>
> From [https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/185/],
> both tests failed 10/10 iterations for me on branch_7x with the seed:
> {noformat}
> Checking out Revision 37fdcb02d87ec44293ec4942c75a3cb709c45418
> (refs/remotes/origin/branch_7x)
> [...]
>[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLocalFSCloudBackupRestore -Dtests.method=test
> -Dtests.seed=3CD4284489C09DB4 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=mk-MK
> -Dtests.timezone=Pacific/Kiritimati -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 10.8s J2 | TestLocalFSCloudBackupRestore.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Node
> 127.0.0.1:43864_solr has 3 replicas. Expected num replicas : 2. state:
>[junit4]>
> DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/9)={
>[junit4]> "pullReplicas":0,
>[junit4]> "replicationFactor":1,
>[junit4]> "shards":{
>[junit4]> "shard2":{
>[junit4]> "range":"0-7fff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node62":{
>[junit4]> "core":"backuprestore_restored_shard2_replica_n61",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type
[jira] [Comment Edited] (SOLR-12866) Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore failures
[
https://issues.apache.org/jira/browse/SOLR-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669415#comment-16669415
]
Varun Thacker edited comment on SOLR-12866 at 10/30/18 11:07 PM:
-
Here is a patch which tries to write a mock testing testing replica asssingment
under the following conditions -
# We have created a collection with createNodeSet=EMPTY , so clusterstate has
already been created but no replicas are present ( testEmptyCollection.json )
# Create one replica for all 3 shards of the collection on a 2 node cluster
I want to test if the assignment engine places all of them on only one node.
This is what I am seeing in RestoreCmd as posted on my earlier comment so I
want to see where are we going wrong.
Unfortunately as of now, I'm stuck in getting the mock to work. When i run the
test I get this error
{code:java}
16:06:45.053
[TEST-TestPolicy.testPolicyForEmptyCollection-seed#[33EAB712F76AB404]] ERROR
org.apache.solr.client.solrj.cloud.autoscaling.Policy - Exception! prefs = [{
"minimize":"cores",
"precision":1}, {"maximize":"freedisk"}], recent r1 = node2, r2 = node1,
matrix = 2
...
Caused by: org.apache.solr.common.SolrException
at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.setApproxValuesAndSortNodes(Policy.java:314)
at
org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.applyRules(Policy.java:606)
...{code}
was (Author: varunthacker):
Here is a patch which tries to write a mock testing testing replica asssingment
under the following conditions -
# We have created a collection with createNodeSet=EMPTY , so clusterstate has
already been created but no replicas are present ( testEmptyCollection.json )
# Create one replica for all 3 shards of the collection on a 2 node cluster
I want to test if the assignment engine places all of them on only one node.
This is what I am seeing in RestoreCmd as posted on my earlier comment so I
want to see where are we going wrong.
> Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore
> failures
> -
>
> Key: SOLR-12866
> URL: https://issues.apache.org/jira/browse/SOLR-12866
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Steve Rowe
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12866.patch
>
>
> From [https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/185/],
> both tests failed 10/10 iterations for me on branch_7x with the seed:
> {noformat}
> Checking out Revision 37fdcb02d87ec44293ec4942c75a3cb709c45418
> (refs/remotes/origin/branch_7x)
> [...]
>[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLocalFSCloudBackupRestore -Dtests.method=test
> -Dtests.seed=3CD4284489C09DB4 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=mk-MK
> -Dtests.timezone=Pacific/Kiritimati -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 10.8s J2 | TestLocalFSCloudBackupRestore.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Node
> 127.0.0.1:43864_solr has 3 replicas. Expected num replicas : 2. state:
>[junit4]>
> DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/9)={
>[junit4]> "pullReplicas":0,
>[junit4]> "replicationFactor":1,
>[junit4]> "shards":{
>[junit4]> "shard2":{
>[junit4]> "range":"0-7fff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node62":{
>[junit4]> "core":"backuprestore_restored_shard2_replica_n61",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266853250"},
>[junit4]> "shard1_1":{
>[junit4]> "range":"c000-",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node64":{
>[junit4]>
> "core":"backuprestore_restored_shard1_1_replica_n63",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"15394597032
[jira] [Commented] (SOLR-12866) Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore failures
[
https://issues.apache.org/jira/browse/SOLR-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669415#comment-16669415
]
Varun Thacker commented on SOLR-12866:
--
Here is a patch which tries to write a mock testing testing replica asssingment
under the following conditions -
# We have created a collection with createNodeSet=EMPTY , so clusterstate has
already been created but no replicas are present ( testEmptyCollection.json )
# Create one replica for all 3 shards of the collection on a 2 node cluster
I want to test if the assignment engine places all of them on only one node.
This is what I am seeing in RestoreCmd as posted on my earlier comment so I
want to see where are we going wrong.
> Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore
> failures
> -
>
> Key: SOLR-12866
> URL: https://issues.apache.org/jira/browse/SOLR-12866
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Steve Rowe
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12866.patch
>
>
> From [https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/185/],
> both tests failed 10/10 iterations for me on branch_7x with the seed:
> {noformat}
> Checking out Revision 37fdcb02d87ec44293ec4942c75a3cb709c45418
> (refs/remotes/origin/branch_7x)
> [...]
>[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLocalFSCloudBackupRestore -Dtests.method=test
> -Dtests.seed=3CD4284489C09DB4 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=mk-MK
> -Dtests.timezone=Pacific/Kiritimati -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 10.8s J2 | TestLocalFSCloudBackupRestore.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Node
> 127.0.0.1:43864_solr has 3 replicas. Expected num replicas : 2. state:
>[junit4]>
> DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/9)={
>[junit4]> "pullReplicas":0,
>[junit4]> "replicationFactor":1,
>[junit4]> "shards":{
>[junit4]> "shard2":{
>[junit4]> "range":"0-7fff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node62":{
>[junit4]> "core":"backuprestore_restored_shard2_replica_n61",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266853250"},
>[junit4]> "shard1_1":{
>[junit4]> "range":"c000-",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node64":{
>[junit4]>
> "core":"backuprestore_restored_shard1_1_replica_n63",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266887720"},
>[junit4]> "shard1_0":{
>[junit4]> "range":"8000-bfff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node66":{
>[junit4]>
> "core":"backuprestore_restored_shard1_0_replica_n65",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266910800"}},
>[junit4]> "router":{
>[junit4]> "name":"compositeId",
>[junit4]> "field":"shard_s"},
>[junit4]> "maxShardsPerNode":"-1",
>[junit4]> "autoAddReplicas":"false",
>[junit4]> "nrtReplicas":1,
>[junit4]> "tlogReplicas":0}
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([3CD4284489C09DB4:B480179E273CF04C]:0)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
>[junit4]> at java.util.HashM
[jira] [Updated] (SOLR-12866) Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore failures
[
https://issues.apache.org/jira/browse/SOLR-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-12866:
-
Attachment: SOLR-12866.patch
> Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore
> failures
> -
>
> Key: SOLR-12866
> URL: https://issues.apache.org/jira/browse/SOLR-12866
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Steve Rowe
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12866.patch
>
>
> From [https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/185/],
> both tests failed 10/10 iterations for me on branch_7x with the seed:
> {noformat}
> Checking out Revision 37fdcb02d87ec44293ec4942c75a3cb709c45418
> (refs/remotes/origin/branch_7x)
> [...]
>[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLocalFSCloudBackupRestore -Dtests.method=test
> -Dtests.seed=3CD4284489C09DB4 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=mk-MK
> -Dtests.timezone=Pacific/Kiritimati -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 10.8s J2 | TestLocalFSCloudBackupRestore.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Node
> 127.0.0.1:43864_solr has 3 replicas. Expected num replicas : 2. state:
>[junit4]>
> DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/9)={
>[junit4]> "pullReplicas":0,
>[junit4]> "replicationFactor":1,
>[junit4]> "shards":{
>[junit4]> "shard2":{
>[junit4]> "range":"0-7fff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node62":{
>[junit4]> "core":"backuprestore_restored_shard2_replica_n61",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266853250"},
>[junit4]> "shard1_1":{
>[junit4]> "range":"c000-",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node64":{
>[junit4]>
> "core":"backuprestore_restored_shard1_1_replica_n63",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266887720"},
>[junit4]> "shard1_0":{
>[junit4]> "range":"8000-bfff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node66":{
>[junit4]>
> "core":"backuprestore_restored_shard1_0_replica_n65",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266910800"}},
>[junit4]> "router":{
>[junit4]> "name":"compositeId",
>[junit4]> "field":"shard_s"},
>[junit4]> "maxShardsPerNode":"-1",
>[junit4]> "autoAddReplicas":"false",
>[junit4]> "nrtReplicas":1,
>[junit4]> "tlogReplicas":0}
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([3CD4284489C09DB4:B480179E273CF04C]:0)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
>[junit4]> at java.util.HashMap.forEach(HashMap.java:1289)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
>[junit4]> at
> org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
>[junit4]> at java.lang.Thread.run(Thread.java:748)
> {noform
[jira] [Created] (SOLR-12944) Bugs around createNodeSet=EMPTY
Varun Thacker created SOLR-12944:
Summary: Bugs around createNodeSet=EMPTY
Key: SOLR-12944
URL: https://issues.apache.org/jira/browse/SOLR-12944
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Varun Thacker
Firstly, As of today we cannot create an empty collection from SolrJ
I have a two node cluster and this API call fails with an error
{code:java}
//Create a coreless collection of 3 shards
CollectionAdminRequest.Create create = CollectionAdminRequest
.createCollection("test_collection", "conf1", 3, 0)
.setCreateNodeSet(OverseerCollectionMessageHandler.CREATE_NODE_SET_EMPTY)
.setMaxShardsPerNode(-1);{code}
Secondly if I use the API directly ,
{{[http://localhost:8983/solr/admin/collections?action=create&name=test_coll&numShards=5&createNodeSet=EMPTY]
}}the state.json has replicationFactor = nrtReplicas = 1 instead of 0
{code:java}
"test_coll":{
"pullReplicas":"0",
"replicationFactor":"1",
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":"1",
"tlogReplicas":"0",
"shards":{
"shard1":{
"range":"8000-b332",
"state":"active",
"replicas":{}},
"shard2":{{code}
On a side note , I think we should rethink how
replicationFactor/nrtReplicas/tlogReplicas/pullReplicas are stored.
These values could be stored at a per shard level such that adding a replica
will actually refelct the total replication count
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12866) Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore failures
[
https://issues.apache.org/jira/browse/SOLR-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1238#comment-1238
]
Varun Thacker commented on SOLR-12866:
--
This is a snippet of the RestoreCmd command that does the replica placement
{code:java}
Assign.AssignRequest assignRequest = new Assign.AssignRequestBuilder()
.forCollection(restoreCollectionName)
.forShard(sliceNames)
.assignNrtReplicas(numNrtReplicas)
.assignTlogReplicas(numTlogReplicas)
.assignPullReplicas(numPullReplicas)
.onNodes(nodeList)
.build();
Assign.AssignStrategyFactory assignStrategyFactory = new
Assign.AssignStrategyFactory(ocmh.cloudManager);
Assign.AssignStrategy assignStrategy =
assignStrategyFactory.create(clusterState, restoreCollection);
List replicaPositions =
assignStrategy.assign(ocmh.cloudManager, assignRequest);
sessionWrapper = PolicyHelper.getLastSessionWrapper(true);{code}
Now we have two nodes and 3 shards to create and this ends up assigning all of
them to one node and hence the test fails.
I'm trying to isolate this in a mock and test more . For reference when I
stepped through this code in debug mode here is what i saw
{code:java}
result = {Assign$AssignRequest@6811}
collectionName = "backuprestore_restored"
shardNames = {ArrayList@6741} size = 3
nodes = {ArrayList@6719} size = 2
numNrtReplicas = 1
numTlogReplicas = 0
numPullReplicas = 0
replicaPositions = {ArrayList@6785} size = 3
0 = {ReplicaPosition@6793} "shard2:1[NRT] @127.0.0.1:61176_solr"
1 = {ReplicaPosition@6794} "shard1_1:1[NRT] @127.0.0.1:61176_solr"
2 = {ReplicaPosition@6795} "shard1_0:1[NRT] @127.0.0.1:61176_solr"{code}
> Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore
> failures
> -
>
> Key: SOLR-12866
> URL: https://issues.apache.org/jira/browse/SOLR-12866
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Steve Rowe
>Assignee: Varun Thacker
>Priority: Major
>
> From [https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/185/],
> both tests failed 10/10 iterations for me on branch_7x with the seed:
> {noformat}
> Checking out Revision 37fdcb02d87ec44293ec4942c75a3cb709c45418
> (refs/remotes/origin/branch_7x)
> [...]
>[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLocalFSCloudBackupRestore -Dtests.method=test
> -Dtests.seed=3CD4284489C09DB4 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=mk-MK
> -Dtests.timezone=Pacific/Kiritimati -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 10.8s J2 | TestLocalFSCloudBackupRestore.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Node
> 127.0.0.1:43864_solr has 3 replicas. Expected num replicas : 2. state:
>[junit4]>
> DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/9)={
>[junit4]> "pullReplicas":0,
>[junit4]> "replicationFactor":1,
>[junit4]> "shards":{
>[junit4]> "shard2":{
>[junit4]> "range":"0-7fff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node62":{
>[junit4]> "core":"backuprestore_restored_shard2_replica_n61",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266853250"},
>[junit4]> "shard1_1":{
>[junit4]> "range":"c000-",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node64":{
>[junit4]>
> "core":"backuprestore_restored_shard1_1_replica_n63",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266887720"},
>[junit4]> "shard1_0":{
>[junit4]> "range":"8000-bfff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node66":{
>[junit4]>
> "core":"backuprestore_restored_shard1_0_replica_n65",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.
[jira] [Commented] (SOLR-12866) Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore failures
[
https://issues.apache.org/jira/browse/SOLR-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664648#comment-16664648
]
Varun Thacker commented on SOLR-12866:
--
I'll work on this in the next couple of days! Sorry it took longer for me to
get to this.
> Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore
> failures
> -
>
> Key: SOLR-12866
> URL: https://issues.apache.org/jira/browse/SOLR-12866
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Steve Rowe
>Assignee: Varun Thacker
>Priority: Major
>
> From [https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/185/],
> both tests failed 10/10 iterations for me on branch_7x with the seed:
> {noformat}
> Checking out Revision 37fdcb02d87ec44293ec4942c75a3cb709c45418
> (refs/remotes/origin/branch_7x)
> [...]
>[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLocalFSCloudBackupRestore -Dtests.method=test
> -Dtests.seed=3CD4284489C09DB4 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=mk-MK
> -Dtests.timezone=Pacific/Kiritimati -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 10.8s J2 | TestLocalFSCloudBackupRestore.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Node
> 127.0.0.1:43864_solr has 3 replicas. Expected num replicas : 2. state:
>[junit4]>
> DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/9)={
>[junit4]> "pullReplicas":0,
>[junit4]> "replicationFactor":1,
>[junit4]> "shards":{
>[junit4]> "shard2":{
>[junit4]> "range":"0-7fff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node62":{
>[junit4]> "core":"backuprestore_restored_shard2_replica_n61",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266853250"},
>[junit4]> "shard1_1":{
>[junit4]> "range":"c000-",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node64":{
>[junit4]>
> "core":"backuprestore_restored_shard1_1_replica_n63",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266887720"},
>[junit4]> "shard1_0":{
>[junit4]> "range":"8000-bfff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node66":{
>[junit4]>
> "core":"backuprestore_restored_shard1_0_replica_n65",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266910800"}},
>[junit4]> "router":{
>[junit4]> "name":"compositeId",
>[junit4]> "field":"shard_s"},
>[junit4]> "maxShardsPerNode":"-1",
>[junit4]> "autoAddReplicas":"false",
>[junit4]> "nrtReplicas":1,
>[junit4]> "tlogReplicas":0}
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([3CD4284489C09DB4:B480179E273CF04C]:0)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
>[junit4]> at java.util.HashMap.forEach(HashMap.java:1289)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
>[junit4]> at
> org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
[jira] [Assigned] (SOLR-12866) Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore failures
[
https://issues.apache.org/jira/browse/SOLR-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker reassigned SOLR-12866:
Assignee: Varun Thacker
> Reproducing TestLocalFSCloudBackupRestore and TestHdfsCloudBackupRestore
> failures
> -
>
> Key: SOLR-12866
> URL: https://issues.apache.org/jira/browse/SOLR-12866
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Steve Rowe
>Assignee: Varun Thacker
>Priority: Major
>
> From [https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/185/],
> both tests failed 10/10 iterations for me on branch_7x with the seed:
> {noformat}
> Checking out Revision 37fdcb02d87ec44293ec4942c75a3cb709c45418
> (refs/remotes/origin/branch_7x)
> [...]
>[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLocalFSCloudBackupRestore -Dtests.method=test
> -Dtests.seed=3CD4284489C09DB4 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=mk-MK
> -Dtests.timezone=Pacific/Kiritimati -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 10.8s J2 | TestLocalFSCloudBackupRestore.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Node
> 127.0.0.1:43864_solr has 3 replicas. Expected num replicas : 2. state:
>[junit4]>
> DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/9)={
>[junit4]> "pullReplicas":0,
>[junit4]> "replicationFactor":1,
>[junit4]> "shards":{
>[junit4]> "shard2":{
>[junit4]> "range":"0-7fff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node62":{
>[junit4]> "core":"backuprestore_restored_shard2_replica_n61",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266853250"},
>[junit4]> "shard1_1":{
>[junit4]> "range":"c000-",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node64":{
>[junit4]>
> "core":"backuprestore_restored_shard1_1_replica_n63",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266887720"},
>[junit4]> "shard1_0":{
>[junit4]> "range":"8000-bfff",
>[junit4]> "state":"active",
>[junit4]> "replicas":{"core_node66":{
>[junit4]>
> "core":"backuprestore_restored_shard1_0_replica_n65",
>[junit4]> "base_url":"https://127.0.0.1:43864/solr";,
>[junit4]> "node_name":"127.0.0.1:43864_solr",
>[junit4]> "state":"active",
>[junit4]> "type":"NRT",
>[junit4]> "force_set_state":"false",
>[junit4]> "leader":"true"}},
>[junit4]> "stateTimestamp":"1539459703266910800"}},
>[junit4]> "router":{
>[junit4]> "name":"compositeId",
>[junit4]> "field":"shard_s"},
>[junit4]> "maxShardsPerNode":"-1",
>[junit4]> "autoAddReplicas":"false",
>[junit4]> "nrtReplicas":1,
>[junit4]> "tlogReplicas":0}
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([3CD4284489C09DB4:B480179E273CF04C]:0)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
>[junit4]> at java.util.HashMap.forEach(HashMap.java:1289)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
>[junit4]> at
> org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
>[junit4]> at
> org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
>[junit4]> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> {noformat}
>[junit4] 2> NOTE
[jira] [Commented] (SOLR-12931) Move Solr's ExitableDirectoryReader test to it's own package
[ https://issues.apache.org/jira/browse/SOLR-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664477#comment-16664477 ] Varun Thacker commented on SOLR-12931: -- Fair enough. Move all 3 classes to - org.apache.solr.search ? > Move Solr's ExitableDirectoryReader test to it's own package > > > Key: SOLR-12931 > URL: https://issues.apache.org/jira/browse/SOLR-12931 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12931) Move Solr's ExitableDirectoryReader test to it's own package
[ https://issues.apache.org/jira/browse/SOLR-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664399#comment-16664399 ] Varun Thacker commented on SOLR-12931: -- Nothing else would be added in the future , but it helps new devs understand where a feature test lies ? Today the same feature's test is spread across three packages. Do you think isolating packges for dedicated features is a bad idea? Overkill? > Move Solr's ExitableDirectoryReader test to it's own package > > > Key: SOLR-12931 > URL: https://issues.apache.org/jira/browse/SOLR-12931 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12931) Move Solr's ExitableDirectoryReader test to it's own package
[ https://issues.apache.org/jira/browse/SOLR-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664371#comment-16664371 ] Varun Thacker commented on SOLR-12931: -- Context to the quesion : I started looking at SOLR-12906 and DelayingSearchComponent caught my eye as it was under the ( org.apache.solr.search ) package. After spending a couple of minutes I figured this has to do with tests around ExitableDirectoryReader Let's use ExitableDirectoryReader as the feature in context that we wanted to test - We wrote a test for single core and then a cloud test. I think this is a typical pattern where the single core test can be more exhaustive, but we want to write a cloud test to make sure it works in the distributed case as well Both are integration tests. DelayingSearchComponent is the helper class and in the ( org.apache.solr.search ) package CloudExitableDirectoryReaderTest is the test that uses this and is in the ( org.apache.solr.cloud ) package ExitableDirectoryReaderTest is the single core test for this in the ( org.apache.solr.core ) package Proposing that we move these three classes to it's own package as a first pass. When we make a clear distinction of mock vs integration tests we could move this package under integration > Move Solr's ExitableDirectoryReader test to it's own package > > > Key: SOLR-12931 > URL: https://issues.apache.org/jira/browse/SOLR-12931 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12931) Move Solr'ExitableDirectoryReader
Varun Thacker created SOLR-12931: Summary: Move Solr'ExitableDirectoryReader Key: SOLR-12931 URL: https://issues.apache.org/jira/browse/SOLR-12931 Project: Solr Issue Type: Test Security Level: Public (Default Security Level. Issues are Public) Reporter: Varun Thacker -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12931) Move Solr's ExitableDirectoryReader test to it's own package
[ https://issues.apache.org/jira/browse/SOLR-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-12931: - Summary: Move Solr's ExitableDirectoryReader test to it's own package (was: Move Solr'ExitableDirectoryReader ) > Move Solr's ExitableDirectoryReader test to it's own package > > > Key: SOLR-12931 > URL: https://issues.apache.org/jira/browse/SOLR-12931 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
[
https://issues.apache.org/jira/browse/SOLR-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664330#comment-16664330
]
Varun Thacker commented on SOLR-12057:
--
Hi Amrit,
Some feedback on
CdcrUpdateProcessor
* Can we add some javadocs as to what this update processor wants to achieve?
* Do we still need to override versionAdd / versionDelete versionDeleteByQuery
?
* It would be nice to add some basic docs to the {{filterParams}} method to
indicate what it's trying to filter etc.
On CdcrReplicaTypesTest
* {{//.withProperty("solr.directoryFactory",
"solr.StandardDirectoryFactory")}} - Can we remove this comment?
* Is {{testTlogReplica}} meant to only have tlog replicas? The create
collection uses a combination of nrtReplicas and tlogReplicas so I'm trying to
understand the motivation here
* "Not really, we can remove this safely, from, all tests; 2 sec sleep is for
loading the Cdcr components and avoiding potentially few retries." - You
mentioned this but the patch still has a 2s delay
* {{int batchSize = (TEST_NIGHTLY ? 100 : 10);}} - does batchSize represent
numBatches? 100 seems to be the batch size in the inner loop
>From a design perspective :
Given the improvements you've made with the patch , are we in a position to
roll up this block from CdcrUpdateProcessor into DistributedUpdateProcessor ?
If yes then we would get CDCR to work even without them having to add an
UpdateProcessor ? We coiuld keep CdcrUpdateProcessor as is for backward compat
but remove references of it from the docs
{code:java}
if (params.get(CDCR_UPDATE) != null) {
result.set(CDCR_UPDATE, "");
result.set(CommonParams.VERSION_FIELD,
params.get(CommonParams.VERSION_FIELD));
}{code}
> CDCR does not replicate to Collections with TLOG Replicas
> -
>
> Key: SOLR-12057
> URL: https://issues.apache.org/jira/browse/SOLR-12057
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.2
>Reporter: Webster Homer
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12057.patch, SOLR-12057.patch, SOLR-12057.patch,
> SOLR-12057.patch, SOLR-12057.patch, cdcr-fail-with-tlog-pull.patch,
> cdcr-fail-with-tlog-pull.patch
>
>
> We created a collection using TLOG replicas in our QA clouds.
> We have a locally hosted solrcloud with 2 nodes, all our collections have 2
> shards. We use CDCR to replicate the collections from this environment to 2
> data centers hosted in Google cloud. This seems to work fairly well for our
> collections with NRT replicas. However the new TLOG collection has problems.
>
> The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
> shards per collection with 2 replicas per shard.
>
> We never see data show up in the cloud collections, but we do see tlog files
> show up on the cloud servers. I can see that all of the servers have cdcr
> started, buffers are disabled.
> The cdcr source configuration is:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"},
> {
>
> "zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"}],
> "replicator":{
> "threadPoolSize":4,
> "schedule":500,
> "batchSize":250},
> "updateLogSynchronizer":\{"schedule":6
>
> The target configurations in the 2 clouds are the same:
> "requestHandler":{"/cdcr":{ "name":"/cdcr",
> "class":"solr.CdcrRequestHandler", "buffer":{"defaultState":"disabled"}}}
>
> All of our collections have a timestamp field, index_date. In the source
> collection all the records have a date of 2/28/2018 but the target
> collections have a latest date of 1/26/2018
>
> I don't see cdcr errors in the logs, but we use logstash to search them, and
> we're still perfecting that.
>
> We have a number of similar collections that behave correctly. This is the
> only collection that is a TLOG collection. It appears that CDCR doesn't
> support TLOG collections.
>
> It looks like the data is getting to the target servers. I see tlog files
> with the right timestamps.
[jira] [Commented] (SOLR-12902) Block Expensive Queries custom Solr component
[ https://issues.apache.org/jira/browse/SOLR-12902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664209#comment-16664209 ] Varun Thacker commented on SOLR-12902: -- Thanks Anshum! In the approach Tirth has taken , it's a search component. So to configure what needs to be blocked one would need to configure the "defaults" section of a request handler. Would this model be consistent with the related hooks that you're referring to? > Block Expensive Queries custom Solr component > - > > Key: SOLR-12902 > URL: https://issues.apache.org/jira/browse/SOLR-12902 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Tirth Rajen Mehta >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Added a Block Expensive Queries custom Solr component ( > [https://github.com/apache/lucene-solr/pull/47|https://github.com/apache/lucene-solr/pull/477)] > ) : > * This search component can be plugged into your SearchHandler if you would > like to block some well known expensive queries. > * The queries that are blocked and failed by component currently are deep > pagination queries as they are known to consume lot of memory and CPU. These > are > * > ** queries with a start offset which is greater than the configured > maxStartOffset config parameter value > ** queries with a row param value which is greater than the configured > maxRowsFetch config parameter value -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12903) Query Source Tracker custom Solr component
[ https://issues.apache.org/jira/browse/SOLR-12903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664201#comment-16664201 ] Varun Thacker commented on SOLR-12903: -- Hi Tirth, What's the motivation behind this? Is it to make sure only clients that have access to the secret values can query the system? > Query Source Tracker custom Solr component > -- > > Key: SOLR-12903 > URL: https://issues.apache.org/jira/browse/SOLR-12903 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Tirth Rajen Mehta >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Added a Query Source Tracker custom Solr component > (https://github.com/apache/lucene-solr/pull/478) : > * This component can be configured for a RequestHandler for query requests. > * This component mandates that clients to pass in a "qi" request parameter > with a valid value which is configured in the SearchComponent definition in > the solrconfig.xml file. > * It fails the query if the "qi" parameter is missing or if the value passed > in is in-valid. This behavior of failing the queries can be controlled by the > failQueries config parameter. > * It also collects the rate per sec metric per unique "qi" value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12902) Block Expensive Queries custom Solr component
[ https://issues.apache.org/jira/browse/SOLR-12902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664178#comment-16664178 ] Varun Thacker commented on SOLR-12902: -- Hi Tirth, I've left a few comments on the PR . I think continuing the review on the PR would be the best approach Future ideas could be block facet requests with rows=-1 , wildcards in the middle ( you've already mentioned leading wildcards in the docs ) etc. [~anshumg] @Tomas do you guys have any thoughts on this approach? > Block Expensive Queries custom Solr component > - > > Key: SOLR-12902 > URL: https://issues.apache.org/jira/browse/SOLR-12902 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Tirth Rajen Mehta >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Added a Block Expensive Queries custom Solr component ( > [https://github.com/apache/lucene-solr/pull/47|https://github.com/apache/lucene-solr/pull/477)] > ) : > * This search component can be plugged into your SearchHandler if you would > like to block some well known expensive queries. > * The queries that are blocked and failed by component currently are deep > pagination queries as they are known to consume lot of memory and CPU. These > are > * > ** queries with a start offset which is greater than the configured > maxStartOffset config parameter value > ** queries with a row param value which is greater than the configured > maxRowsFetch config parameter value -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12793) Move TestCloudJSONFacetJoinDomain / TestCloudJSONFacetSKG to the facet package
[ https://issues.apache.org/jira/browse/SOLR-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker resolved SOLR-12793. -- Resolution: Fixed Fix Version/s: master (8.0) 7.6 > Move TestCloudJSONFacetJoinDomain / TestCloudJSONFacetSKG to the facet package > -- > > Key: SOLR-12793 > URL: https://issues.apache.org/jira/browse/SOLR-12793 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Assignee: Varun Thacker >Priority: Trivial > Fix For: 7.6, master (8.0) > > > This proposal is to move the the following two classes to the facet test > package ( org.apache.solr.search.facet ) > - TestCloudJSONFacetJoinDomain > - TestCloudJSONFacetSKG -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12801) Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.
[
https://issues.apache.org/jira/browse/SOLR-12801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662860#comment-16662860
]
Varun Thacker commented on SOLR-12801:
--
{quote}I want to start having a place for unit tests and integration tests -
maybe we can work on both at once?
{quote}
I didn't quite follow this. Is there anything particular that you wanted me to
look into while i'm working on SOLR-12793 . Happy to help move some tests to
their own packages if that would help here
> Fix the tests, remove BadApples and AwaitsFix annotations, improve env for
> test development.
>
>
> Key: SOLR-12801
> URL: https://issues.apache.org/jira/browse/SOLR-12801
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Critical
>
> A single issue to counteract the single issue adding tons of annotations, the
> continued addition of new flakey tests, and the continued addition of
> flakiness to existing tests.
> Lots more to come.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
[
https://issues.apache.org/jira/browse/SOLR-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662659#comment-16662659
]
Varun Thacker commented on SOLR-12057:
--
{quote}I strongly agree with consolidating CdcrBidirectinalTest with the test
in this patch, and potentially for Cdcr support for pull replicas fix too.
Seeking advice on whether we should do it under this Jira or create new one.
{quote}
I'll leave it up to you since you're putting in the work. If you think this
will derail the main goal of the Jira by all means create a separate Jira to
tackle it separately.
{quote}Not really, we can remove this safely, from, all tests; 2 sec sleep is
for loading the Cdcr components and avoiding potentially few retries.
{quote}
Again feel free to incorporate that in the next iteration or create a separate
jira.
{quote}Sure, I will include the parent-child doc;
{quote}
+1
> CDCR does not replicate to Collections with TLOG Replicas
> -
>
> Key: SOLR-12057
> URL: https://issues.apache.org/jira/browse/SOLR-12057
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.2
>Reporter: Webster Homer
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12057.patch, SOLR-12057.patch, SOLR-12057.patch,
> SOLR-12057.patch, cdcr-fail-with-tlog-pull.patch,
> cdcr-fail-with-tlog-pull.patch
>
>
> We created a collection using TLOG replicas in our QA clouds.
> We have a locally hosted solrcloud with 2 nodes, all our collections have 2
> shards. We use CDCR to replicate the collections from this environment to 2
> data centers hosted in Google cloud. This seems to work fairly well for our
> collections with NRT replicas. However the new TLOG collection has problems.
>
> The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
> shards per collection with 2 replicas per shard.
>
> We never see data show up in the cloud collections, but we do see tlog files
> show up on the cloud servers. I can see that all of the servers have cdcr
> started, buffers are disabled.
> The cdcr source configuration is:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"},
> {
>
> "zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"}],
> "replicator":{
> "threadPoolSize":4,
> "schedule":500,
> "batchSize":250},
> "updateLogSynchronizer":\{"schedule":6
>
> The target configurations in the 2 clouds are the same:
> "requestHandler":{"/cdcr":{ "name":"/cdcr",
> "class":"solr.CdcrRequestHandler", "buffer":{"defaultState":"disabled"}}}
>
> All of our collections have a timestamp field, index_date. In the source
> collection all the records have a date of 2/28/2018 but the target
> collections have a latest date of 1/26/2018
>
> I don't see cdcr errors in the logs, but we use logstash to search them, and
> we're still perfecting that.
>
> We have a number of similar collections that behave correctly. This is the
> only collection that is a TLOG collection. It appears that CDCR doesn't
> support TLOG collections.
>
> It looks like the data is getting to the target servers. I see tlog files
> with the right timestamps. Looking at the timestamps on the documents in the
> collection none of the data appears to have been loaded.In the solr.log I see
> lots of /cdcr messages action=LASTPROCESSEDVERSION,
> action=COLLECTIONCHECKPOINT, and action=SHARDCHECKPOINT
>
> no errors
>
> Target collections autoCommit is set to 6 I tried sending a commit
> explicitly no difference. cdcr is uploading data, but no new data appears in
> the collection.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
[
https://issues.apache.org/jira/browse/SOLR-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker reassigned SOLR-12057:
Assignee: Varun Thacker
> CDCR does not replicate to Collections with TLOG Replicas
> -
>
> Key: SOLR-12057
> URL: https://issues.apache.org/jira/browse/SOLR-12057
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.2
>Reporter: Webster Homer
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12057.patch, SOLR-12057.patch, SOLR-12057.patch,
> SOLR-12057.patch, cdcr-fail-with-tlog-pull.patch,
> cdcr-fail-with-tlog-pull.patch
>
>
> We created a collection using TLOG replicas in our QA clouds.
> We have a locally hosted solrcloud with 2 nodes, all our collections have 2
> shards. We use CDCR to replicate the collections from this environment to 2
> data centers hosted in Google cloud. This seems to work fairly well for our
> collections with NRT replicas. However the new TLOG collection has problems.
>
> The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
> shards per collection with 2 replicas per shard.
>
> We never see data show up in the cloud collections, but we do see tlog files
> show up on the cloud servers. I can see that all of the servers have cdcr
> started, buffers are disabled.
> The cdcr source configuration is:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"},
> {
>
> "zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"}],
> "replicator":{
> "threadPoolSize":4,
> "schedule":500,
> "batchSize":250},
> "updateLogSynchronizer":\{"schedule":6
>
> The target configurations in the 2 clouds are the same:
> "requestHandler":{"/cdcr":{ "name":"/cdcr",
> "class":"solr.CdcrRequestHandler", "buffer":{"defaultState":"disabled"}}}
>
> All of our collections have a timestamp field, index_date. In the source
> collection all the records have a date of 2/28/2018 but the target
> collections have a latest date of 1/26/2018
>
> I don't see cdcr errors in the logs, but we use logstash to search them, and
> we're still perfecting that.
>
> We have a number of similar collections that behave correctly. This is the
> only collection that is a TLOG collection. It appears that CDCR doesn't
> support TLOG collections.
>
> It looks like the data is getting to the target servers. I see tlog files
> with the right timestamps. Looking at the timestamps on the documents in the
> collection none of the data appears to have been loaded.In the solr.log I see
> lots of /cdcr messages action=LASTPROCESSEDVERSION,
> action=COLLECTIONCHECKPOINT, and action=SHARDCHECKPOINT
>
> no errors
>
> Target collections autoCommit is set to 6 I tried sending a commit
> explicitly no difference. cdcr is uploading data, but no new data appears in
> the collection.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
[
https://issues.apache.org/jira/browse/SOLR-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662607#comment-16662607
]
Varun Thacker commented on SOLR-12057:
--
Hi Amrit,
Thanks for the patch! Here's some feedback from just the test case
* CdcrWithDiffReplicaTypesTest -> CdcrReplicaTypeTest - Maybe this is enough
to convey the test intention?
* Some unused imports would need to be removed
* Any reason we're hardcoding StandardDirectoryFactory instead of using of
letting the test framework pick one?
* After CdcrTestsUtil.cdcrStart(cluster1SolrClient); do we need to sleep for 2
seconds? When I see the usage of cdcrStart , I see that some usage has a 2s
sleep and some don't .
* Can we simply the variable naming in this loop. It's adding a batch of docs
right? "docs" is esentially how many batches of 100 docs will we index? Maybe
numBatches?
*
{code:java}
int docs = (TEST_NIGHTLY ? 100 : 10);
int numDocs_c1 = 0;
for (int k = 0; k < docs; k++) {
req = new UpdateRequest();
for (; numDocs_c1 < (k + 1) * 100; numDocs_c1++) {
SolrInputDocument doc = new SolrInputDocument();
doc.addField("id", "cluster1_" + numDocs_c1);
doc.addField("xyz", numDocs_c1);
req.add(doc);
}
req.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
log.info("Adding " + docs + " docs with commit=true, numDocs=" + numDocs_c1);
req.process(cluster1SolrClient);
}{code}
* It would be really cool if we pulled the meat of the test into a separate
method. The method would take two cloud solr client objects ( for the two
clusters ). That way we could test all 3 replica types in the same place by
calling this method. Perhaps consolidate CdcrBidirectionalTest as well?
* I really like how this test checks for all operations to make sure they work
correctly. perhaps we could expand it to add a parent-child document and an
in-place update as well?
> CDCR does not replicate to Collections with TLOG Replicas
> -
>
> Key: SOLR-12057
> URL: https://issues.apache.org/jira/browse/SOLR-12057
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.2
>Reporter: Webster Homer
>Priority: Major
> Attachments: SOLR-12057.patch, SOLR-12057.patch, SOLR-12057.patch,
> SOLR-12057.patch, cdcr-fail-with-tlog-pull.patch,
> cdcr-fail-with-tlog-pull.patch
>
>
> We created a collection using TLOG replicas in our QA clouds.
> We have a locally hosted solrcloud with 2 nodes, all our collections have 2
> shards. We use CDCR to replicate the collections from this environment to 2
> data centers hosted in Google cloud. This seems to work fairly well for our
> collections with NRT replicas. However the new TLOG collection has problems.
>
> The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
> shards per collection with 2 replicas per shard.
>
> We never see data show up in the cloud collections, but we do see tlog files
> show up on the cloud servers. I can see that all of the servers have cdcr
> started, buffers are disabled.
> The cdcr source configuration is:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"},
> {
>
> "zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"}],
> "replicator":{
> "threadPoolSize":4,
> "schedule":500,
> "batchSize":250},
> "updateLogSynchronizer":\{"schedule":6
>
> The target configurations in the 2 clouds are the same:
> "requestHandler":{"/cdcr":{ "name":"/cdcr",
> "class":"solr.CdcrRequestHandler", "buffer":{"defaultState":"disabled"}}}
>
> All of our collections have a timestamp field, index_date. In the source
> collection all the records have a date of 2/28/2018 but the target
> collections have a latest date of 1/26/2018
>
> I don't see cdcr errors in the logs, but we use logstash to search them, and
> we're still perfecting that.
>
> We have a number of similar collections that behave correctly. This is the
> only collection that is a TLOG collection. It appears that CDCR doesn't
> support TLOG collections.
>
> It looks like
[jira] [Commented] (SOLR-12829) Add plist (parallel list) Streaming Expression
[
https://issues.apache.org/jira/browse/SOLR-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662561#comment-16662561
]
Varun Thacker commented on SOLR-12829:
--
Hi Joel,
I noticed that the commit doesn't have a CHANGES entry.
> Add plist (parallel list) Streaming Expression
> --
>
> Key: SOLR-12829
> URL: https://issues.apache.org/jira/browse/SOLR-12829
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Major
> Attachments: SOLR-12829.patch, SOLR-12829.patch
>
>
> The *plist* Streaming Expression wraps any number of streaming expressions
> and opens them in parallel. The results of each of the streams are then
> iterated in the order they appear in the list. Since many streams perform
> heavy pushed down operations when opened, like the FacetStream, this will
> result in the parallelization of these operations. For example plist could
> wrap several facet() expressions and open them each in parallel, which would
> cause the facets to be run in parallel, on different replicas in the cluster.
> Here is sample syntax:
> {code:java}
> plist(tuple(facet1=facet(...)),
> tuple(facet2=facet(...)),
> tuple(facet3=facet(...))) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12908) Add a default set of cluster preferences
[ https://issues.apache.org/jira/browse/SOLR-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662437#comment-16662437 ] Varun Thacker commented on SOLR-12908: -- I missed that! Sorry for the noise. It would be really nice if this was not in code and defined explicitly in the autoscaling.json file . What do you think? > Add a default set of cluster preferences > > > Key: SOLR-12908 > URL: https://issues.apache.org/jira/browse/SOLR-12908 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Varun Thacker >Priority: Major > > Similar to SOLR-12845 where we want to add a set of default cluster policies, > we should add some default cluster preferences as well > > We should always be trying to minimze cores , maximize freedisk for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12902) Block Expensive Queries custom Solr component
[ https://issues.apache.org/jira/browse/SOLR-12902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-12902: - Description: Added a Block Expensive Queries custom Solr component ( [https://github.com/apache/lucene-solr/pull/47|https://github.com/apache/lucene-solr/pull/477)] ) : * This search component can be plugged into your SearchHandler if you would like to block some well known expensive queries. * The queries that are blocked and failed by component currently are deep pagination queries as they are known to consume lot of memory and CPU. These are * ** queries with a start offset which is greater than the configured maxStartOffset config parameter value ** queries with a row param value which is greater than the configured maxRowsFetch config parameter value was: Added a Block Expensive Queries custom Solr component ([https://github.com/apache/lucene-solr/pull/477)] : * This search component can be plugged into your SearchHandler if you would like to block some well known expensive queries. * The queries that are blocked and failed by component currently are deep pagination queries as they are known to consume lot of memory and CPU. These are * ** queries with a start offset which is greater than the configured maxStartOffset config parameter value ** queries with a row param value which is greater than the configured maxRowsFetch config parameter value > Block Expensive Queries custom Solr component > - > > Key: SOLR-12902 > URL: https://issues.apache.org/jira/browse/SOLR-12902 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Tirth Rajen Mehta >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Added a Block Expensive Queries custom Solr component ( > [https://github.com/apache/lucene-solr/pull/47|https://github.com/apache/lucene-solr/pull/477)] > ) : > * This search component can be plugged into your SearchHandler if you would > like to block some well known expensive queries. > * The queries that are blocked and failed by component currently are deep > pagination queries as they are known to consume lot of memory and CPU. These > are > * > ** queries with a start offset which is greater than the configured > maxStartOffset config parameter value > ** queries with a row param value which is greater than the configured > maxRowsFetch config parameter value -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12894) Solr documention for Java Vendors
[
https://issues.apache.org/jira/browse/SOLR-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-12894:
-
Attachment: SOLR-12894.patch
> Solr documention for Java Vendors
> -
>
> Key: SOLR-12894
> URL: https://issues.apache.org/jira/browse/SOLR-12894
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Varun Thacker
>Priority: Major
> Attachments: SOLR-12894.patch, SOLR-12894.patch
>
>
> I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
> which my answer was yes . This was after I checked with Steve on which
> OpenJDK version runs on his jenkins
> For refrerence it currently uses -
> {code:java}
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
> OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
>
> Solr's ref guide (
> [https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
> ) mentions using Oracle 1.8 or higher .
>
> We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps
> even have a compatibility matrix
>
> Also we should note that Java 9 and 10 are short term releases . Hence remove
> wording that using Java8+ with more spefic versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12894) Solr documention for Java Vendors
[
https://issues.apache.org/jira/browse/SOLR-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-12894:
-
Attachment: SOLR-12894.patch
> Solr documention for Java Vendors
> -
>
> Key: SOLR-12894
> URL: https://issues.apache.org/jira/browse/SOLR-12894
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Varun Thacker
>Priority: Major
> Attachments: SOLR-12894.patch
>
>
> I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
> which my answer was yes . This was after I checked with Steve on which
> OpenJDK version runs on his jenkins
> For refrerence it currently uses -
> {code:java}
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
> OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
>
> Solr's ref guide (
> [https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
> ) mentions using Oracle 1.8 or higher .
>
> We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps
> even have a compatibility matrix
>
> Also we should note that Java 9 and 10 are short term releases . Hence remove
> wording that using Java8+ with more spefic versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12894) Solr documention for Java Vendors
[
https://issues.apache.org/jira/browse/SOLR-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661485#comment-16661485
]
Varun Thacker commented on SOLR-12894:
--
How does this change look?
> Solr documention for Java Vendors
> -
>
> Key: SOLR-12894
> URL: https://issues.apache.org/jira/browse/SOLR-12894
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Varun Thacker
>Priority: Major
> Attachments: SOLR-12894.patch
>
>
> I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
> which my answer was yes . This was after I checked with Steve on which
> OpenJDK version runs on his jenkins
> For refrerence it currently uses -
> {code:java}
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
> OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
>
> Solr's ref guide (
> [https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
> ) mentions using Oracle 1.8 or higher .
>
> We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps
> even have a compatibility matrix
>
> Also we should note that Java 9 and 10 are short term releases . Hence remove
> wording that using Java8+ with more spefic versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12801) Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.
[
https://issues.apache.org/jira/browse/SOLR-12801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661473#comment-16661473
]
Varun Thacker commented on SOLR-12801:
--
{quote}I'm going to start addressing tests by package (search package is first).
{quote}
I think today a lot of tests get lumped in the search package or cloud package
when it perfectly deserves it's own package. We could address that in a
separate Jira as well ( SOLR-12793 is one such example )
> Fix the tests, remove BadApples and AwaitsFix annotations, improve env for
> test development.
>
>
> Key: SOLR-12801
> URL: https://issues.apache.org/jira/browse/SOLR-12801
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Critical
>
> A single issue to counteract the single issue adding tons of annotations, the
> continued addition of new flakey tests, and the continued addition of
> flakiness to existing tests.
> Lots more to come.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12894) Solr documention for Java Vendors
[
https://issues.apache.org/jira/browse/SOLR-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661469#comment-16661469
]
Varun Thacker commented on SOLR-12894:
--
So turns out that
[https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java] was
updated at some point and today we have
[https://lucene.apache.org/solr/guide/7_5/solr-system-requirements.html]
If we are to reccomend users to use OpenJDK maybe we should have a link so that
users can download them on Windows , Linux and Mac?
[https://openjdk.java.net/install/] only has packages for linux .
The other way to look at it is , we shouldn't be in the business of
recommending one over the other. All we should say is both OpenJDK and Oracle
JDK are well tested and both work fine. Users can make their choice based on
that information.
> Solr documention for Java Vendors
> -
>
> Key: SOLR-12894
> URL: https://issues.apache.org/jira/browse/SOLR-12894
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Varun Thacker
>Priority: Major
>
> I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
> which my answer was yes . This was after I checked with Steve on which
> OpenJDK version runs on his jenkins
> For refrerence it currently uses -
> {code:java}
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
> OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
>
> Solr's ref guide (
> [https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
> ) mentions using Oracle 1.8 or higher .
>
> We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps
> even have a compatibility matrix
>
> Also we should note that Java 9 and 10 are short term releases . Hence remove
> wording that using Java8+ with more spefic versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12908) Add a default set of cluster preferences
[ https://issues.apache.org/jira/browse/SOLR-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-12908: - Component/s: AutoScaling > Add a default set of cluster preferences > > > Key: SOLR-12908 > URL: https://issues.apache.org/jira/browse/SOLR-12908 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Varun Thacker >Priority: Major > > Similar to SOLR-12845 where we want to add a set of default cluster policies, > we should add some default cluster preferences as well > > We should always be trying to minimze cores , maximize freedisk for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-12845:
-
Component/s: AutoScaling
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-12845.patch
>
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12908) Add a default set of cluster preferences
[ https://issues.apache.org/jira/browse/SOLR-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-12908: - Description: Similar to SOLR-12845 where we want to add a set of default cluster policies, we should add some default cluster preferences as well We should always be trying to minimze cores , maximize freedisk for example. was: Similar to SOLR-12845 where we want to add a set of default cluster policies, we should add some default cluster preferences as well We should always be truing to minimze cores , maximize freedisk for example. > Add a default set of cluster preferences > > > Key: SOLR-12908 > URL: https://issues.apache.org/jira/browse/SOLR-12908 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > > Similar to SOLR-12845 where we want to add a set of default cluster policies, > we should add some default cluster preferences as well > > We should always be trying to minimze cores , maximize freedisk for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12826) Add a default policy to equally distribute replicas of a shard
[
https://issues.apache.org/jira/browse/SOLR-12826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker resolved SOLR-12826.
--
Resolution: Won't Fix
This work will be superceded by SOLR-12845
> Add a default policy to equally distribute replicas of a shard
> --
>
> Key: SOLR-12826
> URL: https://issues.apache.org/jira/browse/SOLR-12826
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Varun Thacker
>Priority: Major
>
> We should have a policy to the effect of "maxShardsPerHost=1" by default.
> I created a 4 node cluster , created lots of collection on node1 and node2.
> Then used the suggestions to move replicas. I ended up with a scenario where
> both replicas of a collection was on one node.
> So if we create a policy such that this can never happen as
> maxShardsPerHost=1 was the default
> {code:java}
> {"replica": "<2", "shard" : "#EACH" , "node" :"#ANY", "strict" : "false"
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12908) Add a default set of cluster preferences
[ https://issues.apache.org/jira/browse/SOLR-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-12908: - Summary: Add a default set of cluster preferences (was: Add a default set of cluster preseferences) > Add a default set of cluster preferences > > > Key: SOLR-12908 > URL: https://issues.apache.org/jira/browse/SOLR-12908 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > > Similar to SOLR-12845 where we want to add a set of default cluster policies, > we should add some default cluster preferences as well > > We should always be truing to minimze cores , maximize freedisk for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12908) Add a default set of cluster preseferences
Varun Thacker created SOLR-12908: Summary: Add a default set of cluster preseferences Key: SOLR-12908 URL: https://issues.apache.org/jira/browse/SOLR-12908 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Varun Thacker Similar to SOLR-12845 where we want to add a set of default cluster policies, we should add some default cluster preferences as well We should always be truing to minimze cores , maximize freedisk for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661325#comment-16661325
]
Varun Thacker edited comment on SOLR-12845 at 10/23/18 9:35 PM:
I can think of two more defaults ( both should be strict:false ) that we could
add to the autoscaling.json file :
* Don't have more than two replicas on the same physical host :
{code:java}
{"replica": "<2", "host": “#ANY”}{code}
* Define a well known system property called "rack" in Solr ( SOLR-12907 ) -
{code:java}
{"replica": "#EQUAL", "shard": "#EACH", "rack": “#EACH”}{code}
was (Author: varunthacker):
I can think of two more defaults ( both should be strict:false ) that we could
add to the autoscaling.json file :
* Don't have more than two replicas on the same physical host : {{{"replica":
"<2", "host": “#ANY”}}}
* Define a well known system property called "rack" in Solr ( SOLR-12907 ) -
{{{"replica": "#EQUAL", "shard": "#EACH", "rack": “#EACH”}}}
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-12845.patch
>
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661325#comment-16661325
]
Varun Thacker commented on SOLR-12845:
--
I can think of two more defaults ( both should be strict:false ) that we could
add to the autoscaling.json file :
* Don't have more than two replicas on the same physical host : {{{"replica":
"<2", "host": “#ANY”}}}
* Define a well known system property called "rack" in Solr ( SOLR-12907 ) -
{{{"replica": "#EQUAL", "shard": "#EACH", "rack": “#EACH”}}}
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-12845.patch
>
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12907) Define a well known system property called rack for autoscaling policies
Varun Thacker created SOLR-12907:
Summary: Define a well known system property called rack for
autoscaling policies
Key: SOLR-12907
URL: https://issues.apache.org/jira/browse/SOLR-12907
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Varun Thacker
I want to setup a rule like to the effect of - Each shard should have their
replicas distributed equally amongst availability zones
For achieveing this today I can create a rule like this
{code:java}
{"replica": "#EQUAL", "shard": "#EACH", "sysprop.az": “#EACH”}{code}
And then make sure that every solr jvm starts up with a system property called
"az"
Another user might call the same property "availability_zone" and for some it's
just a different "rack"
All of them want to achieve the same goal of redundancy
So if we have a well kown property called "rack" it would help standardize
documentation , examples given out during talks etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12894) Solr documention for Java Vendors
[
https://issues.apache.org/jira/browse/SOLR-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659840#comment-16659840
]
Varun Thacker commented on SOLR-12894:
--
Files we'd need to change :
* SYSTEM_REQUIREMENTS.mdtext
* solr-system-requirements.adoc
> Solr documention for Java Vendors
> -
>
> Key: SOLR-12894
> URL: https://issues.apache.org/jira/browse/SOLR-12894
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Varun Thacker
>Priority: Major
>
> I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
> which my answer was yes . This was after I checked with Steve on which
> OpenJDK version runs on his jenkins
> For refrerence it currently uses -
> {code:java}
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
> OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
>
> Solr's ref guide (
> [https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
> ) mentions using Oracle 1.8 or higher .
>
> We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps
> even have a compatibility matrix
>
> Also we should note that Java 9 and 10 are short term releases . Hence remove
> wording that using Java8+ with more spefic versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12894) Solr documention for Java Vendors
[
https://issues.apache.org/jira/browse/SOLR-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-12894:
-
Description:
I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
which my answer was yes . This was after I checked with Steve on which OpenJDK
version runs on his jenkins
For refrerence it currently uses -
{code:java}
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
Solr's ref guide (
[https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
) mentions using Oracle 1.8 or higher .
We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps even
have a compatibility matrix
Also we should note that Java 9 and 10 are short term releases . Hence remove
wording that using Java8+ with more spefic versions.
was:
I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
which my answer was yes . This was after I checked with Steve on which OpenJDK
version runs on his jenkins
For refrerence it currently uses -
{code:java}
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
Solr's ref guide (
[https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
) mentions using Oracle 1.8 or higher .
We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps even
have a compatibility matrix
> Solr documention for Java Vendors
> -
>
> Key: SOLR-12894
> URL: https://issues.apache.org/jira/browse/SOLR-12894
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: Varun Thacker
>Priority: Major
>
> I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
> which my answer was yes . This was after I checked with Steve on which
> OpenJDK version runs on his jenkins
> For refrerence it currently uses -
> {code:java}
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
> OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
>
> Solr's ref guide (
> [https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
> ) mentions using Oracle 1.8 or higher .
>
> We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps
> even have a compatibility matrix
>
> Also we should note that Java 9 and 10 are short term releases . Hence remove
> wording that using Java8+ with more spefic versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12894) Solr documention for Java Vendors
Varun Thacker created SOLR-12894:
Summary: Solr documention for Java Vendors
Key: SOLR-12894
URL: https://issues.apache.org/jira/browse/SOLR-12894
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: documentation
Reporter: Varun Thacker
I was asked a question recently - "Is using OpenJDK safe with Solr 7.4" . To
which my answer was yes . This was after I checked with Steve on which OpenJDK
version runs on his jenkins
For refrerence it currently uses -
{code:java}
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode){code}
Solr's ref guide (
[https://lucene.apache.org/solr/guide/installing-solr.html#got-java|https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java]
) mentions using Oracle 1.8 or higher .
We should mention that both Oracle JDKs and Open JDKs are tested. Perhaps even
have a compatibility matrix
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12846) Policy rules do not support host variable
[
https://issues.apache.org/jira/browse/SOLR-12846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652945#comment-16652945
]
Varun Thacker commented on SOLR-12846:
--
+1
Here's an example that I want to talk about - Disallow two replicas on the same
physical host
{code:java}
{"replica": "<2", "host": “#ANY”}{code}
> Policy rules do not support host variable
> -
>
> Key: SOLR-12846
> URL: https://issues.apache.org/jira/browse/SOLR-12846
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Affects Versions: 7.5
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 7.6, master (8.0)
>
>
> The policy documentation says that there is a host variable supported in
> policy rules at
> https://lucene.apache.org/solr/guide/7_5/solrcloud-autoscaling-policy-preferences.html#node-selection-attributes
> But there is no mention of it in the code. Perhaps it got lost during
> refactorings and there were no tests for it? In any case, we should add it
> back. It'd be great if we can support #EACH for host so that one can write a
> rule to distribute replicas across hosts and not just nodes. This would be
> very useful when one runs multiple Solr JVMs in the same physical node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12822) If a replicationFactor less than specified in collection attribute show suggestion to ADD-REPLICA
[ https://issues.apache.org/jira/browse/SOLR-12822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker resolved SOLR-12822. -- Resolution: Fixed Fix Version/s: master (8.0) 7.6 Hi Noble, I'm resolving this issue. > If a replicationFactor less than specified in collection attribute show > suggestion to ADD-REPLICA > - > > Key: SOLR-12822 > URL: https://issues.apache.org/jira/browse/SOLR-12822 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Major > Fix For: 7.6, master (8.0) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11970) Deprecate maxShardsPerNode while creating collections
[
https://issues.apache.org/jira/browse/SOLR-11970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650312#comment-16650312
]
Varun Thacker commented on SOLR-11970:
--
[~shalinmangar] after the recent changes going into 7.6 ( SOLR-12739 and linked
Jiras ) this issue can perhaps be closed as invalid?
> Deprecate maxShardsPerNode while creating collections
> -
>
> Key: SOLR-11970
> URL: https://issues.apache.org/jira/browse/SOLR-11970
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
>
> Today maxShardsPerNode helps users ensure multiple replicas of the same shard
> don't get assigned to the same node.
> Starting 7.0 , the policy framework can express the same constraint.
> Both can conflict today.
> If a user creates a collection with maxShardsPerNode=1 here's the equivalent
> of it in the policy framework.
> {code}
> {"replica": "<2", "shard": "#EACH", "node": "#ANY"}
> {code}
> http://lucene.apache.org/solr/guide/solrcloud-autoscaling-policy-preferences.html#limit-replica-placement
> We should also change the default of maxShardsPerNode from 1 to -1 so that it
> doesn't fail commands when a user doesn't specify this parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12867) Async request status: Not getting all status messages because a response from one node will overwrite previous responses from that node
[ https://issues.apache.org/jira/browse/SOLR-12867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker resolved SOLR-12867. -- Resolution: Duplicate > Async request status: Not getting all status messages because a response from > one node will overwrite previous responses from that node > --- > > Key: SOLR-12867 > URL: https://issues.apache.org/jira/browse/SOLR-12867 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shawn Heisey >Priority: Major > > Problem noticed with REQUESTSTATUS on an async collections API BACKUP call. > Not all of the responses from different nodes in the collection are being > reported. According to [~shalinmangar], this is because multiple responses > from a node are overwriting earlier responses from that node. > Steps to reproduce: > * Start a cloud example with "bin/solr -e cloud" in a 7.5.0 binary download. > Tell it that you want 3 nodes, accept defaults for all other questions. > * Create a collection with 30 shards: > ** bin\solr create -c test2 -shards 30 -replicationFactor 2 > * Start an async backup of the collection. On a Windows system, the URL > might look like this: > ** > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test2&collection=test2&location=C%3A%5CUsers%5Celyograg%5CDownloads%5Csolrbackups&async=sometag > * After a few seconds (to give the backup time to complete), request the > status of the async operation: > ** > http://localhost:8983/solr/admin/collections?action=REQUESTSTATUS&requestid=sometag > The response will only contain individual statuses for 3 of the 30 shards. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12291) OverseerCollectionMessageHandler sliceCmd assumes only one replica exists on each node
[ https://issues.apache.org/jira/browse/SOLR-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650306#comment-16650306 ] Varun Thacker commented on SOLR-12291: -- The affected APIs by this bug is : BACKUPs , RELOAD collection and RESTORE collection > OverseerCollectionMessageHandler sliceCmd assumes only one replica exists on > each node > -- > > Key: SOLR-12291 > URL: https://issues.apache.org/jira/browse/SOLR-12291 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore, SolrCloud >Reporter: Varun Thacker >Priority: Major > Attachments: SOLR-122911.patch > > > The OverseerCollectionMessageHandler sliceCmd assumes only one replica exists > on one node > When multiple replicas of a slice are on the same node we only track one > replica's async request. This happens because the async requestMap's key is > "node_name" > I discovered this when [~alabax] shared some logs of a restore issue, where > the second replica got added before the first replica had completed it's > restorecore action. > While looking at the logs I noticed that the overseer never called > REQUESTSTATUS for the restorecore action , almost as if it had missed > tracking that particular async request. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12867) Async request status: Not getting all status messages because a response from one node will overwrite previous responses from that node
[ https://issues.apache.org/jira/browse/SOLR-12867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650291#comment-16650291 ] Varun Thacker commented on SOLR-12867: -- Hi Shawn, Unfortunately this has been a known bug for a while :/ I had created SOLR-12291 to track the work but neer got around to committing it. > Async request status: Not getting all status messages because a response from > one node will overwrite previous responses from that node > --- > > Key: SOLR-12867 > URL: https://issues.apache.org/jira/browse/SOLR-12867 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Shawn Heisey >Priority: Major > > Problem noticed with REQUESTSTATUS on an async collections API BACKUP call. > Not all of the responses from different nodes in the collection are being > reported. According to [~shalinmangar], this is because multiple responses > from a node are overwriting earlier responses from that node. > Steps to reproduce: > * Start a cloud example with "bin/solr -e cloud" in a 7.5.0 binary download. > Tell it that you want 3 nodes, accept defaults for all other questions. > * Create a collection with 30 shards: > ** bin\solr create -c test2 -shards 30 -replicationFactor 2 > * Start an async backup of the collection. On a Windows system, the URL > might look like this: > ** > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test2&collection=test2&location=C%3A%5CUsers%5Celyograg%5CDownloads%5Csolrbackups&async=sometag > * After a few seconds (to give the backup time to complete), request the > status of the async operation: > ** > http://localhost:8983/solr/admin/collections?action=REQUESTSTATUS&requestid=sometag > The response will only contain individual statuses for 3 of the 30 shards. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12860) MetricsHistoryHandler does not work with BasicAuth
Varun Thacker created SOLR-12860:
Summary: MetricsHistoryHandler does not work with BasicAuth
Key: SOLR-12860
URL: https://issues.apache.org/jira/browse/SOLR-12860
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Varun Thacker
I setup a 2 node cluster ( bin/solr start -e cloud -noprompt ) and then
uploaded the default security.json from
[http://lucene.apache.org/solr/guide/7_5/basic-authentication-plugin.html] .
I'm seeing errors like these in the logs which would indicate that the metrics
history handler would not work with basic auth enabled?
{code:java}
2018-10-12 22:06:43.496 WARN (MetricsHistoryHandler-12-thread-1) [ ]
o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node
192.168.0.8:7574_solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://192.168.0.8:7574/solr: Expected mime type
application/octet-stream but got text/html.
Error 401 require authentication
HTTP ERROR 401
Problem accessing /solr/admin/metrics. Reason:
require authentication
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:342)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:241)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:76)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:139)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:128)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:58]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(MetricsHistoryHandler.java:498)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHistoryHandler.java:371)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(MetricsHistoryHandler.java:231)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_112]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_112]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_112]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12859) DocExpirationUpdateProcessorFactory does not work with BasicAuth
Varun Thacker created SOLR-12859:
Summary: DocExpirationUpdateProcessorFactory does not work with
BasicAuth
Key: SOLR-12859
URL: https://issues.apache.org/jira/browse/SOLR-12859
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 7.5
Reporter: Varun Thacker
I setup a cluster with basic auth and then wanted to use Solr's TTL feature (
DocExpirationUpdateProcessorFactory ) to auto-delete documents.
Turns out it doesn't work when Basic Auth is enabled. I get the following
stacktrace from the logs
{code:java}
2018-10-12 22:06:38.967 ERROR (autoExpireDocs-42-thread-1) [ ]
o.a.s.u.p.DocExpirationUpdateProcessorFactory Runtime error in periodic
deletion of expired docs: Async exception during distributed update: Error from
server at http://192.168.0.8:8983/solr/gettingstarted_shard2_replica_n6:
require authentication
request:
http://192.168.0.8:8983/solr/gettingstarted_shard2_replica_n6/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F192.168.0.8%3A8983%2Fsolr%2Fgettingstarted_shard1_replica_n2%2F&wt=javabin&version=2
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Error from server at
http://192.168.0.8:8983/solr/gettingstarted_shard2_replica_n6: require
authentication
request:
http://192.168.0.8:8983/solr/gettingstarted_shard2_replica_n6/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F192.168.0.8%3A8983%2Fsolr%2Fgettingstarted_shard1_replica_n2%2F&wt=javabin&version=2
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:964)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1976)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
- 2018-09-18 13:07:55]
at
org.apache.solr.update.processor.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run(DocExpirationUpdateProcessorFactory.java:419)
[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
2018-09-18 13:07:55]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_112]
at j
[jira] [Commented] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643693#comment-16643693
]
Varun Thacker commented on SOLR-12845:
--
Thanks Shalin! This will be a great improvement in terms of how people use the
system - Users don't have to think of making rules to distribute load unless
they are doing something very specific.
{quote}However, autoAddReplicas implementation does not use this flag but maybe
it should? Either that or all the default rules should be made strict=fal
...
# Each shard cannot have more than one replica on the same node if
possible{quote}
Maybe this policy shoud not be strict=false but the other two policies could be
?
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Shalin Shekhar Mangar
>Priority: Major
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12739) Make autoscaling policy based replica placement the default strategy for placing replicas
[
https://issues.apache.org/jira/browse/SOLR-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642495#comment-16642495
]
Varun Thacker commented on SOLR-12739:
--
{quote}Today the default placement strategy is the same one used since Solr 4.x
which is to select nodes on a round robin fashion. I propose to make the
autoscaling policy based replica placement as the default policy for placing
replicas.
{quote}
+1 to make this the default.
Explaining users how does maxShardsPerNode interplay with policeis would be
very difficult no?
If there was no maxShardsPerNode going forward there would only be one way to
set rules up to distirbute replicas - policies
We should also ship with some default policies - "Don't allow more than one
replica of a shard on the same JVM" , "Distribute cores across the cluster
evenly" , "Distribute replicas per collection across the nodes"
> Make autoscaling policy based replica placement the default strategy for
> placing replicas
> -
>
> Key: SOLR-12739
> URL: https://issues.apache.org/jira/browse/SOLR-12739
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 7.6, master (8.0)
>
> Attachments: SOLR-12739.patch, SOLR-12739.patch, SOLR-12739.patch,
> SOLR-12739.patch
>
>
> Today the default placement strategy is the same one used since Solr 4.x
> which is to select nodes on a round robin fashion. I propose to make the
> autoscaling policy based replica placement as the default policy for placing
> replicas.
> This is related to SOLR-12648 where even though we have default cluster
> preferences, we don't use them unless a policy is also configured.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11644) RealTimeGet not working when router.field is not an uniqeKey
[
https://issues.apache.org/jira/browse/SOLR-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642470#comment-16642470
]
Varun Thacker commented on SOLR-11644:
--
{code:java}
// if shards=... then use that
if (zkController != null && params.get(ShardParams.SHARDS) == null) {
CloudDescriptor cloudDescriptor =
rb.req.getCore().getCoreDescriptor().getCloudDescriptor();{code}
We also support a "shards" param that isn't documented in
http://lucene.apache.org/solr/guide/7_5/realtime-get.html
> RealTimeGet not working when router.field is not an uniqeKey
>
>
> Key: SOLR-11644
> URL: https://issues.apache.org/jira/browse/SOLR-11644
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 6.6.2, 7.1
>Reporter: Jarek Mazgaj
>Priority: Major
>
> I have a schema with following fields:
> {code:java}
>
>
>
> candidate_id
> {code}
> A collection was created with following parameters:
> * numShards=4
> * replicationFactor=2
> * *router.field=company_id*
> When I try to do a Real Time Get with no routing information:
> {code:java}
> /get?id=1044101665
> {code}
> I get an empty response.
> When I try to add routing information (search returns document for these
> values):
> {code:java}
> /get?id=1044101665&_route_=77493783
> {code}
> I get an error:
> {code}
> org.apache.solr.common.SolrException: Can't find shard 'applicants_shard7'
> at
> org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:888)
> at
> org.apache.solr.handler.component.RealTimeGetComponent.createSubRequests(RealTimeGetComponent.java:835)
> at
> org.apache.solr.handler.component.RealTimeGetComponent.distributedProcess(RealTimeGetComponent.java:791)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:345)
> at
> org.apache.solr.handler.RealTimeGetHandler.handleRequestBody(RealTimeGetHandler.java:46)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2484)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProd
[jira] [Commented] (SOLR-11644) RealTimeGet not working when router.field is not an uniqeKey
[
https://issues.apache.org/jira/browse/SOLR-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642378#comment-16642378
]
Varun Thacker commented on SOLR-11644:
--
{quote}bq. Please note that RealTime Get or retrieval by id would also require
the parameter _route_ (or shard.keys) to avoid a distributed search.
{quote}
And from Solr 4.5 CHANGES entry
{quote}* The routing parameter "shard.keys" is deprecated as part of SOLR-5017
.The new parameter name is '_route_' .
The old parameter should continue to work for another release (Noble
Paul){quote}
We should remove "shard.keys" from the ref-guide then?
{quote}Given your schema, it should be:
{{get?candidate_id=1044101665}}
{quote}
Are we sure? It would be a strange syntax to support right? Here's an example
from the ref-guide (
[https://lucene.apache.org/solr/guide/6_6/realtime-get.html] )
{code:java}
http://localhost:8983/solr/techproducts/get?ids=mydoc,IW-02{code}
Would this be supported?
Here's a snippet from RealTimeGetComponent that makes me think this won't be
supported
{code:java}
final String id[] = params.getParams(ID);
final String ids[] = params.getParams("ids");{code}
{quote}Routing on one field (with duplicate values) and having a be
a different field then expecting RTG to find the document seems "fraught".
{quote}
But the use-case would be valid right? I want to shard by a field which has
company id information and at search time use the _{{_route__}}_ param so that
I can limit the number of shards I go against. If it works for {{/select}}
queries then shouldn't it also work for {{/get}} queries ?
> RealTimeGet not working when router.field is not an uniqeKey
>
>
> Key: SOLR-11644
> URL: https://issues.apache.org/jira/browse/SOLR-11644
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 6.6.2, 7.1
>Reporter: Jarek Mazgaj
>Priority: Major
>
> I have a schema with following fields:
> {code:java}
>
>
>
> candidate_id
> {code}
> A collection was created with following parameters:
> * numShards=4
> * replicationFactor=2
> * *router.field=company_id*
> When I try to do a Real Time Get with no routing information:
> {code:java}
> /get?id=1044101665
> {code}
> I get an empty response.
> When I try to add routing information (search returns document for these
> values):
> {code:java}
> /get?id=1044101665&_route_=77493783
> {code}
> I get an error:
> {code}
> org.apache.solr.common.SolrException: Can't find shard 'applicants_shard7'
> at
> org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:888)
> at
> org.apache.solr.handler.component.RealTimeGetComponent.createSubRequests(RealTimeGetComponent.java:835)
> at
> org.apache.solr.handler.component.RealTimeGetComponent.distributedProcess(RealTimeGetComponent.java:791)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:345)
> at
> org.apache.solr.handler.RealTimeGetHandler.handleRequestBody(RealTimeGetHandler.java:46)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2484)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(Handl
[jira] [Commented] (SOLR-5004) Allow a shard to be split into 'n' sub-shards
[ https://issues.apache.org/jira/browse/SOLR-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642345#comment-16642345 ] Varun Thacker commented on SOLR-5004: - What if we call this param - "numShards" instead? Also we would need to document this param and make it more clear in the ref-guide that there are three ways to split via the "ranges" , "split.key" and "numShards/numSubShards" param > Allow a shard to be split into 'n' sub-shards > - > > Key: SOLR-5004 > URL: https://issues.apache.org/jira/browse/SOLR-5004 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: 4.3, 4.3.1 >Reporter: Anshum Gupta >Assignee: Anshum Gupta >Priority: Major > Attachments: SOLR-5004.01.patch, SOLR-5004.patch, SOLR-5004.patch > > > As of now, a SPLITSHARD call is hardcoded to create 2 sub-shards from the > parent one. Accept a parameter to split into n sub-shards. > Default it to 2 and perhaps also have an upper bound to it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11718) Deprecate CDCR Buffer APIs and set Buffer to "false"
[ https://issues.apache.org/jira/browse/SOLR-11718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637469#comment-16637469 ] Varun Thacker commented on SOLR-11718: -- Thanks Amrit! I've gone and made some more changes to remove the concept of buffering from our docs. precommit is also happy. I'd like you to review the final patch and then I'll go ahead and commit it. > Deprecate CDCR Buffer APIs and set Buffer to "false" > > > Key: SOLR-11718 > URL: https://issues.apache.org/jira/browse/SOLR-11718 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: CDCR >Reporter: Amrit Sarkar >Assignee: Varun Thacker >Priority: Major > Fix For: 7.6, master (8.0) > > Attachments: SOLR-11718.patch, SOLR-11718.patch, SOLR-11718.patch > > > Kindly see the discussion on SOLR-11652. > Today, if we see the current CDCR documentation page, buffering is "disabled" > by default in both source and target. We don't see any purpose served by Cdcr > buffering and it is quite an overhead considering it can take a lot heap > space (tlogs ptr) and forever retention of tlogs on the disk when enabled. > Also today, even if we disable buffer from API on source , considering it was > enabled at startup, tlogs are never purged on leader node of shards of > source, refer jira: SOLR-11652 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11718) Deprecate CDCR Buffer APIs and set Buffer to "false"
[ https://issues.apache.org/jira/browse/SOLR-11718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-11718: - Attachment: SOLR-11718.patch > Deprecate CDCR Buffer APIs and set Buffer to "false" > > > Key: SOLR-11718 > URL: https://issues.apache.org/jira/browse/SOLR-11718 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: CDCR >Reporter: Amrit Sarkar >Assignee: Varun Thacker >Priority: Major > Fix For: 7.6, master (8.0) > > Attachments: SOLR-11718.patch, SOLR-11718.patch, SOLR-11718.patch > > > Kindly see the discussion on SOLR-11652. > Today, if we see the current CDCR documentation page, buffering is "disabled" > by default in both source and target. We don't see any purpose served by Cdcr > buffering and it is quite an overhead considering it can take a lot heap > space (tlogs ptr) and forever retention of tlogs on the disk when enabled. > Also today, even if we disable buffer from API on source , considering it was > enabled at startup, tlogs are never purged on leader node of shards of > source, refer jira: SOLR-11652 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12830) RestoreCmd should leverage adding 'n' replicas in one API call
[ https://issues.apache.org/jira/browse/SOLR-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-12830: - Description: After SOLR-9317 , We could simplify [this|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java#L329-L383] code and add 'n' replicas in one API call (was: After SOLR-9317 , We could simplify [link this|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java#L329-L383] code and add 'n' replicas in one API call) > RestoreCmd should leverage adding 'n' replicas in one API call > -- > > Key: SOLR-12830 > URL: https://issues.apache.org/jira/browse/SOLR-12830 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Minor > > After SOLR-9317 , We could simplify > [this|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java#L329-L383] > code and add 'n' replicas in one API call -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12830) RestoreCmd should leverage adding 'n' replicas in one API call
Varun Thacker created SOLR-12830: Summary: RestoreCmd should leverage adding 'n' replicas in one API call Key: SOLR-12830 URL: https://issues.apache.org/jira/browse/SOLR-12830 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Reporter: Varun Thacker After SOLR-9317 , We could simplify [link this|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java#L329-L383] code and add 'n' replicas in one API call -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12817) Simply response processing in CreateShardCmd
[
https://issues.apache.org/jira/browse/SOLR-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637278#comment-16637278
]
Varun Thacker commented on SOLR-12817:
--
Do we also need the AddReplica method to take {{Runnable onComplete}} ?
> Simply response processing in CreateShardCmd
>
>
> Key: SOLR-12817
> URL: https://issues.apache.org/jira/browse/SOLR-12817
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Priority: Major
>
> While working on SOLR-12708 , Mano disccovered used the response parsing
> technique from CreateShardCmd
> {code:java}
> final NamedList addResult = new NamedList();
> try {
> ocmh.addReplica(zkStateReader.getClusterState(), addReplicasProps,
> addResult, () -> {
> Object addResultFailure = addResult.get("failure");
> if (addResultFailure != null) {
> SimpleOrderedMap failure = (SimpleOrderedMap) results.get("failure");
> if (failure == null) {
> failure = new SimpleOrderedMap();
> results.add("failure", failure);
> }
> failure.addAll((NamedList) addResultFailure);
> } else {
> SimpleOrderedMap success = (SimpleOrderedMap) results.get("success");
> if (success == null) {
> success = new SimpleOrderedMap();
> results.add("success", success);
> }
> success.addAll((NamedList) addResult.get("success"));
> }
> });
> }{code}
>
> This code works as the response can have either a failure or a success. But
> isn't it the same as doing this?
> {code:java}
> ocmh.addReplica(zkStateReader.getClusterState(), addReplicasProps, results,
> null);{code}
>
> Maybe I am missing the motication here . [~caomanhdat] WDYT? If the usage is
> needed then at-least I'd want to document the reason in the code for future
> refernece.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12826) Add a default policy to equally distribute replicas of a shard
Varun Thacker created SOLR-12826:
Summary: Add a default policy to equally distribute replicas of a
shard
Key: SOLR-12826
URL: https://issues.apache.org/jira/browse/SOLR-12826
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: AutoScaling
Reporter: Varun Thacker
We should have a policy to the effect of "maxShardsPerHost=1" by default.
I created a 4 node cluster , created lots of collection on node1 and node2.
Then used the suggestions to move replicas. I ended up with a scenario where
both replicas of a collection was on one node.
So if we create a policy such that this can never happen as maxShardsPerHost=1
was the default
{code:java}
{"replica": "<2", "shard" : "#EACH" , "node" :"#ANY", "strict" : "false" }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12823) remove clusterstate.json in Lucene/Solr 8.0
[ https://issues.apache.org/jira/browse/SOLR-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-12823: - Summary: remove clusterstate.json in Lucene/Solr 8.0 (was: remove clusterstate.json 8.0) > remove clusterstate.json in Lucene/Solr 8.0 > --- > > Key: SOLR-12823 > URL: https://issues.apache.org/jira/browse/SOLR-12823 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Blocker > > clusterstate.json is an artifact of a pre 5.0 Solr release. We should remove > that in 8.0 > It stays empty unless you explicitly ask to create the collection with the > old "stateFormat" and there is no reason for one to create a collection with > the old stateFormat. > We should also remove the "stateFormat" argument in create collection > We should also remove MIGRATESTATEVERSION as well > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12823) remove clusterstate.json 8.0
Varun Thacker created SOLR-12823: Summary: remove clusterstate.json 8.0 Key: SOLR-12823 URL: https://issues.apache.org/jira/browse/SOLR-12823 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Reporter: Varun Thacker clusterstate.json is an artifact of a pre 5.0 Solr release. We should remove that in 8.0 It stays empty unless you explicitly ask to create the collection with the old "stateFormat" and there is no reason for one to create a collection with the old stateFormat. We should also remove the "stateFormat" argument in create collection We should also remove MIGRATESTATEVERSION as well -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12820) Auto pick method:dvhash based on thresholds
Varun Thacker created SOLR-12820:
Summary: Auto pick method:dvhash based on thresholds
Key: SOLR-12820
URL: https://issues.apache.org/jira/browse/SOLR-12820
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: Facet Module
Reporter: Varun Thacker
I've worked with two users last week where explicitly using method:dvhash
improved the faceting speeds drastically.
The common theme in both the use-cases were: One collection hosting data for
multiple users. We always filter documents for one user ( therby limiting the
number of documents drastically ) and then perfoming a complex nested JSON
facet.
Both use-cases fit perfectly in this criteria that [~yo...@apache.org] mentioed
on SOLR-9142
{quote}faceting on a string field with a high cardinality compared to it's
domain is less efficient than it could be.
{quote}
And DVHASH was the perfect optimization for these use-cases.
We are using the facet stream expression in one of the use-cases which doesn't
expose the method param. We could expose the method param to facet stream but I
feel the better approach to solve this problem would be to address this TODO in
the code withing the JSON Facet Module
{code:java}
if (mincount > 0 && prefix == null && (ntype != null || method ==
FacetMethod.DVHASH)) {
// TODO can we auto-pick for strings when term cardinality is much
greater than DocSet cardinality?
// or if we don't know cardinality but DocSet size is very small
return new FacetFieldProcessorByHashDV(fcontext, this, sf);{code}
I thought about this a little and this was the approach I am thinking currently
to tackle this problem
{code:java}
int matchingDocs = fcontext.base.size();
int totalDocs = fcontext.searcher.getIndexReader().maxDoc();
//if matchingDocs is close to the totalDocs then we aren't filtering many
documents.
//that means the array approach would probably be better than the dvhash
approach
//Trying to find the cardinality for the matchingDocs would be expensive.
//Also for totalDocs we don't have a global cardinality present at index time
but we have a per segment cardinality
//So using the number of matches as an alternate heuristic would do the job
here?{code}
Any thoughts if this approach makes sense? it could be I'm thinking of this
approach just because both the users I worked with last week fell in this
cateogory.
cc [~dsmiley] [~joel.bernstein]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12817) Simply response processing in CreateShardCmd
[
https://issues.apache.org/jira/browse/SOLR-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634553#comment-16634553
]
Varun Thacker commented on SOLR-12817:
--
Hi [~ab] , [~noble.paul] any insights here?
> Simply response processing in CreateShardCmd
>
>
> Key: SOLR-12817
> URL: https://issues.apache.org/jira/browse/SOLR-12817
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Priority: Major
>
> While working on SOLR-12708 , Mano disccovered used the response parsing
> technique from CreateShardCmd
> {code:java}
> final NamedList addResult = new NamedList();
> try {
> ocmh.addReplica(zkStateReader.getClusterState(), addReplicasProps,
> addResult, () -> {
> Object addResultFailure = addResult.get("failure");
> if (addResultFailure != null) {
> SimpleOrderedMap failure = (SimpleOrderedMap) results.get("failure");
> if (failure == null) {
> failure = new SimpleOrderedMap();
> results.add("failure", failure);
> }
> failure.addAll((NamedList) addResultFailure);
> } else {
> SimpleOrderedMap success = (SimpleOrderedMap) results.get("success");
> if (success == null) {
> success = new SimpleOrderedMap();
> results.add("success", success);
> }
> success.addAll((NamedList) addResult.get("success"));
> }
> });
> }{code}
>
> This code works as the response can have either a failure or a success. But
> isn't it the same as doing this?
> {code:java}
> ocmh.addReplica(zkStateReader.getClusterState(), addReplicasProps, results,
> null);{code}
>
> Maybe I am missing the motication here . [~caomanhdat] WDYT? If the usage is
> needed then at-least I'd want to document the reason in the code for future
> refernece.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12818) failed create collection leaves dirty state
Varun Thacker created SOLR-12818:
Summary: failed create collection leaves dirty state
Key: SOLR-12818
URL: https://issues.apache.org/jira/browse/SOLR-12818
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 7.5
Reporter: Varun Thacker
I tried creating a collection in 7.5 that will always error out:
1. bin/solr start -e cloud -noprompt
2.
http://localhost:8983/solr/admin/collections?action=CREATE&name=_name_of_collection_1&numShards=1&replicationFactor=3&autoAddReplicas=true
The create collection failed but the
/collections/_name_of_collection_1/state.json get's left behind
{code:java}
{"_name_of_collection_1":{
"pullReplicas":"0",
"replicationFactor":"3",
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"true",
"nrtReplicas":"3",
"tlogReplicas":"0",
"shards":{"shard1":{
"range":"8000-7fff",
"state":"active",
"replicas":{}{code}
This doesn't happen in 7.4
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
