[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+164) - Build # 19493 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/19493/
Java: 32bit/jdk-9-ea+164 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest

Error Message:


Stack Trace:
java.lang.AssertionError
at 
__randomizedtesting.SeedInfo.seed([F1DBC2D8733E0A49:72AD9D2AA54704E8]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at 
org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest(TestCloudRecovery.java:197)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:563)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 1694 lines...]
   [junit4] JVM J1: stderr was not empty, see: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/temp/junit4-J1-20170427_033127_27212895552774552343827.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) 
   [junit4] Java HotSpot(TM) Server VM warning: 

[jira] [Updated] (SOLR-10431) Make it possible to invoke v2 api calls using SolrJ

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul updated SOLR-10431:
--
Attachment: SOLR-10431.patch

let's use a builder pattern

> Make it possible to invoke v2 api calls using SolrJ
> ---
>
> Key: SOLR-10431
> URL: https://issues.apache.org/jira/browse/SOLR-10431
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: v2 API
>Reporter: Noble Paul
>Assignee: Cao Manh Dat
> Attachments: SOLR-10431.patch, SOLR-10431.patch, SOLR-10431.patch, 
> SOLR-10431.patch, SOLR-10431.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SolrCloud "master mode" planned?

[Greg Pendlebury]

Would it be possible to have an optional node that can dynamically assume a
leadership position?

There has been some small amount of discussion here about whether we could
have a 'search node' join a cluster and give it an empty/null (or zero
length) hash range on the clusterstate file and it would host no lucene
segments, which helps it avoid any GC issues related to commits, or NIC
saturation related to replication. The node could  possibly be tuned and
configured purely for search and a small number of these nodes could be put
behind a traditional load balancer and remove the need for search clients
to understand zookeeper. That last part was attractive to us simply for
search clients like JMeter, but it is nice for other reasons (like
firewalling the ZK nodes and Solr).

Our theory (which never evolved beyond idle chat) was that this would
almost be possible as it currently stands, but those sorts of nodes might
be an attractive place to host any 'leadership' or features which
optionally buy in to 'some nodes more equal than others'.

Ta,
Greg


On 27 April 2017 at 11:27, Walter Underwood  wrote:

> Not fired up about approaches that have “some nodes more equal than
> others”, whether it is zk-only or replica-only. That is the opposite of
> autoscaling.
>
> Getting our Solr Cloud cluster running in test and in prod was
> surprisingly difficult and unfortunately mysterious. I started with Solr
> 1.1, so I’m not exactly a noob.
>
> This cluster needs to handle very long text queries on a large collection
> (17 million docs and growing). After too much work, I’m really happy with
> the performance. This is 4 shards, 4 replicas, with AWS c4.8xlarge nodes.
> Yes, that is nearly $200,000 per year, just for the instances.
>
> Here is what I wanted, and what I ended up with after way too much work.
>
> * Collection configs and startup parameters under version control.
> * Separation between installing software and installing configs.
> * Automated config deploy from Jenkins.
> * Data and logs on /solr EBS volume.
> * HTTP on port 6090 to match our port-to-app mapping (just gave up on this
> one, geez).
> * Five node Zookeeper ensemble to avoid single point of failure during zk
> version or instance upgrade (wasted two weeks, gave up).
> * Graceful scale out and scale in (not even nearly done).
> * Metrics reported to Graphite, the performance bug in 6.4 cost a few
> weeks.
>
> Separating executables, config, and data should not be this hard, right? I
> thought we solved that in the 1990’s.
>
> I’ve never had a problem with getting Zookeeper running, but getting the
> five node ensemble to work was impossible. I wrote my first concurrent code
> 35 years ago, so I should be able to do this. Just could not get 3.4.6 to
> actually work on five nodes, no matter how many weird things we tried, like
> switching AWS instance types. Used an existing 3 node ensemble that we had
> wanted to decommission.
>
> The magic solr script commands do not document what happens when you
> change the port or server directory. Surprisingly, many of them only work
> after you have a local running Solr instance. Also not documented. So port
> must be passed in to many of the commands. I guess the server directory
> needs to be passed in, too, but I never figured that out.
>
> The required contents of the Zookeeper “filesystem” are undocumented, as
> far as I can tell. In fact, I found it really hard to figure out exactly
> what directory to give to either the solr script or the zkCli script.
> Earlier versions of solr had $SOLR_HOME/$collection/conf/…, but where is
> the directory arg in that hierarchy? Especially because … solr.xml
>
> Still don’t have a method that I trust to create a new cluster, but
> updates to our existing clusters are pretty solid. It just seems bogus to
> have a whole filesystem-based deployment, use that to bootstrap, then never
> use it again. I have zero trust that it will work the next time.
>
> I wrote a Python program that takes the URL of any Solr node (I used the
> load balancer), the collection(s) to update, and the base directory for
> configs (use the collection name as a subdirectory). It does this:
>
> 1. Gets info from Solr (requests package, yay!) and parses out the zk
> host, including the chroot. Some ugly parsing there, that should be
> straight JSON.
> 2. Connects to zk (kazoo package, yay!). Uploads solr.xml from the base of
> the configs directory.
> 3. Optionally, removes all the files from the zk config directory we are
> about to upload to.
> 4. Uploads all the files, recursively, from $configs/$collection on the
> filesystem to zk.
> 5. Optionally, links the config to the same name as a collection.
> 6. Sends RELOAD for that collection to the Solr node as async command.
> 7. Waits.
> 8. Waits. Finally completes.
> 9. Parses the response for succeeding and failed nodes. If there are
> failed nodes, exit with a failure result code. NOTE: this could leave the
> 

[jira] [Commented] (SOLR-10507) Core Admin API to emit collection details of each core

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985911#comment-15985911
 ] 

ASF subversion and git services commented on SOLR-10507:


Commit e40044fa4c44e9e6e65e010b067692df58eeada0 in lucene-solr's branch 
refs/heads/branch_6x from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e40044f ]

SOLR-10507: Core Admin status command to emit collection details of each core


> Core Admin API to emit collection details of each core
> --
>
> Key: SOLR-10507
> URL: https://issues.apache.org/jira/browse/SOLR-10507
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>
> see the {{cloud}} attribute
> {code}
>  {"gettingstarted_shard1_replica1":{
>   "name":"gettingstarted_shard1_replica1",
>   
> "instanceDir":"/Users/noble/work/solr4/solr/example/cloud/node1/solr/gettingstarted_shard1_replica1",
>   
> "dataDir":"/Users/noble/work/solr4/solr/example/cloud/node1/solr/gettingstarted_shard1_replica1/data/",
>   "config":"solrconfig.xml",
>   "schema":"managed-schema",
>   "startTime":"2017-04-18T09:51:02.645Z",
>   "uptime":73471,
>   "lastPublished":"active",
>   "configVersion":0,
>   "cloud":{
> "collection":"gettingstarted",
> "shard":"shard1",
> "replica":"core_node2"}}
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10507) Core Admin API to emit collection details of each core

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985909#comment-15985909
 ] 

ASF subversion and git services commented on SOLR-10507:


Commit fba52de066b896b8c768618f41a31612047d4b95 in lucene-solr's branch 
refs/heads/master from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fba52de ]

SOLR-10507: Core Admin status command to emit collection details of each core


> Core Admin API to emit collection details of each core
> --
>
> Key: SOLR-10507
> URL: https://issues.apache.org/jira/browse/SOLR-10507
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>
> see the {{cloud}} attribute
> {code}
>  {"gettingstarted_shard1_replica1":{
>   "name":"gettingstarted_shard1_replica1",
>   
> "instanceDir":"/Users/noble/work/solr4/solr/example/cloud/node1/solr/gettingstarted_shard1_replica1",
>   
> "dataDir":"/Users/noble/work/solr4/solr/example/cloud/node1/solr/gettingstarted_shard1_replica1/data/",
>   "config":"solrconfig.xml",
>   "schema":"managed-schema",
>   "startTime":"2017-04-18T09:51:02.645Z",
>   "uptime":73471,
>   "lastPublished":"active",
>   "configVersion":0,
>   "cloud":{
> "collection":"gettingstarted",
> "shard":"shard1",
> "replica":"core_node2"}}
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_121) - Build # 3377 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/3377/
Java: 64bit/jdk1.8.0_121 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestRebalanceLeaders.test

Error Message:
Checking the rebalance leader command failed, checkAppearOnce=true 
checkElectionZero=false checkZkLeadersAgree=false

Stack Trace:
java.lang.AssertionError: Checking the rebalance leader command failed, 
checkAppearOnce=true checkElectionZero=false checkZkLeadersAgree=false
at 
__randomizedtesting.SeedInfo.seed([F3A897005B6C3749:7BFCA8DAF5905AB1]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.TestRebalanceLeaders.checkConsistency(TestRebalanceLeaders.java:135)
at 
org.apache.solr.cloud.TestRebalanceLeaders.rebalanceLeaderTest(TestRebalanceLeaders.java:112)
at 
org.apache.solr.cloud.TestRebalanceLeaders.test(TestRebalanceLeaders.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 

[jira] [Commented] (SOLR-9248) HttpSolrClient not compatible with compression option

[Michael Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985869#comment-15985869
 ] 

Michael Braun commented on SOLR-9248:
-

Ran the test on 6.5x and master and the problem does not appear to be visible 
from the same test file. Would it be fair to create another ticket to fix the 
implementations of [Gzip/Deflate]DecompressingEntity to respect isRepeatable 
and the behavior of getContent assuming that?

> HttpSolrClient not compatible with compression option
> -
>
> Key: SOLR-9248
> URL: https://issues.apache.org/jira/browse/SOLR-9248
> Project: Solr
>  Issue Type: Bug
>  Components: SolrJ
>Affects Versions: 5.5, 5.5.1
>Reporter: Gary Lee
> Attachments: CompressedConnectionTest.java
>
>
> Since Solr 5.5, using the compression option 
> (solrClient.setAllowCompression(true)) causes the HTTP client to quickly run 
> out of connections in the connection pool. After debugging through this, we 
> found that the GZIPInputStream is incompatible with changes to how the 
> response input stream is closed in 5.5. It is at this point when the 
> GZIPInputStream throws an EOFException, and while this is silently eaten up, 
> the net effect is that the stream is never closed, leaving the connection 
> open. After a number of requests, the pool is exhausted and no further 
> requests can be served.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Looking for development docs.

[Doug Turnbull]

Something I found helpful was to go back to very early Lucene versions.
That let's you see the essential functionality in relatively
straightforward Java code. You can get a sense for how Lucene is
structured. Functionality has been built around this since. The Java has
been battle tested, refactored, and optimized. But those core bits were
really helpful for me to see what Lucene specifically did.

https://sourceforge.net/projects/lucene/

That plus Lucene in Action
On Wed, Apr 26, 2017 at 7:16 PM Erick Erickson 
wrote:

> Solr/Lucene is big. Really big. I'd think seriously about taking
> something you're interested in/know about, finding a JIRA that you'd
> like to work on and diving in. Plus there aren't very many
> architecture docs.
>
> Your characterization of the realms of responsibility is pretty accurate.
>
> Have you seen: https://wiki.apache.org/solr/HowToContribute?
>
> A somewhat painful but "safe" way to get your feet wet is to look at
> the coverage reports on jenkins and see what code is not tested in the
> junit tests and...write a test. At least I think the coverage reports
> are still there.
>
> Best,
> Erick
>
> On Wed, Apr 26, 2017 at 3:12 PM, David Lee 
> wrote:
> > I'd like to have a better understanding of how much of Solr is unique to
> it
> > versus directly extending Lucene.
> >
> > For example, I assume that sharding, replication, etc. is implemented in
> > Solr where-as indexing, querying, etc. would be implemented by Lucene.
> >
> > I'm hoping to learn enough to be able to contribute at some point.
> >
> > Thanks,
> >
> > David
> >
> >
> >
> > ---
> > This email has been checked for viruses by Avast antivirus software.
> > https://www.avast.com/antivirus
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+164) - Build # 19492 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/19492/
Java: 64bit/jdk-9-ea+164 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.SolrCloudExampleTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [TransactionLog, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.TransactionLog  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at org.apache.solr.update.TransactionLog.(TransactionLog.java:190)  at 
org.apache.solr.update.UpdateLog.newTransactionLog(UpdateLog.java:438)  at 
org.apache.solr.update.UpdateLog.ensureLog(UpdateLog.java:1250)  at 
org.apache.solr.update.UpdateLog.add(UpdateLog.java:524)  at 
org.apache.solr.update.UpdateLog.add(UpdateLog.java:509)  at 
org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:349)
  at 
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:268)
  at 
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:218)
  at 
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
  at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:991)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1207)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:753)
  at 
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
  at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
  at 
org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:336)
  at 
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)  
at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:187)
  at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:143)
  at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:313) 
 at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:258)  at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:129)
  at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:278) 
 at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:258)  at 
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:180)  at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:194)
  at 
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:108)
  at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)  
at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
  at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2486)  at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)  at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:355)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:306)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) 
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
at 

Re: SolrCloud "master mode" planned?

[Walter Underwood]

Not fired up about approaches that have “some nodes more equal than others”, 
whether it is zk-only or replica-only. That is the opposite of autoscaling.

Getting our Solr Cloud cluster running in test and in prod was surprisingly 
difficult and unfortunately mysterious. I started with Solr 1.1, so I’m not 
exactly a noob.

This cluster needs to handle very long text queries on a large collection (17 
million docs and growing). After too much work, I’m really happy with the 
performance. This is 4 shards, 4 replicas, with AWS c4.8xlarge nodes. Yes, that 
is nearly $200,000 per year, just for the instances.

Here is what I wanted, and what I ended up with after way too much work.

* Collection configs and startup parameters under version control.
* Separation between installing software and installing configs.
* Automated config deploy from Jenkins.
* Data and logs on /solr EBS volume.
* HTTP on port 6090 to match our port-to-app mapping (just gave up on this one, 
geez).
* Five node Zookeeper ensemble to avoid single point of failure during zk 
version or instance upgrade (wasted two weeks, gave up).
* Graceful scale out and scale in (not even nearly done).
* Metrics reported to Graphite, the performance bug in 6.4 cost a few weeks.

Separating executables, config, and data should not be this hard, right? I 
thought we solved that in the 1990’s.

I’ve never had a problem with getting Zookeeper running, but getting the five 
node ensemble to work was impossible. I wrote my first concurrent code 35 years 
ago, so I should be able to do this. Just could not get 3.4.6 to actually work 
on five nodes, no matter how many weird things we tried, like switching AWS 
instance types. Used an existing 3 node ensemble that we had wanted to 
decommission.

The magic solr script commands do not document what happens when you change the 
port or server directory. Surprisingly, many of them only work after you have a 
local running Solr instance. Also not documented. So port must be passed in to 
many of the commands. I guess the server directory needs to be passed in, too, 
but I never figured that out.

The required contents of the Zookeeper “filesystem” are undocumented, as far as 
I can tell. In fact, I found it really hard to figure out exactly what 
directory to give to either the solr script or the zkCli script. Earlier 
versions of solr had $SOLR_HOME/$collection/conf/…, but where is the directory 
arg in that hierarchy? Especially because … solr.xml

Still don’t have a method that I trust to create a new cluster, but updates to 
our existing clusters are pretty solid. It just seems bogus to have a whole 
filesystem-based deployment, use that to bootstrap, then never use it again. I 
have zero trust that it will work the next time. 

I wrote a Python program that takes the URL of any Solr node (I used the load 
balancer), the collection(s) to update, and the base directory for configs (use 
the collection name as a subdirectory). It does this:

1. Gets info from Solr (requests package, yay!) and parses out the zk host, 
including the chroot. Some ugly parsing there, that should be straight JSON.
2. Connects to zk (kazoo package, yay!). Uploads solr.xml from the base of the 
configs directory.
3. Optionally, removes all the files from the zk config directory we are about 
to upload to.
4. Uploads all the files, recursively, from $configs/$collection on the 
filesystem to zk.
5. Optionally, links the config to the same name as a collection.
6. Sends RELOAD for that collection to the Solr node as async command.
7. Waits.
8. Waits. Finally completes.
9. Parses the response for succeeding and failed nodes. If there are failed 
nodes, exit with a failure result code. NOTE: this could leave the cluster in 
an unfortunate state.
10. Repeat for the next collection on the command line.

A typical invocation looks like this:

   python solrtool/solrtool.py -h solr-cloud.test.cheggnet.com --clear -c 
questions -d configs

Installing a new version of Solr on a 16 node cluster is taking about an hour 
and a half. We follow the hints in the docs (upgrade the overseer last), and 
have turned that into a process. That includes some exciting Unix pipelines 
with “jq” to distinguish between live nodes and nodes in the cluster which are 
effed. We had one node which never could recover its replicas. We finally shot 
it and made a fresh one. About a third of the steps are automated. 
Unfortunately, “solr healthcheck -c colname” cannot be run remotely, so we need 
a sudo to each system and a manual review of the results.

If you want to see the process, ping me. Nothing secret, but too big for this 
email.

Getting to this point was way too hard. And this is not even close to good 
enough. It needs to upload the new config to a timestamped temp directory, then 
move/copy it after it is successful. If the reload fails, it should to roll 
back to the previous config and try another reload.

Our configs are under source control. We will 

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3970 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3970/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Expected to see collection awhollynewcollection_0 null Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/7)={
   "replicationFactor":"4",   "shards":{ "shard1":{   
"range":"8000-b332",   "state":"active",   "replicas":{ 
"core_node19":{   "core":"awhollynewcollection_0_shard1_replica4",  
 "base_url":"http://127.0.0.1:51792/solr;,   
"node_name":"127.0.0.1:51792_solr",   "state":"active"}, 
"core_node20":{   "core":"awhollynewcollection_0_shard1_replica3",  
 "base_url":"http://127.0.0.1:51797/solr;,   
"node_name":"127.0.0.1:51797_solr",   "state":"active",   
"leader":"true"}}}, "shard2":{   "range":"b333-e665",   
"state":"active",   "replicas":{"core_node8":{   
"core":"awhollynewcollection_0_shard2_replica4",   
"base_url":"http://127.0.0.1:51792/solr;,   
"node_name":"127.0.0.1:51792_solr",   "state":"active",   
"leader":"true"}}}, "shard3":{   "range":"e666-1998",   
"state":"active",   "replicas":{ "core_node14":{   
"core":"awhollynewcollection_0_shard3_replica4",   
"base_url":"http://127.0.0.1:51792/solr;,   
"node_name":"127.0.0.1:51792_solr",   "state":"active",   
"leader":"true"}, "core_node16":{   
"core":"awhollynewcollection_0_shard3_replica2",   
"base_url":"http://127.0.0.1:51796/solr;,   
"node_name":"127.0.0.1:51796_solr",   "state":"active"}}}, 
"shard4":{   "range":"1999-4ccb",   "state":"active",   
"replicas":{"core_node9":{   
"core":"awhollynewcollection_0_shard4_replica1",   
"base_url":"http://127.0.0.1:51790/solr;,   
"node_name":"127.0.0.1:51790_solr",   "state":"active",   
"leader":"true"}}}, "shard5":{   "range":"4ccc-7fff",   
"state":"active",   "replicas":{"core_node4":{   
"core":"awhollynewcollection_0_shard5_replica4",   
"base_url":"http://127.0.0.1:51792/solr;,   
"node_name":"127.0.0.1:51792_solr",   "state":"active",   
"leader":"true",   "router":{"name":"compositeId"},   
"maxShardsPerNode":"6",   "autoAddReplicas":"false",   "realtimeReplicas":"-1"}

Stack Trace:
java.lang.AssertionError: Expected to see collection awhollynewcollection_0
null
Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/7)={
  "replicationFactor":"4",
  "shards":{
"shard1":{
  "range":"8000-b332",
  "state":"active",
  "replicas":{
"core_node19":{
  "core":"awhollynewcollection_0_shard1_replica4",
  "base_url":"http://127.0.0.1:51792/solr;,
  "node_name":"127.0.0.1:51792_solr",
  "state":"active"},
"core_node20":{
  "core":"awhollynewcollection_0_shard1_replica3",
  "base_url":"http://127.0.0.1:51797/solr;,
  "node_name":"127.0.0.1:51797_solr",
  "state":"active",
  "leader":"true"}}},
"shard2":{
  "range":"b333-e665",
  "state":"active",
  "replicas":{"core_node8":{
  "core":"awhollynewcollection_0_shard2_replica4",
  "base_url":"http://127.0.0.1:51792/solr;,
  "node_name":"127.0.0.1:51792_solr",
  "state":"active",
  "leader":"true"}}},
"shard3":{
  "range":"e666-1998",
  "state":"active",
  "replicas":{
"core_node14":{
  "core":"awhollynewcollection_0_shard3_replica4",
  "base_url":"http://127.0.0.1:51792/solr;,
  "node_name":"127.0.0.1:51792_solr",
  "state":"active",
  "leader":"true"},
"core_node16":{
  "core":"awhollynewcollection_0_shard3_replica2",
  "base_url":"http://127.0.0.1:51796/solr;,
  "node_name":"127.0.0.1:51796_solr",
  "state":"active"}}},
"shard4":{
  "range":"1999-4ccb",
  "state":"active",
  "replicas":{"core_node9":{
  "core":"awhollynewcollection_0_shard4_replica1",
  "base_url":"http://127.0.0.1:51790/solr;,
  "node_name":"127.0.0.1:51790_solr",
  "state":"active",
  "leader":"true"}}},
"shard5":{
  "range":"4ccc-7fff",
  "state":"active",
  "replicas":{"core_node4":{
  "core":"awhollynewcollection_0_shard5_replica4",
  "base_url":"http://127.0.0.1:51792/solr;,
  "node_name":"127.0.0.1:51792_solr",
  "state":"active",
  "leader":"true",
  "router":{"name":"compositeId"},
  

[jira] [Updated] (LUCENE-7808) PayloadScoreQuery and SpanPayloadCheckQuery have incomplete .equals and .hashCode methods

[Erik Hatcher (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Hatcher updated LUCENE-7808:
-
Attachment: LUCENE-7808.patch

(initially failing) tests and implementation fixes

> PayloadScoreQuery and SpanPayloadCheckQuery have incomplete .equals and 
> .hashCode methods
> -
>
> Key: LUCENE-7808
> URL: https://issues.apache.org/jira/browse/LUCENE-7808
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Erik Hatcher
>Assignee: Erik Hatcher
> Fix For: master (7.0), 6.6
>
> Attachments: LUCENE-7808.patch
>
>
> Both PayloadScoreQuery and SpanPayloadCheckQuery have incomplete .equals and 
> .hashCode methods, causing caching issues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-7808) PayloadScoreQuery and SpanPayloadCheckQuery have incomplete .equals and .hashCode methods

[Erik Hatcher (JIRA)]

Erik Hatcher created LUCENE-7808:


 Summary: PayloadScoreQuery and SpanPayloadCheckQuery have 
incomplete .equals and .hashCode methods
 Key: LUCENE-7808
 URL: https://issues.apache.org/jira/browse/LUCENE-7808
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Erik Hatcher
Assignee: Erik Hatcher
 Fix For: master (7.0), 6.6


Both PayloadScoreQuery and SpanPayloadCheckQuery have incomplete .equals and 
.hashCode methods, causing caching issues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10522) Duplicate keys in "collations" object with JSON response format

[Zoran Avtarovski (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985771#comment-15985771
 ] 

Zoran Avtarovski commented on SOLR-10522:
-

The XML response works because of the way XML is processed. But the tags are 
still incorrectly identified as lst as opposed to the correct arr.

As you point out changing from a NamedList to an ArrayList will do the trick. 
The changes need to occur on the 
org.apache.solr.handler.component.SpellCheckComponent.java object at lines : 
302, 437 and 516

If you change the type to ArrayList and the associated add element code the 
issue appears to be resolved (with my limited testing)

> Duplicate keys in "collations" object with JSON response format
> ---
>
> Key: SOLR-10522
> URL: https://issues.apache.org/jira/browse/SOLR-10522
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: spellchecker
>Affects Versions: 6.5
>Reporter: Nikita Pchelintsev
>Priority: Minor
>
> After upgrading Solr 6.3 -> 6.5 I've noticed a change in how json response 
> writer outputs "collations" response key when spellchecking is enabled 
> (wt=json=arrarr)
> Solr 6.3:
> "collations":
> [
>   ["collation",{
>   "collationQuery":"the",
>   "hits":48,
>   "maxScore":"30.282",
>   "misspellingsAndCorrections":
>   [
> ["thea","the"]]}],
>   ["collation",{
>   "collationQuery":"tea",
>   "hits":3,
>   "maxScore":"2.936",
>   "misspellingsAndCorrections":
>   [
> ["thea","tea"]]}],
>   ...
> Solr 6.5:
> "collations":{
>   "collation":{
> "collationQuery":"the",
> "hits":43,
> "misspellingsAndCorrections":
> [
>   ["thea","the"]]},
>   "collation":{
> "collationQuery":"tea",
> "hits":3,
> "misspellingsAndCorrections":
> [
>   ["thea","tea"]]},
> ...
> Solr 6.5 outputs object instead of an array, and it has duplicate keys which 
> is not valid for JSON format.
> Any help is appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Looking for development docs.

[Erick Erickson]

Solr/Lucene is big. Really big. I'd think seriously about taking
something you're interested in/know about, finding a JIRA that you'd
like to work on and diving in. Plus there aren't very many
architecture docs.

Your characterization of the realms of responsibility is pretty accurate.

Have you seen: https://wiki.apache.org/solr/HowToContribute?

A somewhat painful but "safe" way to get your feet wet is to look at
the coverage reports on jenkins and see what code is not tested in the
junit tests and...write a test. At least I think the coverage reports
are still there.

Best,
Erick

On Wed, Apr 26, 2017 at 3:12 PM, David Lee  wrote:
> I'd like to have a better understanding of how much of Solr is unique to it
> versus directly extending Lucene.
>
> For example, I assume that sharding, replication, etc. is implemented in
> Solr where-as indexing, querying, etc. would be implemented by Lucene.
>
> I'm hoping to learn enough to be able to contribute at some point.
>
> Thanks,
>
> David
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SolrCloud "master mode" planned?

[Otis Gospodnetić]

Right, that "bin/solr zk start" is sort of how one could present that to
users.  I took the liberty of creating
https://issues.apache.org/jira/browse/SOLR-10573 after not being able to
find any such issues (yet hearing about such ideas at Lucene Revolution).

Ishan & Co, you may want to link other related issues or use e.g. "hideZK"
label and treat SOLR-10573 just as an umbrella?

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Wed, Apr 26, 2017 at 4:35 PM, Upayavira  wrote:

> I have done a *lot* of automating this. Redoing it recently it was quite
> embarrassing to realise how much complexity there is involved in it - it is
> crazy hard to get a basic, production ready SolrCloud setup running.
>
> One thing that is hard is getting a ZooKeeper ensemble going - using
> Exhibitor makes it much easier.
>
> Something that has often occurred to me is, why do we require people to go
> download a separate ZooKeeper, and work out how to install and configure
> it, when we have it embedded already? Why can't we just have a 'bin/solr zk
> start' command which starts an "embedded" zookeeper, but without Solr. To
> really make it neat, we offer some way (a la Exhibitor) for multiple
> concurrently started ZK nodes to autodiscover each other, then getting our
> three ZK nodes up won't be quite so treacherous.
>
> Just a thought.
>
> Upayavira
>
> On Wed, 26 Apr 2017, at 03:58 PM, Mike Drob wrote:
>
> Could the zk role also be guaranteed to run the Overseer (and no
> collections)? If we already have that separated out, it would make sense to
> put it with the embedded zk. I think you can already configure and place
> things manually this way, but it would be a huge win to package it all up
> nicely for users and set it to turnkey operation.
>
> I think it was a great improvement for deployment when we dropped tomcat,
> this is the next logical step.
>
> Mike
>
> On Wed, Apr 26, 2017, 4:22 AM Jan Høydahl  wrote:
>
> There have been suggestions to add a “node controller” process which again
> could start Solr and perhaps ZK on a node.
>
> But adding a new “zk” role which would let that node start (embedded) ZK I
> cannot recall. It would of course make a deploy simpler if ZK was hidden as
> a solr role/feature and perhaps assigned to N nodes, moved if needed etc.
> If I’m not mistaken ZK 3.5 would make such more dynamic setups easier but
> is currently in beta.
>
> Also, in these days of containers, I kind of like the concept of spinning
> up N ZK containers that the Solr containers connect to and let Kubernetes
> or whatever you use take care of placement, versions etc. So perhaps the
> need for a production-ready solr-managed zk is not as big as it used to be,
> or maybe even undesirable? For production Windows installs I could still
> clearly see a need though.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 25. apr. 2017 kl. 23.30 skrev Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com>:
>
> Hi Otis,
> I've been working on, and shall be working on, a few issues on the lines
> of "hide ZK".
>
> SOLR-6736: Uploading configsets can now be done through Solr nodes instead
> of uploading them to ZK.
> SOLR-10272: Use a _default configset, with the intention of not needing
> the user to bother about the concept of configsets unless he needs to
> SOLR-10446 (SOLR-9057): User can use CloudSolrClient without access to ZK
> SOLR-8440: Enabling BasicAuth security through bin/solr script
> Ability to edit security.json through the bin/solr script
> Having all this in place, and perhaps some more that I may be missing,
> should hopefully not need the user to know much about ZK.
>
> 1. Do you have suggestions on what more needs to be done for "hiding ZK"?
> 2. Do you have suggestions on how to track this overall theme of "hiding
> ZK"? Some of these issues I mentioned are associated with other epics, so I
> don't know if creating a "hiding ZK" epic and having these (and other
> issues) as sub-tasks is a good idea (maybe it is). Alternatively, how about
> tracking these (and other issues) using some label?
> Regards,
> Ishan
>
>
>
> On Wed, Apr 26, 2017 at 2:39 AM, Otis Gospodnetić <
> otis.gospodne...@gmail.com> wrote:
>
> Hi,
>
> This thread about Solr master-slave vs. SolrCloud deployment poll seems to
> point out people find SolrCloud (the ZK part of it) deployment complex:
>
> http://search-lucene.com/m/Solr/eHNlfm4WpJPVR92?subj=Re+
> Poll+Master+Slave+or+SolrCloud+
>
> It could be just how information is presented...
> ... or how ZK is exposed as something external, which it is...
>
> Are there plans to "hide ZK"?  Or maybe have the notion of master-only
> (not as in master-slave, but as in running ZK only, not hosting data) mode
> for SolrCloud nodes (a la ES)?
>
> I peeked at JIRA, but couldn't find anything about that, although I seem
> to recall 

[jira] [Created] (SOLR-10573) Hide ZooKeeper

[Otis Gospodnetic (JIRA)]

Otis Gospodnetic created SOLR-10573:
---

 Summary: Hide ZooKeeper
 Key: SOLR-10573
 URL: https://issues.apache.org/jira/browse/SOLR-10573
 Project: Solr
  Issue Type: New Feature
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Otis Gospodnetic


It may make sense to either embed ZK in Solr and allow running Solr instances 
with just ZK and no data or something else that hides ZK from Solr users...

Based on what the Solr poll that revealed lowish SolrCloud adoption and 
comments in 
http://search-lucene.com/m/Solr/eHNlm8wPIKJ3v51?subj=Poll+Master+Slave+or+SolrCloud
 that showed that people still find SolrCloud complex, at least partly because 
of the external ZK recommendation.

See also: 
http://search-lucene.com/m/Lucene/l6pAi11rBma0gNoI1?subj=SolrCloud+master+mode+planned+




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10272) Use a default configset and make the configName parameter optional.

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-10272:

Attachment: SOLR-10272.patch.gz

Adding another patch (needed to gzip it since data_driven_schema_configs has 
some international characters that my browser is struggling to upload / server 
is refusing to accept)

# Unit tests can use the _default.
# All existing tests that were looking for conf1, without even specifying the 
configset name during collection creation, now explicitly specify conf1.
# TODO: test the bin/solr.cmd (my windows box is unable to resolve ivy 
dependencies, getting stuck at io.dropwizard.metrics*.)
# If a configset exists with the same name as the collection, and no configName 
is specified during collection creation, then the configset (same name as 
collection) is overwritten by _default configset. This is another backcompat 
break.

Due to the backcompat breaks, I'm planning to commit this only to master/7.0, 
and skip branch_6x.

> Use a default configset and make the configName parameter optional.
> ---
>
> Key: SOLR-10272
> URL: https://issues.apache.org/jira/browse/SOLR-10272
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Ishan Chattopadhyaya
> Attachments: SOLR-10272.patch, SOLR-10272.patch.gz
>
>
> This Jira's motivation is to improve the creating a collection experience 
> better for users.
> To create a collection we need to specify a configName that needs to be 
> present in ZK. When a new user is starting Solr why should he worry about 
> having to know about configsets before he can can create a collection.
> When you create a collection using "bin/solr create" the script uploads a 
> configset and references it. This is great. We should extend this idea to API 
> users as well.
> So here is the rough outline of what I think we can do here:
> 1. When you start solr , the bin script checks to see if 
> "/configs/_baseConfigSet" znode is present . If not it uploads the 
> "basic_configs". 
> We can discuss if its the "basic_configs" or something other default config 
> set. 
> Also we can discuss the name for "/_baseConfigSet". Moving on though
> 2. When a user creates a collection from the API  
> {{admin/collections?action=CREATE=gettingstarted}} here is what we do :
> Use https://cwiki.apache.org/confluence/display/solr/ConfigSets+API to copy 
> over the default config set to a configset with the name of the collection 
> specified.
> collection.configName can truly be an optional parameter. If its specified we 
> don't need to do this step.
> 3. Have the bin scripts use this and remove the logic built in there to do 
> the same thing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Looking for development docs.

[David Lee]

I'd like to have a better understanding of how much of Solr is unique to 
it versus directly extending Lucene.


For example, I assume that sharding, replication, etc. is implemented in 
Solr where-as indexing, querying, etc. would be implemented by Lucene.


I'm hoping to learn enough to be able to contribute at some point.

Thanks,

David



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9555) Leader incorrectly publishes state for replica when it puts replica into LIR.

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated SOLR-9555:

Attachment: SOLR-9555.patch

Cleans up patch to pass {{ant precommit}}

> Leader incorrectly publishes state for replica when it puts replica into LIR.
> -
>
> Key: SOLR-9555
> URL: https://issues.apache.org/jira/browse/SOLR-9555
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
> Attachments: SOLR-9555.patch, SOLR-9555.patch, SOLR-9555.patch, 
> SOLR-9555-WIP-2.patch, SOLR-9555-WIP-3.patch, SOLR-9555-WIP.patch
>
>
> See 
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17888/consoleFull 
> for an example



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10521) sort by string field of the nested child when searching with {!parent}

[Mikhail Khludnev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Khludnev updated SOLR-10521:

Attachment: SOLR-10521.patch

[^SOLR-10521.patch] ByteRefs from Comparator are converted to byte[] and then 
unmarshalled back explicitly. Yet another one spaghetti. It seems like 
no-commit to me.  

> sort by string field of the nested child when searching with {!parent}
> --
>
> Key: SOLR-10521
> URL: https://issues.apache.org/jira/browse/SOLR-10521
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mikhail Khludnev
> Attachments: SOLR-10521.patch, SOLR-10521.patch, SOLR-10521.patch, 
> SOLR-10521-raw.patch
>
>
> The idea is to integrate Lucene's {{ToParentBlockJoinSortField}} 
> The approach to hookup it is a little bit tricky:
> {{sort=\{!childfield bjq=$q field=COLOR_s}desc}}
> the question no.1 wdyt about the syntax? 
> internally it creates a function query with valueSource which produces 
> {{ToParentBlockJoinSortField}} 
> The main challenge is picking Solr field type from  
> {{ToParentBlockJoinSortField}}, because as-is it's broken on {{mergeIds}} - 
> ByteRefs need to be properly marshared and unmarshalled by a field type from 
> child scope. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10295) Decide online location for Ref Guide HTML pages

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985637#comment-15985637
 ] 

Hoss Man commented on SOLR-10295:
-

bq. Second, we have no index.html in the set of files that are generated for 
the webserver to default to if someone goes to just 
https://lucene.apache.org/solr/guide/test-10290, for example. You just get a 
listing of files under that path. We either need to do some redirect magic in 
.htaccess, or change the "home" page of the Guide to be named index.html (which 
has a bunch of other implications to the build process since the page 
"apache-solr-reference-guide.adoc is considered the super-parent of every other 
page of the Guide).

I think I brought this up in an github issue in the old POC repo but we never 
discussed it much...

It would be fairly simple to change the "one time cwiki->asciidoc" code to use 
{{index.adoc}} as the (created) name for the page currently named 
{{apache-solr-reference-guide.adoc}} if that's what we want to -- but that 
would mean all the pages that refer to it as a parent / link to it would have 
to refer to it as {{index}} not {{apache-solr-reference-guide}} (this would 
happen automatically for the "one time cwiki->asciidoc" converted 
links/references).

Alternatively, a {{DirectoryIndex}} entry (or rewrite rule) in an {{.htaccess}} 
file would also be easy to set up (but if we ever expect to ship a ZIPed up 
copy of the directory that people might open on their desktops, we'll 
definitely want an {{index.html}} file there -- even if it's just a link to 
{{apache-solr-reference-guide.adoc}})

> Decide online location for Ref Guide HTML pages
> ---
>
> Key: SOLR-10295
> URL: https://issues.apache.org/jira/browse/SOLR-10295
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>
> One of the biggest decisions we need to make is where to put the new Solr Ref 
> Guide. Confluence at least had the whole web-hosting bits figured out; we 
> have to figure that out on our own.
> An obvious (maybe only to me) choice is to integrate the Ref Guide with the 
> Solr Website. However, due to the size of the Solr Ref Guide (nearly 200 
> pages), I believe trying to publish it solely with existing CMS tools will 
> create problems similar to those described in the Lucene ReleaseTodo when it 
> comes to publishing the Lucene/Solr javadocs (see 
> https://wiki.apache.org/lucene-java/ReleaseTodo#Website_.2B-.3D_javadocs).
> A solution exists already, and it's what is done for the javadocs. From the 
> above link:
> {quote}
> The solution: skip committing javadocs to the source tree, then staging, then 
> publishing, and instead commit javadocs directly to the production tree. 
> Ordinarily this would be problematic, because the CMS wants to keep the 
> production tree in sync with the staging tree, so anything it finds in the 
> production tree that's not in the staging tree gets nuked. However, the CMS 
> has a built-in mechanism to allow exceptions to the 
> keep-production-in-sync-with-staging rule: extpaths.txt.
> {quote}
> This solution (for those who don't know already) is to provide a static text 
> file (extpaths.txt) that includes the javadoc paths that should be presented 
> in production, but which won't exist in CMS staging environments. This way, 
> we can publish HTML files directly to production and they will be preserved 
> when the staging-production trees are synced.
> The rest of the process would be quite similar to what is documented in the 
> ReleaseTodo in sections following the link above - use SVN to update the CMS 
> production site and update extpaths.txt properly. We'd do this in the 
> {{solr}} section of the CMS obviously, and not the {{lucene}} section.
> A drawback to this approach is that we won't have a staging area to view the 
> Guide before publication. Files would be generated and go to production 
> directly. We may want to put a process in place to give some additional 
> confidence that things look right first (someone's people.apache.org 
> directory? a pre-pub validation script that tests...something...?), and agree 
> on what we'd be voting on when a vote to release comes up. However, the CMS 
> is pretty much the only option that I can think of...other ideas are welcome 
> if they might work.
> We also need to agree on URL paths that make sense, considering we'll have a 
> new "site" for each major release - something like 
> {{http://lucene.apache.org/solr/ref-guide/6_1}} might work? Other thoughts 
> are welcome on this point also.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: 

[jira] [Commented] (SOLR-7759) DebugComponent's explain should be implemented as a distributed query

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985609#comment-15985609
 ] 

Alessandro Benedetti commented on SOLR-7759:


UP, any feedback ? [~markus17] ?

> DebugComponent's explain should be implemented as a distributed query
> -
>
> Key: SOLR-7759
> URL: https://issues.apache.org/jira/browse/SOLR-7759
> Project: Solr
>  Issue Type: Bug
>Reporter: Varun Thacker
> Attachments: SOLR_7759.patch
>
>
> Currently when we use debugQuery to see the explanation of the matched 
> documents, the query fired to get the statistics for the matched documents is 
> not a distributed query.
> This is a problem when using distributed idf. The actual documents are being 
> scored using the global stats and not per shard stats , but the explain will 
> show us the score by taking into account the stats from the shard where the 
> document belongs to.
> We should try to implement the explain query as a distributed request so that 
> the scores match the actual document scores.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SolrCloud "master mode" planned?

[Upayavira]

On Wed, 26 Apr 2017, at 10:06 PM, David Smiley wrote:
> 
>> On Apr 26, 2017, at 4:35 PM, Upayavira  wrote:
>> 
>> I have done a *lot* of automating this. Redoing it recently it was
>> quite embarrassing to realise how much complexity there is involved
>> in it - it is crazy hard to get a basic, production ready SolrCloud
>> setup running.> 
> Would you mind enumerating a list of what sort of issues you ran into
> deploying ZooKeeper in a production config?  A quick draft list of
> sorts just to get a sense of what sort of stuff generally you had to
> contend with.  I recently did it in a Docker/Kontena infrastructure.
> I did not find it to be hard; maybe medium :-).  I got the nodes
> working out of the box with minimal effort but had to make changes to
> harden it.> * I found the existing official Docker image for Zookeeper 
> lacking in
>   that I couldn't easily specify the "auto purge" settings, which
>   default to no purging which is unacceptable.> * I set 
> "-XX:+CrashOnOutOfMemoryError" so that the process would end
>   when an OOM occurs so that Kontena (Docker orchestrator) would
>   notice its down so it could restart it (a rare event obviously).
>   Users not using a container environment might not care about this I
>   guess.  This was merely a configuration setting; no Docker image
>   hack needed.> * I also ensured I used the latest ZK 3.4.6 release I 
> recall 3.4.4
>   (or maybe even 3.4.5?) cached DNS entries without re-looking up if
>   it failed which is particularly problematic in a container
>   environment where it's common for services to get a new IP when they
>   are restarted.  Thankfully I did not learn that issue the hard way;
>   I recall a blog warning of this issue by Shalin or Martijn Koster.
>   No action from me here other than ensuring I used an appropriate new
>   version.  Originally out of laziness I used Confluent's Docker image
>   but I knew I would have to switch because of this issue.
I used it for a test case for an app I built when learning more about
deployments. It is all on github at
http://github.com/odoko-devops/uberstack. There's an example there in
examples/apache-solr. I gave up on that effort because (a) I was making
it more complex than it needed to be and (b) I couldn't compete with the
big guns in the devops industry.
What I needed was to have three ZooKeeper nodes start up and
autodiscover each other. That, I handled with Exhibitor (from Netflix).
They provide a (not-production ready!!) Docker image that, whilst it
takes a few minutes, does result in ZK nodes that are working as an
ensemble, when none of them know about each other to start. It requires
an S3 bucket to provide the co-ordination.
The real lesson was "don't fail, retry". I built a wrapper around Solr
so that if ZK wasn't available (and in the correct ensemble) Solr
wouldn't start yet. It just kept retrying. Similarly, In
https://github.com/odoko-devops/solr-utils there's a tool that, when
containerised, will allow you to create a chroot (must be done after ZK
but before Solr), to upload configs (must be done after Solr, but
before collection creation), create a collection (only once configs
present), etc.
It made use of Rancher to provide an overlay network with DNS used for
service discovery - the ZK nodes were accessible to Solr via the
hostname "zookeeper", so Solr didn't have to do anything fancy in order
to find them.
The outcome of this was a reliable one-click install of three ZK nodes,
three Solr cloud nodes, and indexed content and a webapp showing the
content. Was pretty cool.
Or should I say, the outcome was cool, the process to get there
was painful.
Happy to share more of this if it is useful.

Upayavira

>> One thing that is hard is getting a ZooKeeper ensemble going - using
>> Exhibitor makes it much easier.>> 
>> Something that has often occurred to me is, why do we require people
>> to go download a separate ZooKeeper, and work out how to install and
>> configure it, when we have it embedded already? Why can't we just
>> have a 'bin/solr zk start' command which starts an "embedded"
>> zookeeper, but without Solr. To really make it neat, we offer some
>> way (a la Exhibitor) for multiple concurrently started ZK nodes to
>> autodiscover each other, then getting our three ZK nodes up won't be
>> quite so treacherous.> 
> I've often thought the same -- why not just embed it.  People say it's
> not a "production config" but this is only because we all keep telling
> us this is in an echo chamber and we believe ourselves :-P> 
> ~ David
> 
>> 
>> On Wed, 26 Apr 2017, at 03:58 PM, Mike Drob wrote:
>>> Could the zk role also be guaranteed to run the Overseer (and no
>>> collections)? If we already have that separated out, it would make
>>> sense to put it with the embedded zk. I think you can already
>>> configure and place things manually this way, but it would be a huge
>>> win to package it all up nicely for users and set it to turnkey
>>> 

Re: SolrCloud "master mode" planned?

[David Smiley]

> On Apr 26, 2017, at 4:35 PM, Upayavira  wrote:
> 
> I have done a *lot* of automating this. Redoing it recently it was quite 
> embarrassing to realise how much complexity there is involved in it - it is 
> crazy hard to get a basic, production ready SolrCloud setup running.

Would you mind enumerating a list of what sort of issues you ran into deploying 
ZooKeeper in a production config?  A quick draft list of sorts just to get a 
sense of what sort of stuff generally you had to contend with.  I recently did 
it in a Docker/Kontena infrastructure.  I did not find it to be hard; maybe 
medium :-).  I got the nodes working out of the box with minimal effort but had 
to make changes to harden it.
* I found the existing official Docker image for Zookeeper lacking in that I 
couldn't easily specify the "auto purge" settings, which default to no purging 
which is unacceptable.
* I set "-XX:+CrashOnOutOfMemoryError" so that the process would end when an 
OOM occurs so that Kontena (Docker orchestrator) would notice its down so it 
could restart it (a rare event obviously).  Users not using a container 
environment might not care about this I guess.  This was merely a configuration 
setting; no Docker image hack needed.  
* I also ensured I used the latest ZK 3.4.6 release I recall 3.4.4 (or 
maybe even 3.4.5?) cached DNS entries without re-looking up if it failed which 
is particularly problematic in a container environment where it's common for 
services to get a new IP when they are restarted.  Thankfully I did not learn 
that issue the hard way; I recall a blog warning of this issue by Shalin or 
Martijn Koster.  No action from me here other than ensuring I used an 
appropriate new version.  Originally out of laziness I used Confluent's Docker 
image but I knew I would have to switch because of this issue.

> One thing that is hard is getting a ZooKeeper ensemble going - using 
> Exhibitor makes it much easier.
> 
> Something that has often occurred to me is, why do we require people to go 
> download a separate ZooKeeper, and work out how to install and configure it, 
> when we have it embedded already? Why can't we just have a 'bin/solr zk 
> start' command which starts an "embedded" zookeeper, but without Solr. To 
> really make it neat, we offer some way (a la Exhibitor) for multiple 
> concurrently started ZK nodes to autodiscover each other, then getting our 
> three ZK nodes up won't be quite so treacherous.

I've often thought the same -- why not just embed it.  People say it's not a 
"production config" but this is only because we all keep telling us this is in 
an echo chamber and we believe ourselves :-P

~ David

> 
> On Wed, 26 Apr 2017, at 03:58 PM, Mike Drob wrote:
>> Could the zk role also be guaranteed to run the Overseer (and no 
>> collections)? If we already have that separated out, it would make sense to 
>> put it with the embedded zk. I think you can already configure and place 
>> things manually this way, but it would be a huge win to package it all up 
>> nicely for users and set it to turnkey operation.
>> 
>> I think it was a great improvement for deployment when we dropped tomcat, 
>> this is the next logical step.
>> 
>> Mike
>> 
>> On Wed, Apr 26, 2017, 4:22 AM Jan Høydahl > > wrote:
>> There have been suggestions to add a “node controller” process which again 
>> could start Solr and perhaps ZK on a node.
>> 
>> But adding a new “zk” role which would let that node start (embedded) ZK I 
>> cannot recall. It would of course make a deploy simpler if ZK was hidden as 
>> a solr role/feature and perhaps assigned to N nodes, moved if needed etc. If 
>> I’m not mistaken ZK 3.5 would make such more dynamic setups easier but is 
>> currently in beta.
>> 
>> Also, in these days of containers, I kind of like the concept of spinning up 
>> N ZK containers that the Solr containers connect to and let Kubernetes or 
>> whatever you use take care of placement, versions etc. So perhaps the need 
>> for a production-ready solr-managed zk is not as big as it used to be, or 
>> maybe even undesirable? For production Windows installs I could still 
>> clearly see a need though.
>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com 
>> 
>>> 25. apr. 2017 kl. 23.30 skrev Ishan Chattopadhyaya 
>>> >:
>>> 
>>> Hi Otis,
>>> I've been working on, and shall be working on, a few issues on the lines of 
>>> "hide ZK".
>>> 
>>> SOLR-6736: Uploading configsets can now be done through Solr nodes instead 
>>> of uploading them to ZK.
>>> SOLR-10272: Use a _default configset, with the intention of not needing the 
>>> user to bother about the concept of configsets unless he needs to
>>> SOLR-10446 (SOLR-9057): User can use CloudSolrClient without access to ZK
>>> SOLR-8440: Enabling BasicAuth security through 

[jira] [Commented] (SOLR-10295) Decide online location for Ref Guide HTML pages

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985544#comment-15985544
 ] 

Cassandra Targett commented on SOLR-10295:
--

bq. Oh, I see you have a TODO for this: "figure out process for updating guide"

heh, that wasn't what I meant, actually. That was supposed to be something 
about a process for updating the landing page that I completely munged. 
Correctly.

I added a new line for what I really meant and pushed a new version of the file 
to the branch.

> Decide online location for Ref Guide HTML pages
> ---
>
> Key: SOLR-10295
> URL: https://issues.apache.org/jira/browse/SOLR-10295
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>
> One of the biggest decisions we need to make is where to put the new Solr Ref 
> Guide. Confluence at least had the whole web-hosting bits figured out; we 
> have to figure that out on our own.
> An obvious (maybe only to me) choice is to integrate the Ref Guide with the 
> Solr Website. However, due to the size of the Solr Ref Guide (nearly 200 
> pages), I believe trying to publish it solely with existing CMS tools will 
> create problems similar to those described in the Lucene ReleaseTodo when it 
> comes to publishing the Lucene/Solr javadocs (see 
> https://wiki.apache.org/lucene-java/ReleaseTodo#Website_.2B-.3D_javadocs).
> A solution exists already, and it's what is done for the javadocs. From the 
> above link:
> {quote}
> The solution: skip committing javadocs to the source tree, then staging, then 
> publishing, and instead commit javadocs directly to the production tree. 
> Ordinarily this would be problematic, because the CMS wants to keep the 
> production tree in sync with the staging tree, so anything it finds in the 
> production tree that's not in the staging tree gets nuked. However, the CMS 
> has a built-in mechanism to allow exceptions to the 
> keep-production-in-sync-with-staging rule: extpaths.txt.
> {quote}
> This solution (for those who don't know already) is to provide a static text 
> file (extpaths.txt) that includes the javadoc paths that should be presented 
> in production, but which won't exist in CMS staging environments. This way, 
> we can publish HTML files directly to production and they will be preserved 
> when the staging-production trees are synced.
> The rest of the process would be quite similar to what is documented in the 
> ReleaseTodo in sections following the link above - use SVN to update the CMS 
> production site and update extpaths.txt properly. We'd do this in the 
> {{solr}} section of the CMS obviously, and not the {{lucene}} section.
> A drawback to this approach is that we won't have a staging area to view the 
> Guide before publication. Files would be generated and go to production 
> directly. We may want to put a process in place to give some additional 
> confidence that things look right first (someone's people.apache.org 
> directory? a pre-pub validation script that tests...something...?), and agree 
> on what we'd be voting on when a vote to release comes up. However, the CMS 
> is pretty much the only option that I can think of...other ideas are welcome 
> if they might work.
> We also need to agree on URL paths that make sense, considering we'll have a 
> new "site" for each major release - something like 
> {{http://lucene.apache.org/solr/ref-guide/6_1}} might work? Other thoughts 
> are welcome on this point also.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10489) StatsReloadRaceTest.testParallelReloadAndStats failures

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985543#comment-15985543
 ] 

Mikhail Khludnev commented on SOLR-10489:
-

now this test is hard to break. 

> StatsReloadRaceTest.testParallelReloadAndStats failures
> ---
>
> Key: SOLR-10489
> URL: https://issues.apache.org/jira/browse/SOLR-10489
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
>Reporter: Andrzej Bialecki 
>Assignee: Erick Erickson
>
> This test has been failing a lot after the changes in SOLR-9959, for unclear 
> reasons. The failure is always in the same place:
> {code}
> java.lang.AssertionError: Key SEARCHER.searcher.indexVersion not found in 
> registry solr.core.collection1
>   at 
> __randomizedtesting.SeedInfo.seed([28B54D77FD0E3DF1:E72B284E72FF55AE]:0)
>   at org.junit.Assert.fail(Assert.java:93)
>   at org.junit.Assert.assertTrue(Assert.java:43)
>   at 
> org.apache.solr.handler.admin.StatsReloadRaceTest.requestMetrics(StatsReloadRaceTest.java:132)
>   at 
> org.apache.solr.handler.admin.StatsReloadRaceTest.testParallelReloadAndStats(StatsReloadRaceTest.java:70)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10295) Decide online location for Ref Guide HTML pages

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985509#comment-15985509
 ] 

Cassandra Targett commented on SOLR-10295:
--

bq. I plan to try this process out with the content on the jira/solr-10290 
branch

This went without a hitch: 
https://lucene.apache.org/solr/guide/test-10290/apache-solr-reference-guide.html.
 A cool milestone after this much work.

(We'll be able to remove the path at a later time and the content should be 
automatically deleted on the next staging-production sync. If it is not, I'll 
remove it manually. Of course, this shouldn't be announced as live in any way 
to any end users, since it's just a test.)

A couple things I thought of while doing this:

First, we maybe want to have a permanent path {{solr/guide}} that has an 
index.html file in it for directing people to the released HTML versions and 
official PDFs. The update to extpaths.txt would work the same, just there is a 
file there that is a landing page.

Second, we have no index.html in the set of files that are generated for the 
webserver to default to if someone goes to just 
https://lucene.apache.org/solr/guide/test-10290, for example. You just get a 
listing of files under that path. We either need to do some redirect magic in 
.htaccess, or change the "home" page of the Guide to be named index.html (which 
has a bunch of other implications to the build process since the page 
"apache-solr-reference-guide.adoc is considered the super-parent of every other 
page of the Guide).

> Decide online location for Ref Guide HTML pages
> ---
>
> Key: SOLR-10295
> URL: https://issues.apache.org/jira/browse/SOLR-10295
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>
> One of the biggest decisions we need to make is where to put the new Solr Ref 
> Guide. Confluence at least had the whole web-hosting bits figured out; we 
> have to figure that out on our own.
> An obvious (maybe only to me) choice is to integrate the Ref Guide with the 
> Solr Website. However, due to the size of the Solr Ref Guide (nearly 200 
> pages), I believe trying to publish it solely with existing CMS tools will 
> create problems similar to those described in the Lucene ReleaseTodo when it 
> comes to publishing the Lucene/Solr javadocs (see 
> https://wiki.apache.org/lucene-java/ReleaseTodo#Website_.2B-.3D_javadocs).
> A solution exists already, and it's what is done for the javadocs. From the 
> above link:
> {quote}
> The solution: skip committing javadocs to the source tree, then staging, then 
> publishing, and instead commit javadocs directly to the production tree. 
> Ordinarily this would be problematic, because the CMS wants to keep the 
> production tree in sync with the staging tree, so anything it finds in the 
> production tree that's not in the staging tree gets nuked. However, the CMS 
> has a built-in mechanism to allow exceptions to the 
> keep-production-in-sync-with-staging rule: extpaths.txt.
> {quote}
> This solution (for those who don't know already) is to provide a static text 
> file (extpaths.txt) that includes the javadoc paths that should be presented 
> in production, but which won't exist in CMS staging environments. This way, 
> we can publish HTML files directly to production and they will be preserved 
> when the staging-production trees are synced.
> The rest of the process would be quite similar to what is documented in the 
> ReleaseTodo in sections following the link above - use SVN to update the CMS 
> production site and update extpaths.txt properly. We'd do this in the 
> {{solr}} section of the CMS obviously, and not the {{lucene}} section.
> A drawback to this approach is that we won't have a staging area to view the 
> Guide before publication. Files would be generated and go to production 
> directly. We may want to put a process in place to give some additional 
> confidence that things look right first (someone's people.apache.org 
> directory? a pre-pub validation script that tests...something...?), and agree 
> on what we'd be voting on when a vote to release comes up. However, the CMS 
> is pretty much the only option that I can think of...other ideas are welcome 
> if they might work.
> We also need to agree on URL paths that make sense, considering we'll have a 
> new "site" for each major release - something like 
> {{http://lucene.apache.org/solr/ref-guide/6_1}} might work? Other thoughts 
> are welcome on this point also.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: 

Re: SolrCloud "master mode" planned?

[Upayavira]

I have done a *lot* of automating this. Redoing it recently it was quite
embarrassing to realise how much complexity there is involved in it - it
is crazy hard to get a basic, production ready SolrCloud setup running.
One thing that is hard is getting a ZooKeeper ensemble going - using
Exhibitor makes it much easier.
Something that has often occurred to me is, why do we require people to
go download a separate ZooKeeper, and work out how to install and
configure it, when we have it embedded already? Why can't we just have a
'bin/solr zk start' command which starts an "embedded" zookeeper, but
without Solr. To really make it neat, we offer some way (a la Exhibitor)
for multiple concurrently started ZK nodes to autodiscover each other,
then getting our three ZK nodes up won't be quite so treacherous.
Just a thought.

Upayavira

On Wed, 26 Apr 2017, at 03:58 PM, Mike Drob wrote:
> Could the zk role also be guaranteed to run the Overseer (and no
> collections)? If we already have that separated out, it would make
> sense to put it with the embedded zk. I think you can already
> configure and place things manually this way, but it would be a huge
> win to package it all up nicely for users and set it to turnkey
> operation.> 
> I think it was a great improvement for deployment when we dropped
> tomcat, this is the next logical step.> 
> Mike
> 
> On Wed, Apr 26, 2017, 4:22 AM Jan Høydahl
>  wrote:>> There have been suggestions to add a “node 
> controller” process which
>> again could start Solr and perhaps ZK on a node.>> 
>> But adding a new “zk” role which would let that node start (embedded)
>> ZK I cannot recall. It would of course make a deploy simpler if ZK
>> was hidden as a solr role/feature and perhaps assigned to N nodes,
>> moved if needed etc. If I’m not mistaken ZK 3.5 would make such more
>> dynamic setups easier but is currently in beta.>> 
>> Also, in these days of containers, I kind of like the concept of
>> spinning up N ZK containers that the Solr containers connect to and
>> let Kubernetes or whatever you use take care of placement, versions
>> etc. So perhaps the need for a production-ready solr-managed zk is
>> not as big as it used to be, or maybe even undesirable? For
>> production Windows installs I could still clearly see a need though.>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> 
>>> 25. apr. 2017 kl. 23.30 skrev Ishan Chattopadhyaya
>>> :>>> 
>>> Hi Otis,
>>> I've been working on, and shall be working on, a few issues on the
>>> lines of "hide ZK".>>> 
>>> SOLR-6736: Uploading configsets can now be done through Solr nodes
>>> instead of uploading them to ZK.>>> SOLR-10272: Use a _default configset, 
>>> with the intention of not
>>> needing the user to bother about the concept of configsets unless he
>>> needs to>>> SOLR-10446 (SOLR-9057): User can use CloudSolrClient without
>>> access to ZK>>> SOLR-8440: Enabling BasicAuth security through bin/solr 
>>> script
>>> Ability to edit security.json through the bin/solr script
>>> Having all this in place, and perhaps some more that I may be
>>> missing, should hopefully not need the user to know much about ZK.>>> 
>>> 1. Do you have suggestions on what more needs to be done for "hiding
>>>ZK"?>>> 2. Do you have suggestions on how to track this overall theme of
>>>"hiding ZK"? Some of these issues I mentioned are associated with
>>>other epics, so I don't know if creating a "hiding ZK" epic and
>>>having these (and other issues) as sub-tasks is a good idea
>>>(maybe it is). Alternatively, how about tracking these (and other
>>>issues) using some label?>>> Regards,
>>> Ishan
>>> 
>>> 
>>> 
>>> On Wed, Apr 26, 2017 at 2:39 AM, Otis Gospodnetić
>>>  wrote: Hi,
 
 This thread about Solr master-slave vs. SolrCloud deployment poll
 seems to point out people find SolrCloud (the ZK part of it)
 deployment complex: 
 http://search-lucene.com/m/Solr/eHNlfm4WpJPVR92?subj=Re+Poll+Master+Slave+or+SolrCloud+
  
 It could be just how information is presented...
 ... or how ZK is exposed as something external, which it is...
 
 Are there plans to "hide ZK"?  Or maybe have the notion of master-
 only (not as in master-slave, but as in running ZK only, not
 hosting data) mode for SolrCloud nodes (a la ES)? 
 I peeked at JIRA, but couldn't find anything about that, although I
 seem to recall some mention of embedding ZK to make things easier
 for SolrCloud users.  I think I saw that at some Lucene Revolution
 talk? 
 Thanks,
 Otis
 --
 Monitoring - Log Management - Alerting - Anomaly Detection
 Solr & Elasticsearch Consulting Support Training -
 http://sematext.com/ 

[jira] [Commented] (SOLR-10521) sort by string field of the nested child when searching with {!parent}

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985491#comment-15985491
 ] 

Mikhail Khludnev commented on SOLR-10521:
-

Here is the problem. QueryComponent needs to decide about a field type to 
[marshal|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L664]
 and 
[unmarshal|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L1250]
 sort values.  
* it works as expected when {{sort=name asc}}. The correct field type is put to 
{{SortSpec.fields}} 
[here|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/SortSpecParsing.java#L203].
 And then it's used to marshall {{ByteRef}} to {{String}} and back.
* it works surprisingly fine when sorting by function like {{sort=sum(age, 42) 
asc}}. In this case {{SortSpecParsing}} [puts 
null|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/SortSpecParsing.java#L154]
 into {{SortSpec.fields}}. And then this null disables any marshaling that 
allows passing doubles and ints through javabin serialization (I care about 
SolrCloud), but it makes impossible to have a sort field returning 
{{BytesRef}}, because they are grabled with javabin. For example 
{{sort=field(name) asc}} doesn't work.
Do you have an idea how to introduce {{sort=childfield(name) asc}}, without 
giving a lobotomy to -myself- QueryComponent?  



> sort by string field of the nested child when searching with {!parent}
> --
>
> Key: SOLR-10521
> URL: https://issues.apache.org/jira/browse/SOLR-10521
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mikhail Khludnev
> Attachments: SOLR-10521.patch, SOLR-10521.patch, SOLR-10521-raw.patch
>
>
> The idea is to integrate Lucene's {{ToParentBlockJoinSortField}} 
> The approach to hookup it is a little bit tricky:
> {{sort=\{!childfield bjq=$q field=COLOR_s}desc}}
> the question no.1 wdyt about the syntax? 
> internally it creates a function query with valueSource which produces 
> {{ToParentBlockJoinSortField}} 
> The main challenge is picking Solr field type from  
> {{ToParentBlockJoinSortField}}, because as-is it's broken on {{mergeIds}} - 
> ByteRefs need to be properly marshared and unmarshalled by a field type from 
> child scope. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10295) Decide online location for Ref Guide HTML pages

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985489#comment-15985489
 ] 

Steve Rowe commented on SOLR-10295:
---

{{solr-ref-guide/meta-docs/publish.adoc}} looks good to me, with one caveat: 
use of {{svn import}} assumes that the entire destination directory does not 
already exist.  When we re-publish a previously published version, though, a 
different flow will be required: check out (an appropriate subtree of) the CMS 
production tree; use {{svn add}}/{{svn rm}} for file/dir additions/deletions; 
and {{svn commit}}.

Oh, I see you have a TODO for this: "figure out process for updating guide"

> Decide online location for Ref Guide HTML pages
> ---
>
> Key: SOLR-10295
> URL: https://issues.apache.org/jira/browse/SOLR-10295
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>
> One of the biggest decisions we need to make is where to put the new Solr Ref 
> Guide. Confluence at least had the whole web-hosting bits figured out; we 
> have to figure that out on our own.
> An obvious (maybe only to me) choice is to integrate the Ref Guide with the 
> Solr Website. However, due to the size of the Solr Ref Guide (nearly 200 
> pages), I believe trying to publish it solely with existing CMS tools will 
> create problems similar to those described in the Lucene ReleaseTodo when it 
> comes to publishing the Lucene/Solr javadocs (see 
> https://wiki.apache.org/lucene-java/ReleaseTodo#Website_.2B-.3D_javadocs).
> A solution exists already, and it's what is done for the javadocs. From the 
> above link:
> {quote}
> The solution: skip committing javadocs to the source tree, then staging, then 
> publishing, and instead commit javadocs directly to the production tree. 
> Ordinarily this would be problematic, because the CMS wants to keep the 
> production tree in sync with the staging tree, so anything it finds in the 
> production tree that's not in the staging tree gets nuked. However, the CMS 
> has a built-in mechanism to allow exceptions to the 
> keep-production-in-sync-with-staging rule: extpaths.txt.
> {quote}
> This solution (for those who don't know already) is to provide a static text 
> file (extpaths.txt) that includes the javadoc paths that should be presented 
> in production, but which won't exist in CMS staging environments. This way, 
> we can publish HTML files directly to production and they will be preserved 
> when the staging-production trees are synced.
> The rest of the process would be quite similar to what is documented in the 
> ReleaseTodo in sections following the link above - use SVN to update the CMS 
> production site and update extpaths.txt properly. We'd do this in the 
> {{solr}} section of the CMS obviously, and not the {{lucene}} section.
> A drawback to this approach is that we won't have a staging area to view the 
> Guide before publication. Files would be generated and go to production 
> directly. We may want to put a process in place to give some additional 
> confidence that things look right first (someone's people.apache.org 
> directory? a pre-pub validation script that tests...something...?), and agree 
> on what we'd be voting on when a vote to release comes up. However, the CMS 
> is pretty much the only option that I can think of...other ideas are welcome 
> if they might work.
> We also need to agree on URL paths that make sense, considering we'll have a 
> new "site" for each major release - something like 
> {{http://lucene.apache.org/solr/ref-guide/6_1}} might work? Other thoughts 
> are welcome on this point also.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9555) Leader incorrectly publishes state for replica when it puts replica into LIR.

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985465#comment-15985465
 ] 

Mike Drob commented on SOLR-9555:
-

Noticed that on that check if it hits the follower it gets 2 docs, and if it 
hits the leader it gets 0. This seems pretty consistent. Adding a 10s sleep 
like suggested in SOLR-10562 doesn't fix the problem. Based on this, it feels 
like it _should_ fail 50% of the time, but I'm definitely not seeing that.

The good news is that that is the only failure I'm seeing in {{solr/core}} 
tests now, so I'm hoping it's a test issue and not a code issue for me.

> Leader incorrectly publishes state for replica when it puts replica into LIR.
> -
>
> Key: SOLR-9555
> URL: https://issues.apache.org/jira/browse/SOLR-9555
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
> Attachments: SOLR-9555.patch, SOLR-9555.patch, SOLR-9555-WIP-2.patch, 
> SOLR-9555-WIP-3.patch, SOLR-9555-WIP.patch
>
>
> See 
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17888/consoleFull 
> for an example



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10489) StatsReloadRaceTest.testParallelReloadAndStats failures

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985464#comment-15985464
 ] 

Erick Erickson commented on SOLR-10489:
---

I seem to have gotten 1,000 runs without a failure

> StatsReloadRaceTest.testParallelReloadAndStats failures
> ---
>
> Key: SOLR-10489
> URL: https://issues.apache.org/jira/browse/SOLR-10489
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
>Reporter: Andrzej Bialecki 
>Assignee: Erick Erickson
>
> This test has been failing a lot after the changes in SOLR-9959, for unclear 
> reasons. The failure is always in the same place:
> {code}
> java.lang.AssertionError: Key SEARCHER.searcher.indexVersion not found in 
> registry solr.core.collection1
>   at 
> __randomizedtesting.SeedInfo.seed([28B54D77FD0E3DF1:E72B284E72FF55AE]:0)
>   at org.junit.Assert.fail(Assert.java:93)
>   at org.junit.Assert.assertTrue(Assert.java:43)
>   at 
> org.apache.solr.handler.admin.StatsReloadRaceTest.requestMetrics(StatsReloadRaceTest.java:132)
>   at 
> org.apache.solr.handler.admin.StatsReloadRaceTest.testParallelReloadAndStats(StatsReloadRaceTest.java:70)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10568) Automate HTML builds via Jenkins to occur with each commit

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985455#comment-15985455
 ] 

Uwe Schindler commented on SOLR-10568:
--

+1

> Automate HTML builds via Jenkins to occur with each commit
> --
>
> Key: SOLR-10568
> URL: https://issues.apache.org/jira/browse/SOLR-10568
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Spin-off from SOLR-10295.
> The idea is to use a mechanism (possibly gitpubsub and/or svnpubsub?) so 
> Jenkins builds of HTML format of the Ref Guide occur as soon as commits are 
> made to any non-released branch.
> This would allow any committer to see doc changes ASAP after a commit to 
> verify the presentation of the information is as expected.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-10295) Decide online location for Ref Guide HTML pages

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985396#comment-15985396
 ] 

Cassandra Targett edited comment on SOLR-10295 at 4/26/17 7:41 PM:
---

Based on the idea we'll publish the HTML version to the website with a similar 
process as is currently used for the Javadocs, I've written up a strawman 
process, and committed it to the meta-docs in the branch:

https://git1-us-west.apache.org/repos/asf?p=lucene-solr.git;a=blob;f=solr/solr-ref-guide/meta-docs/publish.adoc;h=f01978dba39fba074c7cb19fe1d1ca434c6b6c8e;hb=refs/heads/jira/solr-10290

_edit_: I forgot to mention that I plan to try this process out with the 
content on the jira/solr-10290 branch.


was (Author: ctargett):
Based on the idea we'll publish the HTML version to the website with a similar 
process as is currently used for the Javadocs, I've written up a strawman 
process, and committed it to the meta-docs in the branch:

https://git1-us-west.apache.org/repos/asf?p=lucene-solr.git;a=blob;f=solr/solr-ref-guide/meta-docs/publish.adoc;h=f01978dba39fba074c7cb19fe1d1ca434c6b6c8e;hb=refs/heads/jira/solr-10290


> Decide online location for Ref Guide HTML pages
> ---
>
> Key: SOLR-10295
> URL: https://issues.apache.org/jira/browse/SOLR-10295
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>
> One of the biggest decisions we need to make is where to put the new Solr Ref 
> Guide. Confluence at least had the whole web-hosting bits figured out; we 
> have to figure that out on our own.
> An obvious (maybe only to me) choice is to integrate the Ref Guide with the 
> Solr Website. However, due to the size of the Solr Ref Guide (nearly 200 
> pages), I believe trying to publish it solely with existing CMS tools will 
> create problems similar to those described in the Lucene ReleaseTodo when it 
> comes to publishing the Lucene/Solr javadocs (see 
> https://wiki.apache.org/lucene-java/ReleaseTodo#Website_.2B-.3D_javadocs).
> A solution exists already, and it's what is done for the javadocs. From the 
> above link:
> {quote}
> The solution: skip committing javadocs to the source tree, then staging, then 
> publishing, and instead commit javadocs directly to the production tree. 
> Ordinarily this would be problematic, because the CMS wants to keep the 
> production tree in sync with the staging tree, so anything it finds in the 
> production tree that's not in the staging tree gets nuked. However, the CMS 
> has a built-in mechanism to allow exceptions to the 
> keep-production-in-sync-with-staging rule: extpaths.txt.
> {quote}
> This solution (for those who don't know already) is to provide a static text 
> file (extpaths.txt) that includes the javadoc paths that should be presented 
> in production, but which won't exist in CMS staging environments. This way, 
> we can publish HTML files directly to production and they will be preserved 
> when the staging-production trees are synced.
> The rest of the process would be quite similar to what is documented in the 
> ReleaseTodo in sections following the link above - use SVN to update the CMS 
> production site and update extpaths.txt properly. We'd do this in the 
> {{solr}} section of the CMS obviously, and not the {{lucene}} section.
> A drawback to this approach is that we won't have a staging area to view the 
> Guide before publication. Files would be generated and go to production 
> directly. We may want to put a process in place to give some additional 
> confidence that things look right first (someone's people.apache.org 
> directory? a pre-pub validation script that tests...something...?), and agree 
> on what we'd be voting on when a vote to release comes up. However, the CMS 
> is pretty much the only option that I can think of...other ideas are welcome 
> if they might work.
> We also need to agree on URL paths that make sense, considering we'll have a 
> new "site" for each major release - something like 
> {{http://lucene.apache.org/solr/ref-guide/6_1}} might work? Other thoughts 
> are welcome on this point also.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_121) - Build # 19490 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/19490/
Java: 32bit/jdk1.8.0_121 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.SolrCloudExampleTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [TransactionLog, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.TransactionLog  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at org.apache.solr.update.TransactionLog.(TransactionLog.java:190)  at 
org.apache.solr.update.UpdateLog.newTransactionLog(UpdateLog.java:438)  at 
org.apache.solr.update.UpdateLog.ensureLog(UpdateLog.java:1250)  at 
org.apache.solr.update.UpdateLog.add(UpdateLog.java:524)  at 
org.apache.solr.update.UpdateLog.add(UpdateLog.java:509)  at 
org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:349)
  at 
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:268)
  at 
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:218)
  at 
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
  at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:991)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1207)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:753)
  at 
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
  at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
  at 
org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:336)
  at 
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)  
at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:187)
  at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:143)
  at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:313) 
 at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:258)  at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:129)
  at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:278) 
 at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:258)  at 
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:180)  at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:194)
  at 
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:108)
  at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)  
at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
  at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2486)  at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)  at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:355)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:306)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) 
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
at 

[jira] [Commented] (SOLR-10568) Automate HTML builds via Jenkins to occur with each commit

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985425#comment-15985425
 ] 

Steve Rowe commented on SOLR-10568:
---

bq. publish the resulting guide to the website under correct sub folder (see 
above).

I don't think we should automatically publish any form of the *released* guide, 
even if some forms aren't voted on (like HTML).

By contrast, I'm fine with automatic "publishing" of the *unreleased* guide, as 
long as the location is not advertized to end users.  As Hoss points out, 
Jenkins could easily serve as both the build site and hosting site.

> Automate HTML builds via Jenkins to occur with each commit
> --
>
> Key: SOLR-10568
> URL: https://issues.apache.org/jira/browse/SOLR-10568
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Spin-off from SOLR-10295.
> The idea is to use a mechanism (possibly gitpubsub and/or svnpubsub?) so 
> Jenkins builds of HTML format of the Ref Guide occur as soon as commits are 
> made to any non-released branch.
> This would allow any committer to see doc changes ASAP after a commit to 
> verify the presentation of the information is as expected.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-10568) Automate HTML builds via Jenkins to occur with each commit

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985420#comment-15985420
 ] 

Uwe Schindler edited comment on SOLR-10568 at 4/26/17 7:27 PM:
---

I agree with Hoss here. It is quite easy to instruct ASF Jenkins to run on 
every commit in a branch of SVN or GIT. No need for svbpubsub or anything. The 
results of a Jenkins run can be declared as "release artifacts" (a PDF file) 
and Jenkins copies them away and also archives them. So we have an archive of 
PDF files. If HTML is created it can be handled like "Javadocs". E.g. the 
Release documents of Lucene and Solr are a whole Website structure including 
custom HTML pages, but it is just marked as "Javadocs" in Jenkins builds - so 
published automatically.


was (Author: thetaphi):
I agree with Hoss here. It is quite easy to instruct ASF Jenkins to run on 
every commit in a branch of SVN or GIT. No need for svbpubsub or anything. The 
results of a Jenkins run can be declared as "release artifacts" (a PDF file) 
and Jnekins copies them away and also archives them. So we have an archive of 
PDF files. If HTML is created it can be handled like "Javadocs". E.g. the 
Release documents of Lucene and Solr are a whole Website structure including 
custom HTML pages, but it is just marked as "Javadocs" in Jenkins builds - so 
published automatically.

> Automate HTML builds via Jenkins to occur with each commit
> --
>
> Key: SOLR-10568
> URL: https://issues.apache.org/jira/browse/SOLR-10568
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Spin-off from SOLR-10295.
> The idea is to use a mechanism (possibly gitpubsub and/or svnpubsub?) so 
> Jenkins builds of HTML format of the Ref Guide occur as soon as commits are 
> made to any non-released branch.
> This would allow any committer to see doc changes ASAP after a commit to 
> verify the presentation of the information is as expected.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10568) Automate HTML builds via Jenkins to occur with each commit

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985420#comment-15985420
 ] 

Uwe Schindler commented on SOLR-10568:
--

I agree with Hoss here. It is quite easy to instruct ASF Jenkins to run on 
every commit in a branch of SVN or GIT. No need for svbpubsub or anything. The 
results of a Jenkins run can be declared as "release artifacts" (a PDF file) 
and Jnekins copies them away and also archives them. So we have an archive of 
PDF files. If HTML is created it can be handled like "Javadocs". E.g. the 
Release documents of Lucene and Solr are a whole Website structure including 
custom HTML pages, but it is just marked as "Javadocs" in Jenkins builds - so 
published automatically.

> Automate HTML builds via Jenkins to occur with each commit
> --
>
> Key: SOLR-10568
> URL: https://issues.apache.org/jira/browse/SOLR-10568
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Spin-off from SOLR-10295.
> The idea is to use a mechanism (possibly gitpubsub and/or svnpubsub?) so 
> Jenkins builds of HTML format of the Ref Guide occur as soon as commits are 
> made to any non-released branch.
> This would allow any committer to see doc changes ASAP after a commit to 
> verify the presentation of the information is as expected.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10568) Automate HTML builds via Jenkins to occur with each commit

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985417#comment-15985417
 ] 

Steve Rowe commented on SOLR-10568:
---

{quote}
Assuming we have a jenkins job(s) building the ref guide for "unreleased" 
versions/branches (based on a git change trigger), and we want developers to be 
able to review the output, can't we just use the "lastStable" artifacts URLs to 
serve them?
\[...\]
 ...we could serve the *unreleased* HTML or PDF versions of the ref guide from 
similar URLs, which owuld always update anytime there is a successful build)
{quote}

+1 

I'll explore creating a Jenkins job (hosted on the ASF Jenkins nodes dedicated 
to website building) to build all forms of the unreleased guide.  We can then 
decide about triggering separately, but initially a periodic trigger, at least 
once a day I think, should be fine.

> Automate HTML builds via Jenkins to occur with each commit
> --
>
> Key: SOLR-10568
> URL: https://issues.apache.org/jira/browse/SOLR-10568
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Spin-off from SOLR-10295.
> The idea is to use a mechanism (possibly gitpubsub and/or svnpubsub?) so 
> Jenkins builds of HTML format of the Ref Guide occur as soon as commits are 
> made to any non-released branch.
> This would allow any committer to see doc changes ASAP after a commit to 
> verify the presentation of the information is as expected.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10295) Decide online location for Ref Guide HTML pages

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985396#comment-15985396
 ] 

Cassandra Targett commented on SOLR-10295:
--

Based on the idea we'll publish the HTML version to the website with a similar 
process as is currently used for the Javadocs, I've written up a strawman 
process, and committed it to the meta-docs in the branch:

https://git1-us-west.apache.org/repos/asf?p=lucene-solr.git;a=blob;f=solr/solr-ref-guide/meta-docs/publish.adoc;h=f01978dba39fba074c7cb19fe1d1ca434c6b6c8e;hb=refs/heads/jira/solr-10290


> Decide online location for Ref Guide HTML pages
> ---
>
> Key: SOLR-10295
> URL: https://issues.apache.org/jira/browse/SOLR-10295
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>
> One of the biggest decisions we need to make is where to put the new Solr Ref 
> Guide. Confluence at least had the whole web-hosting bits figured out; we 
> have to figure that out on our own.
> An obvious (maybe only to me) choice is to integrate the Ref Guide with the 
> Solr Website. However, due to the size of the Solr Ref Guide (nearly 200 
> pages), I believe trying to publish it solely with existing CMS tools will 
> create problems similar to those described in the Lucene ReleaseTodo when it 
> comes to publishing the Lucene/Solr javadocs (see 
> https://wiki.apache.org/lucene-java/ReleaseTodo#Website_.2B-.3D_javadocs).
> A solution exists already, and it's what is done for the javadocs. From the 
> above link:
> {quote}
> The solution: skip committing javadocs to the source tree, then staging, then 
> publishing, and instead commit javadocs directly to the production tree. 
> Ordinarily this would be problematic, because the CMS wants to keep the 
> production tree in sync with the staging tree, so anything it finds in the 
> production tree that's not in the staging tree gets nuked. However, the CMS 
> has a built-in mechanism to allow exceptions to the 
> keep-production-in-sync-with-staging rule: extpaths.txt.
> {quote}
> This solution (for those who don't know already) is to provide a static text 
> file (extpaths.txt) that includes the javadoc paths that should be presented 
> in production, but which won't exist in CMS staging environments. This way, 
> we can publish HTML files directly to production and they will be preserved 
> when the staging-production trees are synced.
> The rest of the process would be quite similar to what is documented in the 
> ReleaseTodo in sections following the link above - use SVN to update the CMS 
> production site and update extpaths.txt properly. We'd do this in the 
> {{solr}} section of the CMS obviously, and not the {{lucene}} section.
> A drawback to this approach is that we won't have a staging area to view the 
> Guide before publication. Files would be generated and go to production 
> directly. We may want to put a process in place to give some additional 
> confidence that things look right first (someone's people.apache.org 
> directory? a pre-pub validation script that tests...something...?), and agree 
> on what we'd be voting on when a vote to release comes up. However, the CMS 
> is pretty much the only option that I can think of...other ideas are welcome 
> if they might work.
> We also need to agree on URL paths that make sense, considering we'll have a 
> new "site" for each major release - something like 
> {{http://lucene.apache.org/solr/ref-guide/6_1}} might work? Other thoughts 
> are welcome on this point also.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10568) Automate HTML builds via Jenkins to occur with each commit

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985388#comment-15985388
 ] 

Cassandra Targett commented on SOLR-10568:
--

I didn't explain it all very well in my initial description, maybe. I filed 
this to discuss some of what Jan proposed in his dream scenario:

https://issues.apache.org/jira/browse/SOLR-10295?focusedCommentId=15930934=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15930934

{quote}
I would fully automate the build and publishing of the various versions of the 
HTML ref-guide though a new bot. The bot would watch git commits, build the 
HTML guide and publish it.

I think {{http://lucene.apache.org/solr/guide/}} would be a great landing page 
for the guide. It would be a page highlighting the latest official released 
guide, as well as linking to historic versions and also, less prominently, 
unreleased versions, such as master, branch_6x, branch_6_4 etc. This landing 
page could even be an SPA with auto generated links based on info we already 
have in [DOAP|https://projects.apache.org/json/projects/lucene-core.json] and 
git. So we would have

{noformat}
http://lucene.apache.org/solr/guide/ (main landing page)
http://lucene.apache.org/solr/guide/6_5/ (built from branch_6_5, this way we 
can release bugfixes simply by committing to the branch)
http://lucene.apache.org/solr/guide/branch_6x/ (build from branch_6x, next 
feature version to be released)
http://lucene.apache.org/solr/guide/master/ (build from master, next major 
version to be released)
http://lucene.apache.org/solr/guide/7_0/ (the permlink for Solr7 once it is 
released)
...
{noformat}

So the bot (some script we write but Infra hosts) will 
# monitor every lucene-solr git commit
# if branch in not master, or a release branch, or if commit does not modify 
refguide, then exit
# checkout the branch
# build ref-guide using ant, jekyll etc
# publish the resulting guide to the website under correct sub folder (see 
above).
#* Alternatively, publish to some other place such as www.solr-guide.com or 
wherever we want really.
# if JIRA issue mentioned in commit message, also add a comment to the JIRA 
issue with a link to the newly built version of the guide. 
#* Could even generate a list of deep-links to the individual guide pages 
changed by the commit, for easier review.
# if the build of the refguide fails, then post to the JIRA issue a relevant 
error message
{quote}

This led to the discussion in SOLR-10295 about gitsubpub, to automate building 
on every commit and publishing.

> Automate HTML builds via Jenkins to occur with each commit
> --
>
> Key: SOLR-10568
> URL: https://issues.apache.org/jira/browse/SOLR-10568
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Spin-off from SOLR-10295.
> The idea is to use a mechanism (possibly gitpubsub and/or svnpubsub?) so 
> Jenkins builds of HTML format of the Ref Guide occur as soon as commits are 
> made to any non-released branch.
> This would allow any committer to see doc changes ASAP after a commit to 
> verify the presentation of the information is as expected.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Release Lucene/Solr 6.5.1 RC2

[Joel Bernstein]

Please do continue Jim. And thanks for stepping in.

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, Apr 26, 2017 at 2:25 PM, jim ferenczi 
wrote:

> The maven artifacts are now published. I'll continue the procedure
> tomorrow if Joel is not available.
>
> Cheers,
> Jim
>
> 2017-04-26 19:35 GMT+02:00 jim ferenczi :
>
>> This vote has passed.
>> Thanks to all who voted.
>> Joel, I'll do the publishing to the ASF Mirrors and to Maven Central part
>> since I created the RC. Can I let you do the rest of the procedure ?
>>
>> Jim
>>
>> 2017-04-24 17:09 GMT+02:00 Yonik Seeley :
>>
>>> Oops, replied to the wrong email...
>>> My +1 was for the artifacts at
>>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.
>>> 5.1-RC2-revcd1f23c63abe03ae650c75ec8ccb37762806cc75
>>>
>>> -Yonik
>>>
>>>
>>> On Mon, Apr 24, 2017 at 11:01 AM, Steve Rowe  wrote:
>>> > Andrzej and Yonik, did you vote for the RC2 at commit 7477dcd?  That
>>> was superseded by a later “RC2" at cd1f23c.
>>> >
>>> > We really should not re-use RC numbers.
>>> >
>>> > --
>>> > Steve
>>> > www.lucidworks.com
>>> >
>>> >> On Apr 24, 2017, at 10:48 AM, Yonik Seeley  wrote:
>>> >>
>>> >> +1
>>> >>
>>> >> -Yonik
>>> >>
>>> >>
>>> >> On Thu, Apr 20, 2017 at 10:31 PM, jim ferenczi <
>>> jim.feren...@gmail.com> wrote:
>>> >>> Please vote for release candidate 2 for Lucene/Solr 6.5.1
>>> >>>
>>> >>> The artifacts can be downloaded from:
>>> >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.
>>> 5.1-RC2-rev7477dcd111b6950d4105623ad2cfe60faea463da
>>> >>>
>>> >>> You can run the smoke tester directly with this command:
>>> >>>
>>> >>> python3 -u dev-tools/scripts/smokeTestRelease.py \
>>> >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.
>>> 5.1-RC2-rev7477dcd111b6950d4105623ad2cfe60faea463da
>>> >>
>>> >> -
>>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>>> >>
>>> >
>>> >
>>> > -
>>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>>> >
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>
>>
>

Re: [VOTE] Release Lucene/Solr 6.5.1 RC2

[jim ferenczi]

The maven artifacts are now published. I'll continue the procedure tomorrow
if Joel is not available.

Cheers,
Jim

2017-04-26 19:35 GMT+02:00 jim ferenczi :

> This vote has passed.
> Thanks to all who voted.
> Joel, I'll do the publishing to the ASF Mirrors and to Maven Central part
> since I created the RC. Can I let you do the rest of the procedure ?
>
> Jim
>
> 2017-04-24 17:09 GMT+02:00 Yonik Seeley :
>
>> Oops, replied to the wrong email...
>> My +1 was for the artifacts at
>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.
>> 5.1-RC2-revcd1f23c63abe03ae650c75ec8ccb37762806cc75
>>
>> -Yonik
>>
>>
>> On Mon, Apr 24, 2017 at 11:01 AM, Steve Rowe  wrote:
>> > Andrzej and Yonik, did you vote for the RC2 at commit 7477dcd?  That
>> was superseded by a later “RC2" at cd1f23c.
>> >
>> > We really should not re-use RC numbers.
>> >
>> > --
>> > Steve
>> > www.lucidworks.com
>> >
>> >> On Apr 24, 2017, at 10:48 AM, Yonik Seeley  wrote:
>> >>
>> >> +1
>> >>
>> >> -Yonik
>> >>
>> >>
>> >> On Thu, Apr 20, 2017 at 10:31 PM, jim ferenczi 
>> wrote:
>> >>> Please vote for release candidate 2 for Lucene/Solr 6.5.1
>> >>>
>> >>> The artifacts can be downloaded from:
>> >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.
>> 5.1-RC2-rev7477dcd111b6950d4105623ad2cfe60faea463da
>> >>>
>> >>> You can run the smoke tester directly with this command:
>> >>>
>> >>> python3 -u dev-tools/scripts/smokeTestRelease.py \
>> >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.
>> 5.1-RC2-rev7477dcd111b6950d4105623ad2cfe60faea463da
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>> >>
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>

[jira] [Resolved] (SOLR-10526) facet.heatmap doesn't honor fq tag exclusion in distributed search

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley resolved SOLR-10526.
-
   Resolution: Fixed
Fix Version/s: master (7.0)
   6.6

> facet.heatmap doesn't honor fq tag exclusion in distributed search
> --
>
> Key: SOLR-10526
> URL: https://issues.apache.org/jira/browse/SOLR-10526
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: faceting
>Reporter: David Smiley
>Assignee: David Smiley
> Fix For: 6.6, master (7.0)
>
> Attachments: SOLR_10526_facet_heatmap_fq_exclusion.patch
>
>
> Excluding a tagged filter query on a heatmap facet doesn't work in 
> distributed (sharded) mode -- it's effectively ignored.  Aside from being a 
> bug in semantics (more counts should be returned), it can also thwart an 
> optimization of SOLR-10499.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10526) facet.heatmap doesn't honor fq tag exclusion in distributed search

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985291#comment-15985291
 ] 

ASF subversion and git services commented on SOLR-10526:


Commit 308a2c07f6904aeb13d5bc12738dc4636685b77a in lucene-solr's branch 
refs/heads/branch_6x from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=308a2c0 ]

SOLR-10526: fix facet.heatmap facet exclusion when distributed/sharded

(cherry picked from commit 8a99937)


> facet.heatmap doesn't honor fq tag exclusion in distributed search
> --
>
> Key: SOLR-10526
> URL: https://issues.apache.org/jira/browse/SOLR-10526
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: faceting
>Reporter: David Smiley
>Assignee: David Smiley
> Attachments: SOLR_10526_facet_heatmap_fq_exclusion.patch
>
>
> Excluding a tagged filter query on a heatmap facet doesn't work in 
> distributed (sharded) mode -- it's effectively ignored.  Aside from being a 
> bug in semantics (more counts should be returned), it can also thwart an 
> optimization of SOLR-10499.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10526) facet.heatmap doesn't honor fq tag exclusion in distributed search

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985276#comment-15985276
 ] 

ASF subversion and git services commented on SOLR-10526:


Commit 8a9993798054d2e881961cfef79dd8e82d1fe17e in lucene-solr's branch 
refs/heads/master from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8a99937 ]

SOLR-10526: fix facet.heatmap facet exclusion when distributed/sharded


> facet.heatmap doesn't honor fq tag exclusion in distributed search
> --
>
> Key: SOLR-10526
> URL: https://issues.apache.org/jira/browse/SOLR-10526
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: faceting
>Reporter: David Smiley
>Assignee: David Smiley
> Attachments: SOLR_10526_facet_heatmap_fq_exclusion.patch
>
>
> Excluding a tagged filter query on a heatmap facet doesn't work in 
> distributed (sharded) mode -- it's effectively ignored.  Aside from being a 
> bug in semantics (more counts should be returned), it can also thwart an 
> optimization of SOLR-10499.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-10569) "updateHandler" stats is null when queried via MBeans handler

[Andrzej Bialecki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrzej Bialecki  resolved SOLR-10569.
--
   Resolution: Fixed
Fix Version/s: master (7.0)

> "updateHandler" stats is null when queried via MBeans handler
> -
>
> Key: SOLR-10569
> URL: https://issues.apache.org/jira/browse/SOLR-10569
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: master (7.0)
>Reporter: Tomás Fernández Löbbe
>Assignee: Andrzej Bialecki 
> Fix For: master (7.0)
>
>
> I noticed this in master only, branch_6x doesn't have this problem. I:
> 1) started Solr example techproducts
> 2) queried: 
> http://localhost:8983/solr/techproducts/admin/mbeans?stats=true=true=json
> Among the response I see:
> {noformat}
>   "updateHandler":{
> "class":"org.apache.solr.update.DirectUpdateHandler2",
> "description":"Update handler that efficiently directly updates the 
> on-disk main lucene index",
> "stats":null},
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10569) "updateHandler" stats is null when queried via MBeans handler

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985258#comment-15985258
 ] 

ASF subversion and git services commented on SOLR-10569:


Commit 2d2257926ca8df69fe0af5abf2356f6e33496189 in lucene-solr's branch 
refs/heads/master from [~ab]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2d22579 ]

SOLR-10569: "updateHandler" stats is null when queried via MBeans handler.


> "updateHandler" stats is null when queried via MBeans handler
> -
>
> Key: SOLR-10569
> URL: https://issues.apache.org/jira/browse/SOLR-10569
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: master (7.0)
>Reporter: Tomás Fernández Löbbe
>Assignee: Andrzej Bialecki 
>
> I noticed this in master only, branch_6x doesn't have this problem. I:
> 1) started Solr example techproducts
> 2) queried: 
> http://localhost:8983/solr/techproducts/admin/mbeans?stats=true=true=json
> Among the response I see:
> {noformat}
>   "updateHandler":{
> "class":"org.apache.solr.update.DirectUpdateHandler2",
> "description":"Update handler that efficiently directly updates the 
> on-disk main lucene index",
> "stats":null},
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Release Lucene/Solr 6.5.1 RC2

[jim ferenczi]

This vote has passed.
Thanks to all who voted.
Joel, I'll do the publishing to the ASF Mirrors and to Maven Central part
since I created the RC. Can I let you do the rest of the procedure ?

Jim

2017-04-24 17:09 GMT+02:00 Yonik Seeley :

> Oops, replied to the wrong email...
> My +1 was for the artifacts at
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.5.1-RC2-
> revcd1f23c63abe03ae650c75ec8ccb37762806cc75
>
> -Yonik
>
>
> On Mon, Apr 24, 2017 at 11:01 AM, Steve Rowe  wrote:
> > Andrzej and Yonik, did you vote for the RC2 at commit 7477dcd?  That was
> superseded by a later “RC2" at cd1f23c.
> >
> > We really should not re-use RC numbers.
> >
> > --
> > Steve
> > www.lucidworks.com
> >
> >> On Apr 24, 2017, at 10:48 AM, Yonik Seeley  wrote:
> >>
> >> +1
> >>
> >> -Yonik
> >>
> >>
> >> On Thu, Apr 20, 2017 at 10:31 PM, jim ferenczi 
> wrote:
> >>> Please vote for release candidate 2 for Lucene/Solr 6.5.1
> >>>
> >>> The artifacts can be downloaded from:
> >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.5.1-RC2-
> rev7477dcd111b6950d4105623ad2cfe60faea463da
> >>>
> >>> You can run the smoke tester directly with this command:
> >>>
> >>> python3 -u dev-tools/scripts/smokeTestRelease.py \
> >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.5.1-RC2-
> rev7477dcd111b6950d4105623ad2cfe60faea463da
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[jira] [Updated] (SOLR-10572) (from 7.0.0 onwards) remove three no-longer-supported warnings in SolrIndexConfig

[Christine Poerschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christine Poerschke updated SOLR-10572:
---
Attachment: SOLR-10572.patch

> (from 7.0.0 onwards) remove three no-longer-supported warnings in 
> SolrIndexConfig
> -
>
> Key: SOLR-10572
> URL: https://issues.apache.org/jira/browse/SOLR-10572
> Project: Solr
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-10572.patch
>
>
> The mergeScheduler, mergePolicy and luceneAutoCommit [no-longer-supported 
> warnings|https://github.com/apache/lucene-solr/blame/48ca9fc3f4f8d95293cee7bb59eff61247ede181/solr/core/src/java/org/apache/solr/update/SolrIndexConfig.java#L130-L140]
>  date back to 
> [2012|https://github.com/apache/lucene-solr/commit/058179d177208450850c5e3e3103710cadd3b53e].
>  Let's remove them from 7.0.0 onwards for clarity. This caught my attention 
> as part of SOLR-8668's mergePolicy support removal.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10572) (from 7.0.0 onwards) remove three no-longer-supported warnings in SolrIndexConfig

[Christine Poerschke (JIRA)]

Christine Poerschke created SOLR-10572:
--

 Summary: (from 7.0.0 onwards) remove three no-longer-supported 
warnings in SolrIndexConfig
 Key: SOLR-10572
 URL: https://issues.apache.org/jira/browse/SOLR-10572
 Project: Solr
  Issue Type: Task
Reporter: Christine Poerschke
Assignee: Christine Poerschke
Priority: Minor


The mergeScheduler, mergePolicy and luceneAutoCommit [no-longer-supported 
warnings|https://github.com/apache/lucene-solr/blame/48ca9fc3f4f8d95293cee7bb59eff61247ede181/solr/core/src/java/org/apache/solr/update/SolrIndexConfig.java#L130-L140]
 date back to 
[2012|https://github.com/apache/lucene-solr/commit/058179d177208450850c5e3e3103710cadd3b53e].
 Let's remove them from 7.0.0 onwards for clarity. This caught my attention as 
part of SOLR-8668's mergePolicy support removal.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10568) Automate HTML builds via Jenkins to occur with each commit

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985170#comment-15985170
 ] 

Hoss Man commented on SOLR-10568:
-

bq. The idea is to use a mechanism (possibly gitpubsub and/or svnpubsub?) so 
Jenkins builds of HTML format of the Ref Guide occur as soon as commits are 
made to any non-released branch.

Do we really need any special publishing "mechanism" here?

Assuming we have a jenkins job(s) building the ref guide for "unreleased" 
versions/branches (based on a git change trigger), and we want _developers_ to 
be able to review the output, can't we just use the "lastStable" artifacts URLs 
to serve them?

Example: here's the Changes.html file served from jenkins for the last stable 
build of the 6.x artifacts jenkins job...

https://builds.apache.org/job/Lucene-Artifacts-6.x/lastStableBuild/artifact/lucene/dist/changes/Changes.html

...we could serve the *unreleased* HTML or PDF versions of the ref guide from 
similar URLs, which owuld always update anytime there is a successful build)

(this isn't a general purpose solution for publishing content/code, but for the 
purpose of developers reviewing the generated output of recently committed 
changes -- that's pretty much exactly what jenkins "artifacts" hosting is for)

> Automate HTML builds via Jenkins to occur with each commit
> --
>
> Key: SOLR-10568
> URL: https://issues.apache.org/jira/browse/SOLR-10568
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>Priority: Minor
>
> Spin-off from SOLR-10295.
> The idea is to use a mechanism (possibly gitpubsub and/or svnpubsub?) so 
> Jenkins builds of HTML format of the Ref Guide occur as soon as commits are 
> made to any non-released branch.
> This would allow any committer to see doc changes ASAP after a commit to 
> verify the presentation of the information is as expected.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10570) Changing solr.cloud.wait-for-updates-with-stale-state-pause for tests

[Mano Kovacs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mano Kovacs updated SOLR-10570:
---
Description: 
{{solr.cloud.wait-for-updates-with-stale-state-pause}} is by default 2.5 
seconds and it controls how long the follower waits for updates on leader (see 
SOLR-7141).

SOLR-9849 changes the sysprop in the test but it does not always applies to the 
tests.

  was:
{{solr.cloud.wait-for-updates-with-stale-state-pause}} is by default 2.5 
seconds and it controls how long the follower waits for updates on leader (see 
SOLR-7141).

[~mihaly.toth] is proposing to change it to a lower value so
- tests getting faster (10% of improvement based on first measurements)
- less falkyness by timeouts on distributed tests.


> Changing solr.cloud.wait-for-updates-with-stale-state-pause for tests
> -
>
> Key: SOLR-10570
> URL: https://issues.apache.org/jira/browse/SOLR-10570
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Mano Kovacs
>
> {{solr.cloud.wait-for-updates-with-stale-state-pause}} is by default 2.5 
> seconds and it controls how long the follower waits for updates on leader 
> (see SOLR-7141).
> SOLR-9849 changes the sysprop in the test but it does not always applies to 
> the tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Reopened] (SOLR-10570) Changing solr.cloud.wait-for-updates-with-stale-state-pause for tests

[Mano Kovacs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mano Kovacs reopened SOLR-10570:


> Changing solr.cloud.wait-for-updates-with-stale-state-pause for tests
> -
>
> Key: SOLR-10570
> URL: https://issues.apache.org/jira/browse/SOLR-10570
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Mano Kovacs
>
> {{solr.cloud.wait-for-updates-with-stale-state-pause}} is by default 2.5 
> seconds and it controls how long the follower waits for updates on leader 
> (see SOLR-7141).
> [~mihaly.toth] is proposing to change it to a lower value so
> - tests getting faster (10% of improvement based on first measurements)
> - less falkyness by timeouts on distributed tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7796) Make reThrow idiom declare Error return type so callers may use it in a way that compiler knows subsequent code is unreachable

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985144#comment-15985144
 ] 

Hoss Man commented on LUCENE-7796:
--

bq. Let me know what you think, especially Hoss Man since you provided an 
alternative view ...

I don't have strong opinions on this at all, i was just putting out some 
strawmen to address the two diff calling models you were trying to fix 
w/minimal changes -- unifying those models to force the caller to do the null 
check is completely fine with me.

> Make reThrow idiom declare Error return type so callers may use it in a way 
> that compiler knows subsequent code is unreachable
> --
>
> Key: LUCENE-7796
> URL: https://issues.apache.org/jira/browse/LUCENE-7796
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Trivial
> Fix For: 6.x, master (7.0)
>
> Attachments: LUCENE-7796.patch
>
>
> A spinoff from LUCENE-7792: reThrow can be declared to return an unchecked 
> exception so that callers can choose to use {{throw reThrow(...)}} as an 
> idiom to let the compiler know any subsequent code will be unreachable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9530) Add an Atomic Update Processor

[Amrit Sarkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amrit Sarkar updated SOLR-9530:
---
Attachment: commit()-doesn't-work.png
assertU(...)-works.png
SOLR-9530.patch

Figured out. _commit()_ itself doesn't work but *assertU(commit())*.

I have attached screenshots for the same, looked into assertU(..) 
implementation, don't know what special is there. Thank you Erick for looking 
into it.

Updated patch, Noble I think we got everything correct this time. Kindly review 
whenever you get time.

> Add an Atomic Update Processor 
> ---
>
> Key: SOLR-9530
> URL: https://issues.apache.org/jira/browse/SOLR-9530
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
> Attachments: assertU(...)-works.png, commit()-doesn't-work.png, 
> SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, 
> SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, 
> SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch
>
>
> I'd like to explore the idea of adding a new update processor to help ingest 
> partial updates.
> Example use-case - There are two datasets with a common id field. How can I 
> merge both of them at index time?
> Proposed Solution: 
> {code}
> 
>   
> add
>   
>   
>   
> 
> {code}
> So the first JSON dump could be ingested against 
> {{http://localhost:8983/solr/gettingstarted/update/json}}
> And then the second JSON could be ingested against
> {{http://localhost:8983/solr/gettingstarted/update/json?processor=atomic}}
> The Atomic Update Processor could support all the atomic update operations 
> currently supported.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10262) Collect request latency metrics for histograms

[Andrzej Bialecki (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985072#comment-15985072
 ] 

Andrzej Bialecki  commented on SOLR-10262:
--

[~otis] - actually, I hesitated to add either HdrHistogram or the one from 
rolling-metrics. The first one doesn't implement exponential decay or at least 
some form of incremental reset (eg. time-window based), and reset-on-snapshot 
will never work correctly with multiple reporters. The rolling-metrics package 
implements exponential decay and much more, but it does it by using multiple 
partial histograms for each Histogram instance, which concerns me from the 
point of view of overall memory impact (we create hundreds of metrics per 
core), background thread mgmt, etc.

So, from my point of view the patch as it is now may be sufficient and maybe we 
should stop here - it allows you to customize what implementation and 
parameters of each metric is used, and it allows you to provide your own 
implementation and evaluate its impact. Please review the patch and let me know 
what you think.

> Collect request latency metrics for histograms
> --
>
> Key: SOLR-10262
> URL: https://issues.apache.org/jira/browse/SOLR-10262
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Otis Gospodnetic
>Assignee: Andrzej Bialecki 
> Fix For: master (7.0)
>
> Attachments: SOLR-10262.patch
>
>
> Since [~ab] is on a role with metrics...
> There is no way to accurately compute request latency percentiles from 
> metrics exposed by Solr today. We should consider making that possible. c.f. 
> https://github.com/HdrHistogram/HdrHistogram



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8762) DIH entity child=true should respond nested documents on debug

[gopikannan venugopalsamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

gopikannan venugopalsamy updated SOLR-8762:
---
Attachment: SOLR-8762.patch

[~mkhludnev],
Understood, Please check this patch.

> DIH entity child=true should respond nested documents on debug
> --
>
> Key: SOLR-8762
> URL: https://issues.apache.org/jira/browse/SOLR-8762
> Project: Solr
>  Issue Type: Improvement
>  Components: contrib - DataImportHandler
>Reporter: Mikhail Khludnev
>Priority: Minor
>  Labels: newbie, newdev
> Attachments: SOLR-8762.patch, SOLR-8762.patch
>
>
> Problem is described in 
> [comment|https://issues.apache.org/jira/browse/SOLR-5147?focusedCommentId=14744852=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14744852]
>  of SOLR-5147 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10046) Create UninvertDocValuesMergePolicy

[Christine Poerschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christine Poerschke updated SOLR-10046:
---
Fix Version/s: master (7.0)

> Create UninvertDocValuesMergePolicy
> ---
>
> Key: SOLR-10046
> URL: https://issues.apache.org/jira/browse/SOLR-10046
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Keith Laban
>Assignee: Christine Poerschke
> Fix For: master (7.0)
>
>
> Create a merge policy that can detect schema changes and use 
> UninvertingReader to uninvert fields and write docvalues into merged segments 
> when a field has docvalues enabled.
> The current behavior is to write null values in the merged segment which can 
> lead to data integrity problems when sorting or faceting pending a full 
> reindex. 
> With this patch it would still be recommended to reindex when adding 
> docvalues for performance reasons, as it not guarenteed all segments will be 
> merged with docvalues turned on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10046) Create UninvertDocValuesMergePolicy

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985051#comment-15985051
 ] 

ASF subversion and git services commented on SOLR-10046:


Commit 90b3ef18dee4d7f583d08047da3bd95d49d859cd in lucene-solr's branch 
refs/heads/master from [~cpoerschke]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=90b3ef1 ]

SOLR-10046: move from 6.6.0 to 7.0.0 CHANGES.txt (backport yet to be completed)


> Create UninvertDocValuesMergePolicy
> ---
>
> Key: SOLR-10046
> URL: https://issues.apache.org/jira/browse/SOLR-10046
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Keith Laban
>Assignee: Christine Poerschke
>
> Create a merge policy that can detect schema changes and use 
> UninvertingReader to uninvert fields and write docvalues into merged segments 
> when a field has docvalues enabled.
> The current behavior is to write null values in the merged segment which can 
> lead to data integrity problems when sorting or faceting pending a full 
> reindex. 
> With this patch it would still be recommended to reindex when adding 
> docvalues for performance reasons, as it not guarenteed all segments will be 
> merged with docvalues turned on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-10569) "updateHandler" stats is null when queried via MBeans handler

[Andrzej Bialecki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrzej Bialecki  reassigned SOLR-10569:


Assignee: Andrzej Bialecki 

> "updateHandler" stats is null when queried via MBeans handler
> -
>
> Key: SOLR-10569
> URL: https://issues.apache.org/jira/browse/SOLR-10569
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: master (7.0)
>Reporter: Tomás Fernández Löbbe
>Assignee: Andrzej Bialecki 
>
> I noticed this in master only, branch_6x doesn't have this problem. I:
> 1) started Solr example techproducts
> 2) queried: 
> http://localhost:8983/solr/techproducts/admin/mbeans?stats=true=true=json
> Among the response I see:
> {noformat}
>   "updateHandler":{
> "class":"org.apache.solr.update.DirectUpdateHandler2",
> "description":"Update handler that efficiently directly updates the 
> on-disk main lucene index",
> "stats":null},
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10569) "updateHandler" stats is null when queried via MBeans handler

[Andrzej Bialecki (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985042#comment-15985042
 ] 

Andrzej Bialecki  commented on SOLR-10569:
--

Yes, I'm fixing it.

> "updateHandler" stats is null when queried via MBeans handler
> -
>
> Key: SOLR-10569
> URL: https://issues.apache.org/jira/browse/SOLR-10569
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: master (7.0)
>Reporter: Tomás Fernández Löbbe
>Assignee: Andrzej Bialecki 
>
> I noticed this in master only, branch_6x doesn't have this problem. I:
> 1) started Solr example techproducts
> 2) queried: 
> http://localhost:8983/solr/techproducts/admin/mbeans?stats=true=true=json
> Among the response I see:
> {noformat}
>   "updateHandler":{
> "class":"org.apache.solr.update.DirectUpdateHandler2",
> "description":"Update handler that efficiently directly updates the 
> on-disk main lucene index",
> "stats":null},
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9530) Add an Atomic Update Processor

[Amrit Sarkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985035#comment-15985035
 ] 

Amrit Sarkar commented on SOLR-9530:


Yeah it worked. If I put _commitwithin_ anything > 0 say X, and put sleep for Y 
>> X, I don't need to reload or issue commit() and it is working as expected.

Well I also put ModifiableSolrParam, "commit", "true" like we do in a curl 
update. That too is not getting picked for explicit commit by the test.

{code}
 AddUpdateCommand cmd = new AddUpdateCommand(new 
LocalSolrQueryRequest(h.getCore(),
new ModifiableSolrParams()
.add("processor", "Atomic")
.add("Atomic.cat", "delete")
.add("commit","true")
));
{code}
{code}
factory.getInstance(cmd.getReq(), new SolrQueryResponse(),
new DistributedUpdateProcessor(cmd.getReq(), new 
SolrQueryResponse(),
new RunUpdateProcessor(cmd.getReq(), 
null))).processAdd(cmd);
{code}

I like the stop-gap loop, retries are better, will incorporate this. Thank you.


> Add an Atomic Update Processor 
> ---
>
> Key: SOLR-9530
> URL: https://issues.apache.org/jira/browse/SOLR-9530
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
> Attachments: SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, 
> SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, 
> SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch
>
>
> I'd like to explore the idea of adding a new update processor to help ingest 
> partial updates.
> Example use-case - There are two datasets with a common id field. How can I 
> merge both of them at index time?
> Proposed Solution: 
> {code}
> 
>   
> add
>   
>   
>   
> 
> {code}
> So the first JSON dump could be ingested against 
> {{http://localhost:8983/solr/gettingstarted/update/json}}
> And then the second JSON could be ingested against
> {{http://localhost:8983/solr/gettingstarted/update/json?processor=atomic}}
> The Atomic Update Processor could support all the atomic update operations 
> currently supported.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-10565) SolrJmxReporterTest.testEnabled() failure

[Andrzej Bialecki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrzej Bialecki  resolved SOLR-10565.
--
   Resolution: Fixed
Fix Version/s: master (7.0)

> SolrJmxReporterTest.testEnabled() failure
> -
>
> Key: SOLR-10565
> URL: https://issues.apache.org/jira/browse/SOLR-10565
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Steve Rowe
>Assignee: Andrzej Bialecki 
> Fix For: master (7.0)
>
>
> My Jenkins found a reproducing branch_6x seed:
> {noformat}
> Checking out Revision ea3f9bb87d1e6c3cf812122c3a6f9160a8b49a19 
> (refs/remotes/origin/branch_6x)
> [...]
>[junit4]   2> 86968 INFO  
> (TEST-SolrJmxReporterTest.testEnabled-seed#[79BD33EADDE24AC0]) [
> x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testEnabled
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=SolrJmxReporterTest -Dtests.method=testEnabled 
> -Dtests.seed=79BD33EADDE24AC0 -Dtests.slow=true -Dtests.locale=cs 
> -Dtests.timezone=Europe/Amsterdam -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 0.82s J6  | SolrJmxReporterTest.testEnabled <<<
>[junit4]> Throwable #1: java.lang.AssertionError: expected:<1> but 
> was:<2>
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([79BD33EADDE24AC0:4E7C92349C473AB0]:0)
>[junit4]>  at 
> org.apache.solr.metrics.reporters.SolrJmxReporterTest.testEnabled(SolrJmxReporterTest.java:197)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, 
> docValues:{}, maxPointsInLeafNode=1415, maxMBSortInHeap=6.528209197753226, 
> sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=cs, 
> timezone=Europe/Amsterdam
>[junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=154074736,total=443023360
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10565) SolrJmxReporterTest.testEnabled() failure

[Andrzej Bialecki (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985031#comment-15985031
 ] 

Andrzej Bialecki  commented on SOLR-10565:
--

This failure was caused by rootName collisions from the reporter initialized in 
{{beforeTest()}} and the one initialized in {{testEnabled()}}.

> SolrJmxReporterTest.testEnabled() failure
> -
>
> Key: SOLR-10565
> URL: https://issues.apache.org/jira/browse/SOLR-10565
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Steve Rowe
>Assignee: Andrzej Bialecki 
>
> My Jenkins found a reproducing branch_6x seed:
> {noformat}
> Checking out Revision ea3f9bb87d1e6c3cf812122c3a6f9160a8b49a19 
> (refs/remotes/origin/branch_6x)
> [...]
>[junit4]   2> 86968 INFO  
> (TEST-SolrJmxReporterTest.testEnabled-seed#[79BD33EADDE24AC0]) [
> x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testEnabled
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=SolrJmxReporterTest -Dtests.method=testEnabled 
> -Dtests.seed=79BD33EADDE24AC0 -Dtests.slow=true -Dtests.locale=cs 
> -Dtests.timezone=Europe/Amsterdam -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 0.82s J6  | SolrJmxReporterTest.testEnabled <<<
>[junit4]> Throwable #1: java.lang.AssertionError: expected:<1> but 
> was:<2>
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([79BD33EADDE24AC0:4E7C92349C473AB0]:0)
>[junit4]>  at 
> org.apache.solr.metrics.reporters.SolrJmxReporterTest.testEnabled(SolrJmxReporterTest.java:197)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, 
> docValues:{}, maxPointsInLeafNode=1415, maxMBSortInHeap=6.528209197753226, 
> sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=cs, 
> timezone=Europe/Amsterdam
>[junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=154074736,total=443023360
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10565) SolrJmxReporterTest.testEnabled() failure

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985028#comment-15985028
 ] 

ASF subversion and git services commented on SOLR-10565:


Commit 98b2cba53c322248480b682a410ddf85247b80ef in lucene-solr's branch 
refs/heads/master from [~ab]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=98b2cba ]

SOLR-10565: Make root names more unique.


> SolrJmxReporterTest.testEnabled() failure
> -
>
> Key: SOLR-10565
> URL: https://issues.apache.org/jira/browse/SOLR-10565
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Steve Rowe
>Assignee: Andrzej Bialecki 
>
> My Jenkins found a reproducing branch_6x seed:
> {noformat}
> Checking out Revision ea3f9bb87d1e6c3cf812122c3a6f9160a8b49a19 
> (refs/remotes/origin/branch_6x)
> [...]
>[junit4]   2> 86968 INFO  
> (TEST-SolrJmxReporterTest.testEnabled-seed#[79BD33EADDE24AC0]) [
> x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testEnabled
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=SolrJmxReporterTest -Dtests.method=testEnabled 
> -Dtests.seed=79BD33EADDE24AC0 -Dtests.slow=true -Dtests.locale=cs 
> -Dtests.timezone=Europe/Amsterdam -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 0.82s J6  | SolrJmxReporterTest.testEnabled <<<
>[junit4]> Throwable #1: java.lang.AssertionError: expected:<1> but 
> was:<2>
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([79BD33EADDE24AC0:4E7C92349C473AB0]:0)
>[junit4]>  at 
> org.apache.solr.metrics.reporters.SolrJmxReporterTest.testEnabled(SolrJmxReporterTest.java:197)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, 
> docValues:{}, maxPointsInLeafNode=1415, maxMBSortInHeap=6.528209197753226, 
> sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=cs, 
> timezone=Europe/Amsterdam
>[junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=154074736,total=443023360
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10209) UI: Convert all Collections api calls to async requests, add new features/buttons

[Amrit Sarkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985027#comment-15985027
 ] 

Amrit Sarkar commented on SOLR-10209:
-

Been a while posted any update on this.

With the existing UI code, it is little challenging incorporating a new message 
box, floating preferred, to show current status, as we intend to refresh the 
entire scope at the end of each request. I intend to change that specially for 
heavy APIs and post full and final patch within a week.

> UI: Convert all Collections api calls to async requests, add new 
> features/buttons
> -
>
> Key: SOLR-10209
> URL: https://issues.apache.org/jira/browse/SOLR-10209
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Admin UI
>Reporter: Amrit Sarkar
> Attachments: SOLR-10209.patch, SOLR-10209.patch, SOLR-10209.patch, 
> SOLR-10209-v1.patch
>
>
> We are having discussion on multiple jiras for requests for Collections apis 
> from UI and how to improve them:
> -SOLR-9818: Solr admin UI rapidly retries any request(s) if it loses 
> connection with the server-
> -SOLR-10146: Admin UI: Button to delete a shard-
> SOLR-10201: Add Collection "creates collection", "Connection to Solr lost", 
> when replicationFactor>1
> Proposal =>
> *Phase 1:*
> Convert all Collections api calls to async requests and utilise REQUESTSTATUS 
> to fetch the information. There will be performance hit, but the requests 
> will be safe and sound. A progress bar will be added for request status.
> {noformat}
> > submit the async request
> if (the initial call failed or there was no status to be found)
> { report an error and suggest the user look check their system before 
> resubmitting the request. Bail out in this case, no retries, no attempt to 
> drive on. }
> else
> { put up a progress indicator while periodically checking the status, 
> Continue spinning until we can report the final status. }
> {noformat}
> *Phase 2:*
> Add new buttons/features to collections.html
> a) "Split" shard
> -b) "Delete" shard-
> c) "Backup" collection
> d) "Restore" collection
> Open to suggestions and feedbacks on this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8997) useCompoundFile should default to true like Lucene

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985026#comment-15985026
 ] 

David Smiley commented on SOLR-8997:


I think changing the default could happen whenever.  This is a tuning option.

> useCompoundFile should default to true like Lucene
> --
>
> Key: SOLR-8997
> URL: https://issues.apache.org/jira/browse/SOLR-8997
> Project: Solr
>  Issue Type: Improvement
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
>
> See the discussion at the end of SOLR-4941.  Essentially, {{useCompoundFile}} 
> should default to true, like Lucene.  In this sense, we can just not set it 
> if useCompoundFile isn't explicitly set.
> Furthermore, it's possible for some combinations of this setting and merge 
> policy's setting to make no sense -- specifically configurations that go from 
> not using CFS, to using CFS, then to not using CFS again!  If we can detect 
> that this happens, we should log a warning.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8997) useCompoundFile should default to true like Lucene

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985025#comment-15985025
 ] 

Christine Poerschke commented on SOLR-8997:
---

When Solr defaults to the same default as Lucene then yes it would be good to 
remove the {{}} from the example configs since as you say it 
clutters up the configs.

Changing Solr to default to the same default as Lucene, would that be something 
that is best be done from 7.0 onwards or could it happen (in principle) in a 
6.x release also?

> useCompoundFile should default to true like Lucene
> --
>
> Key: SOLR-8997
> URL: https://issues.apache.org/jira/browse/SOLR-8997
> Project: Solr
>  Issue Type: Improvement
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
>
> See the discussion at the end of SOLR-4941.  Essentially, {{useCompoundFile}} 
> should default to true, like Lucene.  In this sense, we can just not set it 
> if useCompoundFile isn't explicitly set.
> Furthermore, it's possible for some combinations of this setting and merge 
> policy's setting to make no sense -- specifically configurations that go from 
> not using CFS, to using CFS, then to not using CFS again!  If we can detect 
> that this happens, we should log a warning.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10566) Add timeseries Streaming Expression

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985021#comment-15985021
 ] 

ASF subversion and git services commented on SOLR-10566:


Commit 0e963f7a8aeac0b8a831cd44fd48cd0c6bda11d2 in lucene-solr's branch 
refs/heads/master from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0e963f7 ]

SOLR-10566: Add timeseries Streaming Expression


> Add timeseries Streaming Expression
> ---
>
> Key: SOLR-10566
> URL: https://issues.apache.org/jira/browse/SOLR-10566
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Attachments: SOLR-10566.patch, SOLR-10566.patch
>
>
> This ticket is to add specific time series support to streaming expressions. 
> Under the covers the timeseries expression will use the json facet API. 
> sample syntax:
> {code}
> timeseries(collection, 
>q="*:*", 
>field="rectime_dt",
>start = "2011-04-01T01:00:00.000Z",
>  end = "2016-04-01T01:00:00.000Z",
>  gap = "+1MONTH",
>count(*),
>sum(price))
> {code}
> This goes nicely with SOLR-10559. The idea is to support expressions that 
> cross-correlate and regress time series data.
> {code}
> let(cell(timeseriesA, timeseries(...)), 
> cell(timeSeriesB, timeseries(...)), 
> list(cell(a, get(timeseriesA)),
>  cell(b, get(timeseriesB)),
>  cell(stats, regres(col(timeseriesA, count(*)), col(timeseriesB, 
> count(*))
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10566) Add timeseries Streaming Expression

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985022#comment-15985022
 ] 

ASF subversion and git services commented on SOLR-10566:


Commit 679eaae9a53dc0750d25af7f80c51b58deb5192a in lucene-solr's branch 
refs/heads/master from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=679eaae ]

SOLR-10566: Fix precommit


> Add timeseries Streaming Expression
> ---
>
> Key: SOLR-10566
> URL: https://issues.apache.org/jira/browse/SOLR-10566
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Attachments: SOLR-10566.patch, SOLR-10566.patch
>
>
> This ticket is to add specific time series support to streaming expressions. 
> Under the covers the timeseries expression will use the json facet API. 
> sample syntax:
> {code}
> timeseries(collection, 
>q="*:*", 
>field="rectime_dt",
>start = "2011-04-01T01:00:00.000Z",
>  end = "2016-04-01T01:00:00.000Z",
>  gap = "+1MONTH",
>count(*),
>sum(price))
> {code}
> This goes nicely with SOLR-10559. The idea is to support expressions that 
> cross-correlate and regress time series data.
> {code}
> let(cell(timeseriesA, timeseries(...)), 
> cell(timeSeriesB, timeseries(...)), 
> list(cell(a, get(timeseriesA)),
>  cell(b, get(timeseriesB)),
>  cell(stats, regres(col(timeseriesA, count(*)), col(timeseriesB, 
> count(*))
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10506) Possible memory leak upon collection reload

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985007#comment-15985007
 ] 

Christine Poerschke commented on SOLR-10506:


Hi Torsten, thanks for finding and analysing this issue, and for putting up a 
fix for it.

I've just tried to send you a pull request with two small suggestions from my 
[jira/solr-10506-branch_6_5|https://github.com/cpoerschke/lucene-solr/tree/jira/solr-10506-branch_6_5]
 branch but something in the settings didn't allow it. Anyhow, the commit is 
there at the top of the branch ...

bq. ... To eliminate the memory leak, the schema reader is held inside a 
`WeakReference` and the reference is explicitly removed on core close. ...

As I read the code and patch, this will allow the ZkIndexSchemaReader (and 
SolrCore?) to be garbage collected but the Watcher would stay around, right? 
And ideally we'd also wish for the Watcher to not stay around ...

bq. ... Initially I wanted to supply a test case but unfortunately did not find 
a good starting point ...

Hmm, yeah, tricky. Building upon the above "but the watcher stays around" 
observation, perhaps something like this could work:
{code}
# start instance without cores
# determine baseline number of (managed-schema) watchers
# create a core
# reload the core a couple of times
# delete the core
# determine final number of (managed-schema) watchers
# test that no 'extra' watchers are still around
{code}

> Possible memory leak upon collection reload
> ---
>
> Key: SOLR-10506
> URL: https://issues.apache.org/jira/browse/SOLR-10506
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Affects Versions: 6.5
>Reporter: Torsten Bøgh Köster
> Attachments: solr_collection_reload_13_cores.png, 
> solr_gc_path_via_zk_WatchManager.png
>
>
> Upon manual Solr Collection reloading, references to the closed {{SolrCore}} 
> are not fully removed by the garbage collector as a strong reference to the 
> {{ZkIndexSchemaReader}} is held in a ZooKeeper {{Watcher}} that watches for 
> schema changes.
> In our case, this leads to a massive memory leak as managed resources are 
> still referenced by the closed {{SolrCore}}. Our Solr cloud environment 
> utilizes rather large managed resources (synonyms, stopwords). To reproduce, 
> we fired out environment up and reloaded the collection 13 times. As a result 
> we fully exhausted our heap. A closer look with the Yourkit profiler revealed 
> 13 {{SolrCore}} instances, still holding strong references to the garbage 
> collection root (see screenshot 1).
> Each {{SolrCore}} instance holds a single path with strong references to the 
> gc root via a `Watcher` in `ZkIndexSchemaReader` (see screenshot 2). The 
> {{ZkIndexSchemaReader}} registers a close hook in the {{SolrCore}} but the 
> Zookeeper is not removed upon core close.
> We supplied a Github Pull Request 
> (https://github.com/apache/lucene-solr/pull/190) that extracts the zookeeper 
> `Watcher` as a static inner class. To eliminate the memory leak, the schema 
> reader is held inside a `WeakReference` and the reference is explicitly 
> removed on core close.
> Initially I wanted to supply a test case but unfortunately did not find a 
> good starting point ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7141) RecoveryStrategy: Raise time that we wait for any updates from the leader before they saw the recovery state to have finished.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985005#comment-15985005
 ] 

Mark Miller commented on SOLR-7141:
---

[~mihaly.toth], on lowering that for tests see SOLR-9849 Use a very low value 
for solr.cloud.wait-for-updates-with-stale-state-pause in tests.

> RecoveryStrategy: Raise time that we wait for any updates from the leader 
> before they saw the recovery state to have finished.
> --
>
> Key: SOLR-7141
> URL: https://issues.apache.org/jira/browse/SOLR-7141
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.1, 6.0
>
> Attachments: SOLR-7141.patch
>
>
> The current wait of 3 seconds is pushing the envelope a bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7041) Nuke defaultSearchField and solrQueryParser from schema

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984998#comment-15984998
 ] 

Erick Erickson commented on SOLR-7041:
--

+1 to put the warning in CHANGES 6.6.0 that these won't work any more, removing 
them from the tests and committing the exception when the tests pass.

> Nuke defaultSearchField and solrQueryParser from schema
> ---
>
> Key: SOLR-7041
> URL: https://issues.apache.org/jira/browse/SOLR-7041
> Project: Solr
>  Issue Type: Improvement
>  Components: Schema and Analysis
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Fix For: 6.0
>
> Attachments: SOLR-7041.patch
>
>
> The two tags {{}} and {{solrQueryParser}} were deprecated 
> in Solr3.6 (SOLR-2724). Time to nuke them from code and {{schema.xml}} in 5.0?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9530) Add an Atomic Update Processor

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984985#comment-15984985
 ] 

Erick Erickson commented on SOLR-9530:
--

1 ms is much too short. I'd just issue the commit() when done sending docs then 
the wait.

The question is did your change work? As in prevent errors? If so let's try the 
loop above as a stop-gap.

> Add an Atomic Update Processor 
> ---
>
> Key: SOLR-9530
> URL: https://issues.apache.org/jira/browse/SOLR-9530
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
> Attachments: SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, 
> SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, 
> SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch, SOLR-9530.patch
>
>
> I'd like to explore the idea of adding a new update processor to help ingest 
> partial updates.
> Example use-case - There are two datasets with a common id field. How can I 
> merge both of them at index time?
> Proposed Solution: 
> {code}
> 
>   
> add
>   
>   
>   
> 
> {code}
> So the first JSON dump could be ingested against 
> {{http://localhost:8983/solr/gettingstarted/update/json}}
> And then the second JSON could be ingested against
> {{http://localhost:8983/solr/gettingstarted/update/json?processor=atomic}}
> The Atomic Update Processor could support all the atomic update operations 
> currently supported.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10537) Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984969#comment-15984969
 ] 

David Smiley commented on SOLR-10537:
-

Simply because toQueryString() does defined right above it.  I didn't think 
about that detail honestly.

> Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal
> --
>
> Key: SOLR-10537
> URL: https://issues.apache.org/jira/browse/SOLR-10537
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Fix For: 6.6, master (7.0)
>
> Attachments: SOLR_10537_toLocalParamsString.patch
>
>
> SolrParams ought to have a {{toLocalParamsString}} method.  I needed such a 
> thing while working on SOLR-10526 but I thought it deserved it's own issue.  
> In addition, this method needs to call {{QueryParsing.encodeLocalParamVal}} 
> but that's in Solr core (not SolrJ) so I think it can be refactored/moved to 
> {{ClientUtils}}.  I've wanted to call such a method before in client code and 
> there was none.  Surprisingly it's only used by FacetComponent within Solr.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10537) Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984963#comment-15984963
 ] 

Mike Drob commented on SOLR-10537:
--

Why did you choose to do explicit Iterator rather than a for-each loop? Just 
readability?

> Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal
> --
>
> Key: SOLR-10537
> URL: https://issues.apache.org/jira/browse/SOLR-10537
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Fix For: 6.6, master (7.0)
>
> Attachments: SOLR_10537_toLocalParamsString.patch
>
>
> SolrParams ought to have a {{toLocalParamsString}} method.  I needed such a 
> thing while working on SOLR-10526 but I thought it deserved it's own issue.  
> In addition, this method needs to call {{QueryParsing.encodeLocalParamVal}} 
> but that's in Solr core (not SolrJ) so I think it can be refactored/moved to 
> {{ClientUtils}}.  I've wanted to call such a method before in client code and 
> there was none.  Surprisingly it's only used by FacetComponent within Solr.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10522) Duplicate keys in "collations" object with JSON response format

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984960#comment-15984960
 ] 

Shawn Heisey commented on SOLR-10522:
-

I haven't looked at the code, but if I had to guess, I would imagine that the 
internal representation is NamedList, which doesn't have a problem with 
multiple mappings where the key is the same.

Do we need to deal with backward compatiblity at all, or do we consider the new 
format to be completely broken?  Were XML and Javabin responses also affected 
by SOLR-9972?


> Duplicate keys in "collations" object with JSON response format
> ---
>
> Key: SOLR-10522
> URL: https://issues.apache.org/jira/browse/SOLR-10522
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: spellchecker
>Affects Versions: 6.5
>Reporter: Nikita Pchelintsev
>Priority: Minor
>
> After upgrading Solr 6.3 -> 6.5 I've noticed a change in how json response 
> writer outputs "collations" response key when spellchecking is enabled 
> (wt=json=arrarr)
> Solr 6.3:
> "collations":
> [
>   ["collation",{
>   "collationQuery":"the",
>   "hits":48,
>   "maxScore":"30.282",
>   "misspellingsAndCorrections":
>   [
> ["thea","the"]]}],
>   ["collation",{
>   "collationQuery":"tea",
>   "hits":3,
>   "maxScore":"2.936",
>   "misspellingsAndCorrections":
>   [
> ["thea","tea"]]}],
>   ...
> Solr 6.5:
> "collations":{
>   "collation":{
> "collationQuery":"the",
> "hits":43,
> "misspellingsAndCorrections":
> [
>   ["thea","the"]]},
>   "collation":{
> "collationQuery":"tea",
> "hits":3,
> "misspellingsAndCorrections":
> [
>   ["thea","tea"]]},
> ...
> Solr 6.5 outputs object instead of an array, and it has duplicate keys which 
> is not valid for JSON format.
> Any help is appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-10537) Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley resolved SOLR-10537.
-
Resolution: Fixed

> Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal
> --
>
> Key: SOLR-10537
> URL: https://issues.apache.org/jira/browse/SOLR-10537
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Fix For: 6.6, master (7.0)
>
> Attachments: SOLR_10537_toLocalParamsString.patch
>
>
> SolrParams ought to have a {{toLocalParamsString}} method.  I needed such a 
> thing while working on SOLR-10526 but I thought it deserved it's own issue.  
> In addition, this method needs to call {{QueryParsing.encodeLocalParamVal}} 
> but that's in Solr core (not SolrJ) so I think it can be refactored/moved to 
> {{ClientUtils}}.  I've wanted to call such a method before in client code and 
> there was none.  Surprisingly it's only used by FacetComponent within Solr.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SolrCloud "master mode" planned?

[Mike Drob]

Could the zk role also be guaranteed to run the Overseer (and no
collections)? If we already have that separated out, it would make sense to
put it with the embedded zk. I think you can already configure and place
things manually this way, but it would be a huge win to package it all up
nicely for users and set it to turnkey operation.

I think it was a great improvement for deployment when we dropped tomcat,
this is the next logical step.

Mike

On Wed, Apr 26, 2017, 4:22 AM Jan Høydahl  wrote:

> There have been suggestions to add a “node controller” process which again
> could start Solr and perhaps ZK on a node.
>
> But adding a new “zk” role which would let that node start (embedded) ZK I
> cannot recall. It would of course make a deploy simpler if ZK was hidden as
> a solr role/feature and perhaps assigned to N nodes, moved if needed etc.
> If I’m not mistaken ZK 3.5 would make such more dynamic setups easier but
> is currently in beta.
>
> Also, in these days of containers, I kind of like the concept of spinning
> up N ZK containers that the Solr containers connect to and let Kubernetes
> or whatever you use take care of placement, versions etc. So perhaps the
> need for a production-ready solr-managed zk is not as big as it used to be,
> or maybe even undesirable? For production Windows installs I could still
> clearly see a need though.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 25. apr. 2017 kl. 23.30 skrev Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com>:
>
> Hi Otis,
> I've been working on, and shall be working on, a few issues on the lines
> of "hide ZK".
>
> SOLR-6736: Uploading configsets can now be done through Solr nodes instead
> of uploading them to ZK.
> SOLR-10272: Use a _default configset, with the intention of not needing
> the user to bother about the concept of configsets unless he needs to
> SOLR-10446 (SOLR-9057): User can use CloudSolrClient without access to ZK
> SOLR-8440: Enabling BasicAuth security through bin/solr script
> Ability to edit security.json through the bin/solr script
>
> Having all this in place, and perhaps some more that I may be missing,
> should hopefully not need the user to know much about ZK.
>
> 1. Do you have suggestions on what more needs to be done for "hiding ZK"?
> 2. Do you have suggestions on how to track this overall theme of "hiding
> ZK"? Some of these issues I mentioned are associated with other epics, so I
> don't know if creating a "hiding ZK" epic and having these (and other
> issues) as sub-tasks is a good idea (maybe it is). Alternatively, how about
> tracking these (and other issues) using some label?
>
> Regards,
> Ishan
>
>
>
> On Wed, Apr 26, 2017 at 2:39 AM, Otis Gospodnetić <
> otis.gospodne...@gmail.com> wrote:
>
>> Hi,
>>
>> This thread about Solr master-slave vs. SolrCloud deployment poll seems
>> to point out people find SolrCloud (the ZK part of it) deployment complex:
>>
>>
>> http://search-lucene.com/m/Solr/eHNlfm4WpJPVR92?subj=Re+Poll+Master+Slave+or+SolrCloud+
>>
>> It could be just how information is presented...
>> ... or how ZK is exposed as something external, which it is...
>>
>> Are there plans to "hide ZK"?  Or maybe have the notion of master-only
>> (not as in master-slave, but as in running ZK only, not hosting data) mode
>> for SolrCloud nodes (a la ES)?
>>
>> I peeked at JIRA, but couldn't find anything about that, although I seem
>> to recall some mention of embedding ZK to make things easier for SolrCloud
>> users.  I think I saw that at some Lucene Revolution talk?
>>
>> Thanks,
>> Otis
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>
>

RE: JDBCStream and loading drivers

[Dyer, James]

Thank you for the quick replies.  I can see how it would be powerful to be able 
to execute streaming expressions outside of solr, giving yourself the option of 
moving some of the work to the client.  I wouldn't necessarily tie it into core 
because being able to join a solr stream with a rdbms result -- either within 
solr, or in your driver program -- that could be a nice set of options to have. 
 But the patch on SOLR-1015 seems to get this right in (it seems from a quick 
look) that it uses the core's classloader when it is available, and falls back 
when it is not.  It might be nice -- especially as the streaming code base 
grows -- to consider packaging it separately from the solrj client itself.

Along these lines:  I was initially confused by the examples in 
https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions in that 
the cURL example at the top is materially different from the SolrJ example 
following it.  That is, with the cURL example, all of the work occurs in Solr 
and only the final result is streamed back.  With the SolrJ example, some of 
that work is now being done in the client.  This is easy to discover if you try 
the JDBC expression:  following the cURL example, the query originates in Solr 
; on the SolrJ example, the query originates on the client -- the server has no 
involvement at all.

Is my understanding here correct?  I can see how this design has great 
advantage as it gives us the ability to write driver programs that use the solr 
cores as worker nodes.  But this wasn't immediately clear to me.  I also 
wonder:  do we have an (easy) way with SolrJ currently to simply execute a 
(chain of) streaming expression(s) and get the result back, like in the cURL 
example (besides using JDBC)?

James Dyer
Ingram Content Group

From: Joel Bernstein [mailto:joels...@gmail.com]
Sent: Tuesday, April 25, 2017 6:25 PM
To: lucene dev 
Subject: Re: JDBCStream and loading drivers

There are a few stream impl's that have access to SolrCore (ClassifyStream, 
AnalyzeEvaluator) because they use analyzers. These classes have been added to 
core. We could move the JdbcStream to core as well if it makes the user 
experience nicer.

Originally the idea was that you could run the Streaming API Java classes like 
you would other Solrj clients. I think over time this may become important 
again, as I believe there is work underway for spinning up worker nodes that 
are not attached to a SolrCore.

Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Apr 25, 2017 at 3:25 PM, Dyer, James 
> wrote:
Using JDBCStream, Solr cannot find my database driver if I put the .jar in the 
shared lib directory ($SOLR_HOME/lib).  In order for the classloader to find 
it, the driver has to be in the server's lib directory.  Looking at why, I see 
that to get the full classpath, including what is in the shared lib directory, 
we'd typically get a reference to a SolrCore, call "getResourceLoader" and then 
"findClass".  This makes use of the URLClassLoader that knows about the shared 
lib.

But fixing JDBCStream to do this might not be so easy?  Best I can tell, 
Streaming Expressions are written nearly stand-alone as client code that merely 
executes in the Solr JVM.  Is this correct?  Indeed, the code itself is 
included with the client, in the SolrJ package, despite it mostly being 
server-side code … Maybe I misunderstand?

On the one hand, it isn't a huge deal as to where you need to put your drivers 
to make this work.  But on the other hand, it isn't really the best user 
experience, in my opinion at least, to have to dig around the server 
directories to find where your driver needs to go.  And also, if this is truly 
server-side code, why do we ship it with the client jar?  Unless there is a 
desire to make a stand-alone Streaming Expression engine that interacts with 
Solr as a client, would it be acceptable to somehow expose the SolrCore to it 
for loading resources like this?

James Dyer
Ingram Content Group

[jira] [Updated] (SOLR-10566) Add timeseries Streaming Expression

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-10566:
--
Attachment: SOLR-10566.patch

Added tests

> Add timeseries Streaming Expression
> ---
>
> Key: SOLR-10566
> URL: https://issues.apache.org/jira/browse/SOLR-10566
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Attachments: SOLR-10566.patch, SOLR-10566.patch
>
>
> This ticket is to add specific time series support to streaming expressions. 
> Under the covers the timeseries expression will use the json facet API. 
> sample syntax:
> {code}
> timeseries(collection, 
>q="*:*", 
>field="rectime_dt",
>start = "2011-04-01T01:00:00.000Z",
>  end = "2016-04-01T01:00:00.000Z",
>  gap = "+1MONTH",
>count(*),
>sum(price))
> {code}
> This goes nicely with SOLR-10559. The idea is to support expressions that 
> cross-correlate and regress time series data.
> {code}
> let(cell(timeseriesA, timeseries(...)), 
> cell(timeSeriesB, timeseries(...)), 
> list(cell(a, get(timeseriesA)),
>  cell(b, get(timeseriesB)),
>  cell(stats, regres(col(timeseriesA, count(*)), col(timeseriesB, 
> count(*))
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-6.x-Linux (64bit/jdk-9-ea+164) - Build # 3373 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/3373/
Java: 64bit/jdk-9-ea+164 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.lucene.util.bkd.TestBKD

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([1ED798F378514723]:0)


FAILED:  org.apache.lucene.util.bkd.TestBKD.testWithExceptions

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([1ED798F378514723]:0)




Build Log:
[...truncated 1604 lines...]
   [junit4] JVM J0: stderr was not empty, see: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/build/core/test/temp/junit4-J0-20170426_114112_33113574255137272372381.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) 
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option 
UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in 
a future release.
   [junit4] <<< JVM J0: EOF 

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/build/core/test/temp/junit4-J1-20170426_114112_33112607605300501973138.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) 
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option 
UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in 
a future release.
   [junit4] <<< JVM J1: EOF 

[...truncated 114 lines...]
   [junit4] Suite: org.apache.lucene.util.bkd.TestBKD
   [junit4] IGNOR/A 0.00s J2 | TestBKD.testRandomBinaryBig
   [junit4]> Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4]   2> ??.?. ??,  ??:??:?? ?? 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: 
org.apache.lucene.util.bkd.TestBKD
   [junit4]   2>1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2> at java.base@9-ea/java.lang.Object.wait(Native Method)
   [junit4]   2> at 
java.base@9-ea/java.lang.Thread.join(Thread.java:1353)
   [junit4]   2> at 
java.base@9-ea/java.lang.Thread.join(Thread.java:1427)
   [junit4]   2> at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
   [junit4]   2> at 
app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
   [junit4]   2> at 
app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
   [junit4]   2> at 
app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
   [junit4]   2> at 
app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>2) Thread[id=434, 
name=TEST-TestBKD.testWithExceptions-seed#[1ED798F378514723], state=WAITING, 
group=TGRP-TestBKD]
   [junit4]   2> at java.base@9-ea/jdk.internal.misc.Unsafe.park(Native 
Method)
   [junit4]   2> at 
java.base@9-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:192)
   [junit4]   2> at 
java.base@9-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:871)
   [junit4]   2> at 
java.base@9-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1024)
   [junit4]   2> at 
java.base@9-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1331)
   [junit4]   2> at 
java.base@9-ea/java.util.concurrent.Semaphore.acquire(Semaphore.java:318)
   [junit4]   2> at 
app//org.apache.lucene.util.OfflineSorter.readPartition(OfflineSorter.java:407)
   [junit4]   2> at 
app//org.apache.lucene.util.OfflineSorter.sort(OfflineSorter.java:274)
   [junit4]   2> at 
app//org.apache.lucene.util.bkd.BKDWriter.sort(BKDWriter.java:918)
   [junit4]   2> at 
app//org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:994)
   [junit4]   2> at 
app//org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:724)
   [junit4]   2> at 
app//org.apache.lucene.util.bkd.TestBKD.testWithExceptions(TestBKD.java:403)
   [junit4]   2> at 
java.base@9-ea/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2> at 
java.base@9-ea/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at 
java.base@9-ea/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at 
java.base@9-ea/java.lang.reflect.Method.invoke(Method.java:563)
   [junit4]  

[jira] [Updated] (LUCENE-7805) TestRandomChains.testRandomChainsWithLargeStrings() failures

[Steve Rowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe updated LUCENE-7805:
---
Summary: TestRandomChains.testRandomChainsWithLargeStrings() failures  
(was: TestRandomChains.testRandomChainsWithLargeStrings() failure)

> TestRandomChains.testRandomChainsWithLargeStrings() failures
> 
>
> Key: LUCENE-7805
> URL: https://issues.apache.org/jira/browse/LUCENE-7805
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Steve Rowe
>
> My Jenkins found a reproducing master seed, looks like FlattenGraphFilter is 
> where the problem happens:
> {noformat}
> Checking out Revision 680f4d7fd378868254786107de92a894758f667c 
> (refs/remotes/origin/master)
> [...]
>[junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
>[junit4]   2> TEST FAIL: useCharFilter=false text='\u0003J\u522f  nwqbl  
> uwtps  ob zdyokom ){0'
>[junit4]   2> Exception from random analyzer: 
>[junit4]   2> charfilters=
>[junit4]   2>   
> org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(java.io.StringReader@3ab617ae,
>  [])
>[junit4]   2>   
> org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(org.apache.lucene.analysis.charfilter.HTMLStripCharFilter@23e3c717)
>[junit4]   2> tokenizer=
>[junit4]   2>   org.apache.lucene.analysis.ngram.NGramTokenizer(9, 43)
>[junit4]   2> filters=
>[junit4]   2>   
> org.apache.lucene.analysis.miscellaneous.CodepointCountFilter(ValidatingTokenFilter@6b4708ea
>  
> term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,
>  33, 44)
>[junit4]   2>   
> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@5533fb25
>  
> term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,
>  )
>[junit4]   2>   
> org.apache.lucene.analysis.core.FlattenGraphFilter(ValidatingTokenFilter@4ef4c44
>  
> term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word)
>[junit4]   2>   
> org.apache.lucene.analysis.miscellaneous.KeepWordFilter(ValidatingTokenFilter@15baa1c7
>  
> term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,
>  [akbnucwt, vrkwm, jtomhk, jxgmfalr])
>[junit4]   2> offsetsAreCorrect=true
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomChains 
> -Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=E9460213902F2F82 
> -Dtests.slow=true -Dtests.locale=fi-FI -Dtests.timezone=Europe/Malta 
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 0.19s J7 | 
> TestRandomChains.testRandomChainsWithLargeStrings <<<
>[junit4]> Throwable #1: java.lang.AssertionError: outputEndNode=3 vs 
> inputTo=2
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([E9460213902F2F82:831DBD02C9610F71]:0)
>[junit4]>  at 
> org.apache.lucene.analysis.core.FlattenGraphFilter.incrementToken(FlattenGraphFilter.java:335)
>[junit4]>  at 
> org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
>[junit4]>  at 
> org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:51)
>[junit4]>  at 
> org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
>[junit4]>  at 
> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:731)
>[junit4]>  at 
> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:642)
>[junit4]>  at 
> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:540)
>[junit4]>  at 
> org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:880)
>[junit4]>  at java.lang.Thread.run(Thread.java:745)
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
> {dummy=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{}, 
> maxPointsInLeafNode=807, maxMBSortInHeap=5.007333045299232, 
> sim=RandomSimilarity(queryNorm=true): {}, locale=fi-FI, timezone=Europe/Malta
>[junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=492062472,total=525336576
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-7805) TestRandomChains.testRandomChainsWithLargeStrings() failure

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984922#comment-15984922
 ] 

Steve Rowe edited comment on LUCENE-7805 at 4/26/17 2:37 PM:
-

Another reproducing failure from my Jenkins, this one on branch_6x, no 
FlattenGraphFilter here though:

{noformat}
Checking out Revision b90bfaba1f065598033b60f0ba5ffaa40053eb42 
(refs/remotes/origin/branch_6x)
[...]
   [junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
   [junit4]   2> TEST FAIL: useCharFilter=false 
text='\u26a7\u26e4\u26e7\u2672\u2694\u2604\u26a5\u26d1 
\u0566\u054e\u057d\u0553\u057f\u0549 os\uecda\ud92c\udcd2  fhyysvvo vzebx 
\u30b2\u30d7\u30d8\u30dd\u30e3\u30f5\u30ea\u30dd\u30f7\u30ae\u30e6\u30c9 
\uaa3e\uaa1a\uaa5b\uaa43\uaa49\uaa53\uaa19\uaa4f\uaa5c gvrfm fe '
   [junit4]   2> Exception from random analyzer: 
   [junit4]   2> charfilters=
   [junit4]   2>   
org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(java.io.StringReader@24cfdcc4)
   [junit4]   2>   
org.apache.lucene.analysis.fa.PersianCharFilter(org.apache.lucene.analysis.charfilter.HTMLStripCharFilter@12dc3e69)
   [junit4]   2> tokenizer=
   [junit4]   2>   
org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer(org.apache.lucene.util.AttributeFactory$1@cf373344)
   [junit4]   2> filters=
   [junit4]   2>   
org.apache.lucene.analysis.gl.GalicianMinimalStemFilter(ValidatingTokenFilter@aec428a
 
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
   [junit4]   2>   
org.apache.lucene.analysis.miscellaneous.KeywordRepeatFilter(ValidatingTokenFilter@79bef7cf
 
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
   [junit4]   2>   
org.apache.lucene.analysis.cjk.CJKBigramFilter(ValidatingTokenFilter@55fed8e4 
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false,
 20)
   [junit4]   2>   
org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@226ff415 
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false,
 32)
   [junit4]   2> offsetsAreCorrect=false
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomChains 
-Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=19C23372FBAA95FB 
-Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Pacific/Ponape 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.47s J7 | 
TestRandomChains.testRandomChainsWithLargeStrings <<<
   [junit4]> Throwable #1: java.lang.IllegalArgumentException: startOffset 
must be non-negative, and endOffset must be >= startOffset; got 
startOffset=41,endOffset=40
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([19C23372FBAA95FB:73998C63A2E4B508]:0)
   [junit4]>at 
org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:110)
   [junit4]>at 
org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
   [junit4]>at 
org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
   [junit4]>at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:730)
   [junit4]>at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:641)
   [junit4]>at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:539)
   [junit4]>at 
org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:880)
   [junit4]>at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{dummy=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, 
maxPointsInLeafNode=920, maxMBSortInHeap=6.66438430375091, 
sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=bg, 
timezone=Pacific/Ponape
   [junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
1.8.0_77 (64-bit)/cpus=16,threads=1,free=383185632,total=512229376
{noformat}


was (Author: steve_rowe):
Another reproducing failure, this one on branch_6x, from my Jenkins, no 
FlattenGraphFilter here though:

{noformat}
Checking out Revision b90bfaba1f065598033b60f0ba5ffaa40053eb42 
(refs/remotes/origin/branch_6x)
[...]
   [junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
   [junit4]   2> TEST FAIL: useCharFilter=false 
text='\u26a7\u26e4\u26e7\u2672\u2694\u2604\u26a5\u26d1 
\u0566\u054e\u057d\u0553\u057f\u0549 os\uecda\ud92c\udcd2  fhyysvvo vzebx 
\u30b2\u30d7\u30d8\u30dd\u30e3\u30f5\u30ea\u30dd\u30f7\u30ae\u30e6\u30c9 

[jira] [Commented] (SOLR-10489) StatsReloadRaceTest.testParallelReloadAndStats failures

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984903#comment-15984903
 ] 

Erick Erickson commented on SOLR-10489:
---

I'll beast this on a new machine we have in the house today and see what falls 
out...



> StatsReloadRaceTest.testParallelReloadAndStats failures
> ---
>
> Key: SOLR-10489
> URL: https://issues.apache.org/jira/browse/SOLR-10489
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
>Reporter: Andrzej Bialecki 
>Assignee: Erick Erickson
>
> This test has been failing a lot after the changes in SOLR-9959, for unclear 
> reasons. The failure is always in the same place:
> {code}
> java.lang.AssertionError: Key SEARCHER.searcher.indexVersion not found in 
> registry solr.core.collection1
>   at 
> __randomizedtesting.SeedInfo.seed([28B54D77FD0E3DF1:E72B284E72FF55AE]:0)
>   at org.junit.Assert.fail(Assert.java:93)
>   at org.junit.Assert.assertTrue(Assert.java:43)
>   at 
> org.apache.solr.handler.admin.StatsReloadRaceTest.requestMetrics(StatsReloadRaceTest.java:132)
>   at 
> org.apache.solr.handler.admin.StatsReloadRaceTest.testParallelReloadAndStats(StatsReloadRaceTest.java:70)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7807) Test*RangeFieldQueries.testRandomBig() failures

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984895#comment-15984895
 ] 

Steve Rowe commented on LUCENE-7807:


I used the {{TestIntRangeFieldQueries}} repro line with {{git bisect}}; the 
first failing commit is {{907c43ce7af389c42ef200e5c2ecefbc5eee8a7a}}, which was 
committed under LUCENE-7449.

> Test*RangeFieldQueries.testRandomBig() failures
> ---
>
> Key: LUCENE-7807
> URL: https://issues.apache.org/jira/browse/LUCENE-7807
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Steve Rowe
>
> My Jenkins found a failing master seed for {{testRandomBig()}} in all the 
> {{Test\*RangeFieldQueries}} suites:
> {noformat}
> Checking out Revision 2f6101b2ab9e93173a9764450b5f71935495802e 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestIntRangeFieldQueries -Dtests.method=testRandomBig 
> -Dtests.seed=90CE64A8F1BFDF78 -Dtests.multiplier=2 -Dtests.nightly=true 
> -Dtests.slow=true 
> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt 
> -Dtests.locale=es-VE -Dtests.timezone=US/Hawaii -Dtests.asserts=true 
> -Dtests.file.encoding=UTF-8
>[junit4] FAILURE  116s J2  | TestIntRangeFieldQueries.testRandomBig <<<
>[junit4]> Throwable #1: java.lang.AssertionError: wrong hit (first of 
> possibly more):
>[junit4]> FAIL (iter 28): id=973300 should not match but did
>[junit4]>  queryRange=Box(-2147483648 TO 2147483647)
>[junit4]>  box=Box(-1240758905 TO 1326270659)
>[junit4]>  queryType=CONTAINS
>[junit4]>  deleted?=false
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([90CE64A8F1BFDF78:1799192760E6A3F8]:0)
>[junit4]>  at 
> org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
>[junit4]>  at 
> org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
>[junit4]>  at 
> org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomBig(BaseRangeFieldQueryTestCase.java:73)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
> {id=Lucene50(blocksize=128)}, docValues:{id=DocValuesFormat(name=Memory), 
> intRangeField=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=805, 
> maxMBSortInHeap=6.031593684373835, sim=RandomSimilarity(queryNorm=true): {}, 
> locale=es-VE, timezone=US/Hawaii
>[junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=238346048,total=532152320
> {noformat}
> {noformat}
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestFloatRangeFieldQueries -Dtests.method=testRandomBig 
> -Dtests.seed=90CE64A8F1BFDF78 -Dtests.multiplier=2 -Dtests.nightly=true 
> -Dtests.slow=true 
> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt 
> -Dtests.locale=no-NO -Dtests.timezone=America/Lima -Dtests.asserts=true 
> -Dtests.file.encoding=UTF-8
>[junit4] FAILURE  117s J9  | TestFloatRangeFieldQueries.testRandomBig <<<
>[junit4]> Throwable #1: java.lang.AssertionError: wrong hit (first of 
> possibly more):
>[junit4]> FAIL (iter 28): id=973300 should not match but did
>[junit4]>  queryRange=Box(-Infinity TO Infinity)
>[junit4]>  box=Box(-7.845437E37 TO 1.6013746E38)
>[junit4]>  queryType=CONTAINS
>[junit4]>  deleted?=false
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([90CE64A8F1BFDF78:1799192760E6A3F8]:0)
>[junit4]>  at 
> org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
>[junit4]>  at 
> org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
>[junit4]>  at 
> org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomBig(BaseRangeFieldQueryTestCase.java:73)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
> {id=PostingsFormat(name=Asserting)}, 
> docValues:{floatRangeField=DocValuesFormat(name=Memory), 
> id=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=69, 
> maxMBSortInHeap=7.5037451812250175, sim=RandomSimilarity(queryNorm=true): {}, 
> locale=no-NO, timezone=America/Lima
> {noformat}
> {noformat}
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestDoubleRangeFieldQueries -Dtests.method=testRandomBig 
> -Dtests.seed=90CE64A8F1BFDF78 -Dtests.multiplier=2 -Dtests.nightly=true 
> -Dtests.slow=true 
> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt 
> -Dtests.locale=th-TH -Dtests.timezone=America/Argentina/Tucuman 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 99.0s J3  | 

[jira] [Created] (LUCENE-7807) Test*RangeFieldQueries.testRandomBig() failures

[Steve Rowe (JIRA)]

Steve Rowe created LUCENE-7807:
--

 Summary: Test*RangeFieldQueries.testRandomBig() failures
 Key: LUCENE-7807
 URL: https://issues.apache.org/jira/browse/LUCENE-7807
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Steve Rowe


My Jenkins found a failing master seed for {{testRandomBig()}} in all the 
{{Test\*RangeFieldQueries}} suites:

{noformat}
Checking out Revision 2f6101b2ab9e93173a9764450b5f71935495802e 
(refs/remotes/origin/master)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestIntRangeFieldQueries -Dtests.method=testRandomBig 
-Dtests.seed=90CE64A8F1BFDF78 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=es-VE -Dtests.timezone=US/Hawaii -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE  116s J2  | TestIntRangeFieldQueries.testRandomBig <<<
   [junit4]> Throwable #1: java.lang.AssertionError: wrong hit (first of 
possibly more):
   [junit4]> FAIL (iter 28): id=973300 should not match but did
   [junit4]>  queryRange=Box(-2147483648 TO 2147483647)
   [junit4]>  box=Box(-1240758905 TO 1326270659)
   [junit4]>  queryType=CONTAINS
   [junit4]>  deleted?=false
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([90CE64A8F1BFDF78:1799192760E6A3F8]:0)
   [junit4]>at 
org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]>at 
org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]>at 
org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomBig(BaseRangeFieldQueryTestCase.java:73)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{id=Lucene50(blocksize=128)}, docValues:{id=DocValuesFormat(name=Memory), 
intRangeField=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=805, 
maxMBSortInHeap=6.031593684373835, sim=RandomSimilarity(queryNorm=true): {}, 
locale=es-VE, timezone=US/Hawaii
   [junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
1.8.0_77 (64-bit)/cpus=16,threads=1,free=238346048,total=532152320
{noformat}

{noformat}
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestFloatRangeFieldQueries -Dtests.method=testRandomBig 
-Dtests.seed=90CE64A8F1BFDF78 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=no-NO -Dtests.timezone=America/Lima -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE  117s J9  | TestFloatRangeFieldQueries.testRandomBig <<<
   [junit4]> Throwable #1: java.lang.AssertionError: wrong hit (first of 
possibly more):
   [junit4]> FAIL (iter 28): id=973300 should not match but did
   [junit4]>  queryRange=Box(-Infinity TO Infinity)
   [junit4]>  box=Box(-7.845437E37 TO 1.6013746E38)
   [junit4]>  queryType=CONTAINS
   [junit4]>  deleted?=false
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([90CE64A8F1BFDF78:1799192760E6A3F8]:0)
   [junit4]>at 
org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]>at 
org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]>at 
org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomBig(BaseRangeFieldQueryTestCase.java:73)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{id=PostingsFormat(name=Asserting)}, 
docValues:{floatRangeField=DocValuesFormat(name=Memory), 
id=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=69, 
maxMBSortInHeap=7.5037451812250175, sim=RandomSimilarity(queryNorm=true): {}, 
locale=no-NO, timezone=America/Lima
{noformat}

{noformat}
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestDoubleRangeFieldQueries -Dtests.method=testRandomBig 
-Dtests.seed=90CE64A8F1BFDF78 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=th-TH -Dtests.timezone=America/Argentina/Tucuman 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 99.0s J3  | TestDoubleRangeFieldQueries.testRandomBig <<<
   [junit4]> Throwable #1: java.lang.AssertionError: wrong hit (first of 
possibly more):
   [junit4]> FAIL (iter 28): id=973300 should not match but did
   [junit4]>  queryRange=Box(-Infinity TO Infinity)
   [junit4]>  box=Box(-7.36085005913449E307 TO 8.523870504147885E307)
   [junit4]>  queryType=CONTAINS
   [junit4]>  deleted?=false
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([90CE64A8F1BFDF78:1799192760E6A3F8]:0)
   [junit4]>at 

[jira] [Commented] (LUCENE-7805) TestRandomChains.testRandomChainsWithLargeStrings() failure

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984922#comment-15984922
 ] 

Steve Rowe commented on LUCENE-7805:


Another reproducing failure, this one on branch_6x, from my Jenkins, no 
FlattenGraphFilter here though:

{noformat}
Checking out Revision b90bfaba1f065598033b60f0ba5ffaa40053eb42 
(refs/remotes/origin/branch_6x)
[...]
   [junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
   [junit4]   2> TEST FAIL: useCharFilter=false 
text='\u26a7\u26e4\u26e7\u2672\u2694\u2604\u26a5\u26d1 
\u0566\u054e\u057d\u0553\u057f\u0549 os\uecda\ud92c\udcd2  fhyysvvo vzebx 
\u30b2\u30d7\u30d8\u30dd\u30e3\u30f5\u30ea\u30dd\u30f7\u30ae\u30e6\u30c9 
\uaa3e\uaa1a\uaa5b\uaa43\uaa49\uaa53\uaa19\uaa4f\uaa5c gvrfm fe '
   [junit4]   2> Exception from random analyzer: 
   [junit4]   2> charfilters=
   [junit4]   2>   
org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(java.io.StringReader@24cfdcc4)
   [junit4]   2>   
org.apache.lucene.analysis.fa.PersianCharFilter(org.apache.lucene.analysis.charfilter.HTMLStripCharFilter@12dc3e69)
   [junit4]   2> tokenizer=
   [junit4]   2>   
org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer(org.apache.lucene.util.AttributeFactory$1@cf373344)
   [junit4]   2> filters=
   [junit4]   2>   
org.apache.lucene.analysis.gl.GalicianMinimalStemFilter(ValidatingTokenFilter@aec428a
 
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
   [junit4]   2>   
org.apache.lucene.analysis.miscellaneous.KeywordRepeatFilter(ValidatingTokenFilter@79bef7cf
 
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
   [junit4]   2>   
org.apache.lucene.analysis.cjk.CJKBigramFilter(ValidatingTokenFilter@55fed8e4 
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false,
 20)
   [junit4]   2>   
org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@226ff415 
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false,
 32)
   [junit4]   2> offsetsAreCorrect=false
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomChains 
-Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=19C23372FBAA95FB 
-Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Pacific/Ponape 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.47s J7 | 
TestRandomChains.testRandomChainsWithLargeStrings <<<
   [junit4]> Throwable #1: java.lang.IllegalArgumentException: startOffset 
must be non-negative, and endOffset must be >= startOffset; got 
startOffset=41,endOffset=40
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([19C23372FBAA95FB:73998C63A2E4B508]:0)
   [junit4]>at 
org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:110)
   [junit4]>at 
org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
   [junit4]>at 
org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
   [junit4]>at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:730)
   [junit4]>at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:641)
   [junit4]>at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:539)
   [junit4]>at 
org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:880)
   [junit4]>at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{dummy=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, 
maxPointsInLeafNode=920, maxMBSortInHeap=6.66438430375091, 
sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=bg, 
timezone=Pacific/Ponape
   [junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
1.8.0_77 (64-bit)/cpus=16,threads=1,free=383185632,total=512229376
{noformat}

> TestRandomChains.testRandomChainsWithLargeStrings() failure
> ---
>
> Key: LUCENE-7805
> URL: https://issues.apache.org/jira/browse/LUCENE-7805
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Steve Rowe
>
> My Jenkins found a reproducing master seed, looks like FlattenGraphFilter is 
> where the problem happens:
> {noformat}
> Checking out Revision 680f4d7fd378868254786107de92a894758f667c 
> (refs/remotes/origin/master)
> [...]
>[junit4] Suite: 

[jira] [Commented] (LUCENE-7792) Add optional concurrency to OfflineSorter

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984819#comment-15984819
 ] 

ASF subversion and git services commented on LUCENE-7792:
-

Commit e83829478e891eecd88a15067e29995f1b706cff in lucene-solr's branch 
refs/heads/branch_6x from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e838294 ]

LUCENE-7792: add try/finally to make sure semaphore is released on exceptions


> Add optional concurrency to OfflineSorter
> -
>
> Key: LUCENE-7792
> URL: https://issues.apache.org/jira/browse/LUCENE-7792
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: master (7.0), 6.6
>
> Attachments: LUCENE-7792.patch, LUCENE-7792.patch, LUCENE-7792.patch
>
>
> OfflineSorter is a heavy operation and is really an embarrassingly concurrent 
> problem at heart, and if you have enough hardware concurrency (e.g. fast 
> SSDs, multiple CPU cores) it can be a big speedup.
> E.g., after reading a partition from the input, one thread can sort and write 
> it, while another thread reads the next partition, etc.  Merging partitions 
> can also be done in the background.  Some things still cannot be concurrent, 
> e.g. the initial read from the input must be a single thread, as well as the 
> final merge and writing to the final output.
> I think I found a fairly non-invasive way to add optional concurrency to this 
> class, by adding an optional ExecutorService to OfflineSorter's ctor (similar 
> to IndexSearcher) and using futures to represent each partition as we sort, 
> and creating Callable classes for sorting and merging partitions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7792) Add optional concurrency to OfflineSorter

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984818#comment-15984818
 ] 

ASF subversion and git services commented on LUCENE-7792:
-

Commit 25f1dd2f9b2a984cc59202fe56f7a8860f514657 in lucene-solr's branch 
refs/heads/master from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=25f1dd2 ]

LUCENE-7792: add try/finally to make sure semaphore is released on exceptions


> Add optional concurrency to OfflineSorter
> -
>
> Key: LUCENE-7792
> URL: https://issues.apache.org/jira/browse/LUCENE-7792
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: master (7.0), 6.6
>
> Attachments: LUCENE-7792.patch, LUCENE-7792.patch, LUCENE-7792.patch
>
>
> OfflineSorter is a heavy operation and is really an embarrassingly concurrent 
> problem at heart, and if you have enough hardware concurrency (e.g. fast 
> SSDs, multiple CPU cores) it can be a big speedup.
> E.g., after reading a partition from the input, one thread can sort and write 
> it, while another thread reads the next partition, etc.  Merging partitions 
> can also be done in the background.  Some things still cannot be concurrent, 
> e.g. the initial read from the input must be a single thread, as well as the 
> final merge and writing to the final output.
> I think I found a fairly non-invasive way to add optional concurrency to this 
> class, by adding an optional ExecutorService to OfflineSorter's ctor (similar 
> to IndexSearcher) and using futures to represent each partition as we sort, 
> and creating Callable classes for sorting and merging partitions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7796) Make reThrow idiom declare Error return type so callers may use it in a way that compiler knows subsequent code is unreachable

[Dawid Weiss (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Weiss updated LUCENE-7796:

Summary: Make reThrow idiom declare Error return type so callers may use it 
in a way that compiler knows subsequent code is unreachable  (was: Make reThrow 
idiom declare RuntimeException return type so callers may use it in a way that 
compiler knows subsequent code is unreachable)

> Make reThrow idiom declare Error return type so callers may use it in a way 
> that compiler knows subsequent code is unreachable
> --
>
> Key: LUCENE-7796
> URL: https://issues.apache.org/jira/browse/LUCENE-7796
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Trivial
> Fix For: 6.x, master (7.0)
>
> Attachments: LUCENE-7796.patch
>
>
> A spinoff from LUCENE-7792: reThrow can be declared to return an unchecked 
> exception so that callers can choose to use {{throw reThrow(...)}} as an 
> idiom to let the compiler know any subsequent code will be unreachable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7796) Make reThrow idiom declare RuntimeException return type so callers may use it in a way that compiler knows subsequent code is unreachable

[Dawid Weiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984740#comment-15984740
 ] 

Dawid Weiss commented on LUCENE-7796:
-

I'll let this bake on my test machine for a bit and see I can catch anything.

> Make reThrow idiom declare RuntimeException return type so callers may use it 
> in a way that compiler knows subsequent code is unreachable
> -
>
> Key: LUCENE-7796
> URL: https://issues.apache.org/jira/browse/LUCENE-7796
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Trivial
> Fix For: 6.x, master (7.0)
>
> Attachments: LUCENE-7796.patch
>
>
> A spinoff from LUCENE-7792: reThrow can be declared to return an unchecked 
> exception so that callers can choose to use {{throw reThrow(...)}} as an 
> idiom to let the compiler know any subsequent code will be unreachable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10537) Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984733#comment-15984733
 ] 

ASF subversion and git services commented on SOLR-10537:


Commit 73d3a77ad5b9a99944c1d7c1b64192e2c528dcb3 in lucene-solr's branch 
refs/heads/branch_6x from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=73d3a77 ]

SOLR-10537: Added SolrParams.toLocalParamsString() and moved 
QP.encodeLocalParamVal to ClientUtils

(cherry picked from commit f45017b)


> Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal
> --
>
> Key: SOLR-10537
> URL: https://issues.apache.org/jira/browse/SOLR-10537
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Fix For: 6.6, master (7.0)
>
> Attachments: SOLR_10537_toLocalParamsString.patch
>
>
> SolrParams ought to have a {{toLocalParamsString}} method.  I needed such a 
> thing while working on SOLR-10526 but I thought it deserved it's own issue.  
> In addition, this method needs to call {{QueryParsing.encodeLocalParamVal}} 
> but that's in Solr core (not SolrJ) so I think it can be refactored/moved to 
> {{ClientUtils}}.  I've wanted to call such a method before in client code and 
> there was none.  Surprisingly it's only used by FacetComponent within Solr.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7796) Make reThrow idiom declare RuntimeException return type so callers may use it in a way that compiler knows subsequent code is unreachable

[Robert Muir (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984728#comment-15984728
 ] 

Robert Muir commented on LUCENE-7796:
-

Yes, the last case is my major concern, thanks for making clear how you tackled 
it. I agree with this approach. 

> Make reThrow idiom declare RuntimeException return type so callers may use it 
> in a way that compiler knows subsequent code is unreachable
> -
>
> Key: LUCENE-7796
> URL: https://issues.apache.org/jira/browse/LUCENE-7796
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Trivial
> Fix For: 6.x, master (7.0)
>
> Attachments: LUCENE-7796.patch
>
>
> A spinoff from LUCENE-7792: reThrow can be declared to return an unchecked 
> exception so that callers can choose to use {{throw reThrow(...)}} as an 
> idiom to let the compiler know any subsequent code will be unreachable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10537) Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984719#comment-15984719
 ] 

ASF subversion and git services commented on SOLR-10537:


Commit f45017b2d4597193929c587393bb4f2351d9cd06 in lucene-solr's branch 
refs/heads/master from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f45017b ]

SOLR-10537: Added SolrParams.toLocalParamsString() and moved 
QP.encodeLocalParamVal to ClientUtils


> Add SolrParams.toLocalParamsString and ClientUtils.encodeLocalParamVal
> --
>
> Key: SOLR-10537
> URL: https://issues.apache.org/jira/browse/SOLR-10537
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Fix For: 6.6, master (7.0)
>
> Attachments: SOLR_10537_toLocalParamsString.patch
>
>
> SolrParams ought to have a {{toLocalParamsString}} method.  I needed such a 
> thing while working on SOLR-10526 but I thought it deserved it's own issue.  
> In addition, this method needs to call {{QueryParsing.encodeLocalParamVal}} 
> but that's in Solr core (not SolrJ) so I think it can be refactored/moved to 
> {{ClientUtils}}.  I've wanted to call such a method before in client code and 
> there was none.  Surprisingly it's only used by FacetComponent within Solr.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Release 6.6

[David Smiley]

+1 to start complaining in logs ASAP.
(Thanks for working on this Jan).

On Wed, Apr 26, 2017 at 5:32 AM Jan Høydahl  wrote:

> I was hoping to nuke the  and solrQueryParser from
> schema https://issues.apache.org/jira/browse/SOLR-7041
> so 7.0.0 could get rid of that code/config completely. But a ton of tests
> still depend on these deprecated configs, and fixing it all in one patch
> was a bit heavy.
> Should we commit the “complaining in logs” part of the patch and then
> convert each test in a series of commits later?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 25. apr. 2017 kl. 19.46 skrev Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com>:
>
> Hi,
> We have lots of changes, optimizations, bug fixes for 6.6. I'd like to
> propose a 6.6 release (perhaps the last 6x non-bug-fix release before 7.0
> release).
>
> I can volunteer to release this, and I can cut the release branch around
> 4th May, in order to let some time for devs to finish current issues.
>
> What do you think?
>
> Regards,
> Ishan
>
>
> --
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

[jira] [Commented] (LUCENE-7796) Make reThrow idiom declare RuntimeException return type so callers may use it in a way that compiler knows subsequent code is unreachable

[Dawid Weiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984687#comment-15984687
 ] 

Dawid Weiss commented on LUCENE-7796:
-

I am very confident in my changes. :) I mostly changed places like this one:
{code}
 } catch (Throwable t) {
   if (doSave) {
-IOUtils.reThrow(t);
+throw IOUtils.rethrowAlways(t);
{code}

Which are guaranteed not to ever be null. Other places already had null checks, 
but seemed like they could fall through. Now they're clear.
{code}
   if (this.tragedy != null) {
 // Another thread is already dealing / has dealt with the tragedy:
-IOUtils.reThrow(tragedy);
+throw IOUtils.rethrowAlways(tragedy);
   }
{code}
or this gem here:
{code}
-assert th != null; // extra safety - if we get here, it means the callable 
failed
-IOUtils.reThrow(th);
-return null; // silly, if we're here, IOUtils.reThrow always throws an 
exception
+throw IOUtils.rethrowAlways(th);
{code}

Many other places did have the possibility of a null argument though (and 
legitiamately fall through); then I explicitly emulated the behavior before 
with an explicit if:
{code}
   if (containsDestructive(options)) {
 sop("newAsynchronousFileChannel" + options + ": " + path(path), 
exception);
   } else {
-IOUtils.reThrow(exception);
+if (exception != null) {
+  throw IOUtils.rethrowAlways(exception);
+}
   }
 }
 throw new AssertionError();
{code}

So if you need a stronger assertion then yes -- I'm 100% sure. Of course the 
note to this should read: "beware of bugs in the above code; I have only proved 
it correct, not tried it.".

> Make reThrow idiom declare RuntimeException return type so callers may use it 
> in a way that compiler knows subsequent code is unreachable
> -
>
> Key: LUCENE-7796
> URL: https://issues.apache.org/jira/browse/LUCENE-7796
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Trivial
> Fix For: 6.x, master (7.0)
>
> Attachments: LUCENE-7796.patch
>
>
> A spinoff from LUCENE-7792: reThrow can be declared to return an unchecked 
> exception so that callers can choose to use {{throw reThrow(...)}} as an 
> idiom to let the compiler know any subsequent code will be unreachable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7141) RecoveryStrategy: Raise time that we wait for any updates from the leader before they saw the recovery state to have finished.

[Mano Kovacs (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984679#comment-15984679
 ] 

Mano Kovacs commented on SOLR-7141:
---

Sorry, the 7 seconds was already lowered in SOLR-9848.

> RecoveryStrategy: Raise time that we wait for any updates from the leader 
> before they saw the recovery state to have finished.
> --
>
> Key: SOLR-7141
> URL: https://issues.apache.org/jira/browse/SOLR-7141
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.1, 6.0
>
> Attachments: SOLR-7141.patch
>
>
> The current wait of 3 seconds is pushing the envelope a bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org