[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4120 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4120/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
shard1 is not consistent.  Got 1365 from 
http://127.0.0.1:59271/collection1_shard1_replica_n0 (previous client) and got 
1362 from http://127.0.0.1:59276/collection1_shard1_replica_n1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 1365 from 
http://127.0.0.1:59271/collection1_shard1_replica_n0 (previous client) and got 
1362 from http://127.0.0.1:59276/collection1_shard1_replica_n1
at 
__randomizedtesting.SeedInfo.seed([A4FFC706D4A61C01:2CABF8DC7A5A71F9]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1324)
at 
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:222)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  

[JENKINS] Lucene-Solr-Tests-master - Build # 1928 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1928/

All tests passed

Build Log:
[...truncated 56504 lines...]
ERROR: No artifacts found that match the file pattern 
"**/*.events,heapdumps/**,**/hs_err_pid*,README.txt". Configuration error?
ERROR: java.lang.InterruptedException: no matches found within 1
Build step 'Archive the artifacts' changed build result to FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Windows (64bit/jdk-9-ea+173) - Build # 6694 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6694/
Java: 64bit/jdk-9-ea+173 -XX:-UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.common.util.TestNamedListCodec

Error Message:
The test or suite printed 10318 bytes to stdout and stderr, even though the 
limit was set to 8192 bytes. Increase the limit with @Limit, ignore it 
completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

Stack Trace:
java.lang.AssertionError: The test or suite printed 10318 bytes to stdout and 
stderr, even though the limit was set to 8192 bytes. Increase the limit with 
@Limit, ignore it completely with @SuppressSysoutChecks or run with 
-Dtests.verbose=true
at __randomizedtesting.SeedInfo.seed([3093DC507BC3D92F]:0)
at 
org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:211)
at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.handler.V2ApiIntegrationTest.testCollectionsApi

Error Message:
Error from server at http://127.0.0.1:50529/solr: 
java.nio.file.InvalidPathException: Illegal char <�> at index 53: 
C:UsersjenkinsworkspaceLucene-Solr-master-Windowssolr�uildsolr-core estJ0 
empsolr.handler.V2ApiIntegrationTest_5CAF6C3E145908-001 empDir-002

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: 
Error from server at http://127.0.0.1:50529/solr: 
java.nio.file.InvalidPathException: Illegal char <�> at index 53: 
C:UsersjenkinsworkspaceLucene-Solr-master-Windowssolr�uildsolr-core  estJ0  
 empsolr.handler.V2ApiIntegrationTest_5CAF6C3E145908-001 empDir-002
at 
__randomizedtesting.SeedInfo.seed([5CAF6C3E145908:DCC248ED860D3DB6]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:805)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:600)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:470)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:400)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1102)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:843)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.handler.V2ApiIntegrationTest.testCollectionsApi(V2ApiIntegrationTest.java:141)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene

[jira] [Updated] (SOLR-10993) lots of empty highlight entries

[Christoph Hack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christoph Hack updated SOLR-10993:
--
Description: 
I have indexed documents with lots of different text fields representing 
different properties in Solr (version 6.6). Those text fields are indexed with 
storeOffsetsWithPositions=true and termVectors=true to speed up highlighting 
using the UnifiedHighlighter.

During a search, i would like to highlight those properties and I have set 
hl.fl to wildcard match all properties. Everything is working fine, except that 
the responses are huge.

Every document only has a small set of properties (let's say 10 in total, with 
1-2 matching ones), but Solr returns in the highlighting section, a dictionary 
with every possible property (about 10k) for every item. Nearly all of the 
entries are empty, but decoding the keys of the map takes a considerable amount 
of time.

In fact, the time spent decoding this unnecessary entries is enormous. Solr 
takes about 174ms for the search + encoding (i expect that the timing could be 
much better) and decoding the response in Go (using the default JSON package 
from the standard library) takes 695ms.

I guess the offending line is somewhere around:
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java#L175

Why is Solr generating map entries for missing values in the first place?

The question had been posted on stackoverflow before:
https://stackoverflow.com/questions/44846220/solr-huge-and-slow-highlighting-response-with-mostly-empty-fields


  was:
I have indexed documents with lots of different text fields representing 
different properties in Solr (version 6.6). Those text fields are indexed with 
storeOffsetsWithPositions=true and termVectors=true to speed up highlighting 
using the UnifiedHighlighter.

During a search, i would like to highlight those properties and I have set 
hl.fl to wildcard match all properties. Everything is working fine, except that 
the responses are huge.

Every document only has a small set of properties (let's say 10 in total, with 
1-2 matching ones), but Solr returns in the highlighting section, a dictionary 
with every possible property (about 10k) for every item. Nearly all of the 
entries are empty, but decoding the keys of the map takes a considerable amount 
of time.

In fact, the time spent decoding this unnecessary entries is enormous. Solr 
takes about 174ms for the search + encoding (i expect that the timing could be 
much better) and decoding the response in Go (using the default JSON package 
from the standard library) takes 695ms.

I guess the offending line is:
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java#L175

Why is Solr generating map entries for missing values in the first place?

The question had been posted on stackoverflow before:
https://stackoverflow.com/questions/44846220/solr-huge-and-slow-highlighting-response-with-mostly-empty-fields



> lots of empty highlight entries
> ---
>
> Key: SOLR-10993
> URL: https://issues.apache.org/jira/browse/SOLR-10993
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Affects Versions: 6.6
>Reporter: Christoph Hack
>
> I have indexed documents with lots of different text fields representing 
> different properties in Solr (version 6.6). Those text fields are indexed 
> with storeOffsetsWithPositions=true and termVectors=true to speed up 
> highlighting using the UnifiedHighlighter.
> During a search, i would like to highlight those properties and I have set 
> hl.fl to wildcard match all properties. Everything is working fine, except 
> that the responses are huge.
> Every document only has a small set of properties (let's say 10 in total, 
> with 1-2 matching ones), but Solr returns in the highlighting section, a 
> dictionary with every possible property (about 10k) for every item. Nearly 
> all of the entries are empty, but decoding the keys of the map takes a 
> considerable amount of time.
> In fact, the time spent decoding this unnecessary entries is enormous. Solr 
> takes about 174ms for the search + encoding (i expect that the timing could 
> be much better) and decoding the response in Go (using the default JSON 
> package from the standard library) takes 695ms.
> I guess the offending line is somewhere around:
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java#L175
> Why is Solr generating map entries for missing values in the first place?
> The question had been posted on stackoverflow before:
> https://stackoverflow.com/questions/44846220/solr-huge-and-sl

[jira] [Created] (SOLR-10993) lots of empty highlight entries

[Christoph Hack (JIRA)]

Christoph Hack created SOLR-10993:
-

 Summary: lots of empty highlight entries
 Key: SOLR-10993
 URL: https://issues.apache.org/jira/browse/SOLR-10993
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: highlighter
Affects Versions: 6.6
Reporter: Christoph Hack


I have indexed documents with lots of different text fields representing 
different properties in Solr (version 6.6). Those text fields are indexed with 
storeOffsetsWithPositions=true and termVectors=true to speed up highlighting 
using the UnifiedHighlighter.

During a search, i would like to highlight those properties and I have set 
hl.fl to wildcard match all properties. Everything is working fine, except that 
the responses are huge.

Every document only has a small set of properties (let's say 10 in total, with 
1-2 matching ones), but Solr returns in the highlighting section, a dictionary 
with every possible property (about 10k) for every item. Nearly all of the 
entries are empty, but decoding the keys of the map takes a considerable amount 
of time.

In fact, the time spent decoding this unnecessary entries is enormous. Solr 
takes about 174ms for the search + encoding (i expect that the timing could be 
much better) and decoding the response in Go (using the default JSON package 
from the standard library) takes 695ms.

I guess the offending line is:
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java#L175

Why is Solr generating map entries for missing values in the first place?

The question had been posted on stackoverflow before:
https://stackoverflow.com/questions/44846220/solr-huge-and-slow-highlighting-response-with-mostly-empty-fields




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5822) Create a markdown formatted README file for Lucene/Solr

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070967#comment-16070967
 ] 

Mike Drob commented on LUCENE-5822:
---

Didn't know about the nightly smoke target. I'll get this fixed tomorrow, 
thanks for the pointer!

> Create a markdown formatted README file for Lucene/Solr
> ---
>
> Key: LUCENE-5822
> URL: https://issues.apache.org/jira/browse/LUCENE-5822
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Shalin Shekhar Mangar
>Assignee: Mike Drob
> Fix For: master (7.0)
>
> Attachments: LUCENE-5822-addemdum.patch, LUCENE-5822.patch, 
> LUCENE-5822.patch
>
>
> We have a minimal plain text readme file right now. Github is very popular 
> these days and we are even accepting pull requests from there. I think we 
> should add a proper markdown formatted README file which not only talks about 
> the features of Lucene/Solr but also include a short tutorial on both Lucene 
> and Solr.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10954) Refactor code to standardize replica assignment

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070956#comment-16070956
 ] 

ASF subversion and git services commented on SOLR-10954:


Commit 2e56e7eb28a99109aa0b7c35a9698250f531b984 in lucene-solr's branch 
refs/heads/master from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2e56e7e ]

SOLR-10954: this was supposed to be in the solrj package


> Refactor code to standardize replica assignment
> ---
>
> Key: SOLR-10954
> URL: https://issues.apache.org/jira/browse/SOLR-10954
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>
> Today, we have different mechanisms to assign replicas to nodes
> # the default
> # rules based replica placement
> # policy based replica placement
> different commands such as add-replica, add shard, create collection etc uses 
> different code paths to assign replicas to nodes. it should be refactored to 
> unify all this into a single method, if possible



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10992) the default for stopwords should be disabled

[Rick Leir (JIRA)]

Rick Leir created SOLR-10992:


 Summary: the default for stopwords should be disabled
 Key: SOLR-10992
 URL: https://issues.apache.org/jira/browse/SOLR-10992
 Project: Solr
  Issue Type: Wish
  Security Level: Public (Default Security Level. Issues are Public)
  Components: config-api
Affects Versions: 6.6
 Environment: All
Reporter: Rick Leir
Priority: Minor


Suggest removing the stopword filter from the example configs. It is not a 
“best practice” or even a recommended practice. 

This was discussed in the user mailing list (link below).

I chose the wrong component above, what should it have been?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5822) Create a markdown formatted README file for Lucene/Solr

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070945#comment-16070945
 ] 

Steve Rowe commented on LUCENE-5822:


[~mdrob], I tried the patch using {{ant nightly-smoke}}, and it failed:

{noformat}
prepare-release-no-sign:
[mkdir] Created dir: 
/home/sarowe/git/lucene-solr/lucene/build/smokeTestRelease/dist
 [copy] Copying 476 files to 
/home/sarowe/git/lucene-solr/lucene/build/smokeTestRelease/dist/lucene
 [copy] Copying 215 files to 
/home/sarowe/git/lucene-solr/lucene/build/smokeTestRelease/dist/solr
   [smoker]   File 
"/home/sarowe/git/lucene-solr/dev-tools/scripts/smokeTestRelease.py", line 653
   [smoker] else
   [smoker]^
   [smoker] SyntaxError: invalid syntax

BUILD FAILED
/home/sarowe/git/lucene-solr/build.xml:606: exec returned: 1
{noformat}

I'll take a look at fixing it this weekend.

> Create a markdown formatted README file for Lucene/Solr
> ---
>
> Key: LUCENE-5822
> URL: https://issues.apache.org/jira/browse/LUCENE-5822
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Shalin Shekhar Mangar
>Assignee: Mike Drob
> Fix For: master (7.0)
>
> Attachments: LUCENE-5822-addemdum.patch, LUCENE-5822.patch, 
> LUCENE-5822.patch
>
>
> We have a minimal plain text readme file right now. Github is very popular 
> these days and we are even accepting pull requests from there. I think we 
> should add a proper markdown formatted README file which not only talks about 
> the features of Lucene/Solr but also include a short tutorial on both Lucene 
> and Solr.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1413 - Still Failing!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1413/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.HdfsTransactionLog  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.update.HdfsTransactionLog.(HdfsTransactionLog.java:132)  
at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)  at 
org.apache.solr.update.UpdateHandler.(UpdateHandler.java:161)  at 
org.apache.solr.update.UpdateHandler.(UpdateHandler.java:116)  at 
org.apache.solr.update.DirectUpdateHandler2.(DirectUpdateHandler2.java:108)
  at sun.reflect.GeneratedConstructorAccessor137.newInstance(Unknown Source)  
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)  at 
org.apache.solr.core.SolrCore.createInstance(SolrCore.java:773)  at 
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:835)  at 
org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:958)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:843)  at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)  at 
org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
 at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.HdfsTransactionLog
at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
at 
org.apache.solr.update.HdfsTransactionLog.(HdfsTransactionLog.java:132)
at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)
at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:161)
at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:116)
at 
org.apache.solr.update.DirectUpdateHandler2.(DirectUpdateHandler2.java:108)
at sun.reflect.GeneratedConstructorAccessor137.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:773)
at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:835)
at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)
at org.apache.solr.core.SolrCore.(SolrCore.java:958)
at org.apache.solr.core.SolrCore.(SolrCore.java:843)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)
at 
org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)


at __randomizedtesting.SeedInfo.seed([F63395BBD76CFEF0]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:296)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
at 
com.carrotsearch.randomizedtesting.

[jira] [Commented] (SOLR-10814) Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos authentication

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070859#comment-16070859
 ] 

Noble Paul commented on SOLR-10814:
---

This should be a configuration for the kerberos plugin . It can return 
appropriate name based on that configuration. Then, we don't need to touch the 
RuleBasedAuthorizationPlugin

> Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos 
> authentication
> ---
>
> Key: SOLR-10814
> URL: https://issues.apache.org/jira/browse/SOLR-10814
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>
> Solr allows configuring roles to control user access to the system. This is 
> accomplished through rule-based permission definitions which are assigned to 
> users.
> The authorization framework in Solr passes the information about the request 
> (to be authorized) using an instance of AuthorizationContext class. Currently 
> the only way to extract authenticated user is via getUserPrincipal() method 
> which returns an instance of java.security.Principal class. The 
> RuleBasedAuthorizationPlugin implementation invokes getName() method on the 
> Principal instance to fetch the list of associated roles.
> https://github.com/apache/lucene-solr/blob/2271e73e763b17f971731f6f69d6ffe46c40b944/solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPlugin.java#L156
> In case of basic authentication mechanism, the principal is the userName. 
> Hence it works fine. But in case of kerberos authentication, the user 
> principal also contains the RELM information e.g. instead of foo, it would 
> return f...@example.com. This means if the user changes the authentication 
> mechanism, he would also need to change the user-role mapping in 
> authorization section to use f...@example.com instead of foo. This is not 
> good from usability perspective.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10703) Add prepare() and finish() into DocTransformer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070828#comment-16070828
 ] 

ASF GitHub Bot commented on SOLR-10703:
---

Github user diegoceccarelli commented on the issue:

https://github.com/apache/lucene-solr/pull/202
  
@m-khl  I updated the patch, highlights: 
   1. I removed `prepare` and relied on the old `setContext()`;
   2. now a DocTransformer implements Closeable and provides the method 
`close()`
   3. If a transformer fails it doesn't affect the other trasformers (+ 
Test)

TODO:
I didn't find a way to check that DocTrasformer::close is always called in 
the tests, any idea? 

The `close()` is called from `DocsStreamer` so RealTimeGetRequest component 
should not be affected by this change, do you think it should? 






minor: I think we should rename the Jira item in something like `Make 
DocTransformer Closeable` or something like that


> Add prepare() and finish() into DocTransformer 
> ---
>
> Key: SOLR-10703
> URL: https://issues.apache.org/jira/browse/SOLR-10703
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Diego Ceccarelli
>Priority: Minor
> Fix For: master (7.0)
>
>
> This patch adds a {{prepare}} and a {{finish}} method to the interface of 
> {{DocTransformer}} allowing a developer to perform actions before/after a doc 
> transformer is applied to a result set. My use case was to benchmark the 
> performance of a transformer, since transformer time is not part of 
> {{QTime}}. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #202: SOLR-10703 Add prepare and finish in DocTrasformer

[diegoceccarelli]

Github user diegoceccarelli commented on the issue:

https://github.com/apache/lucene-solr/pull/202
  
@m-khl  I updated the patch, highlights: 
   1. I removed `prepare` and relied on the old `setContext()`;
   2. now a DocTransformer implements Closeable and provides the method 
`close()`
   3. If a transformer fails it doesn't affect the other trasformers (+ 
Test)

TODO:
I didn't find a way to check that DocTrasformer::close is always called in 
the tests, any idea? 

The `close()` is called from `DocsStreamer` so RealTimeGetRequest component 
should not be affected by this change, do you think it should? 






minor: I think we should rename the Jira item in something like `Make 
DocTransformer Closeable` or something like that


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10814) Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos authentication

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070826#comment-16070826
 ] 

Ishan Chattopadhyaya commented on SOLR-10814:
-

My preference is option a, with properly worded javadocs (which includes 
details on which authentication plugins return what, etc.) to help the third 
party plugin writer to take the right decision.

> Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos 
> authentication
> ---
>
> Key: SOLR-10814
> URL: https://issues.apache.org/jira/browse/SOLR-10814
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>
> Solr allows configuring roles to control user access to the system. This is 
> accomplished through rule-based permission definitions which are assigned to 
> users.
> The authorization framework in Solr passes the information about the request 
> (to be authorized) using an instance of AuthorizationContext class. Currently 
> the only way to extract authenticated user is via getUserPrincipal() method 
> which returns an instance of java.security.Principal class. The 
> RuleBasedAuthorizationPlugin implementation invokes getName() method on the 
> Principal instance to fetch the list of associated roles.
> https://github.com/apache/lucene-solr/blob/2271e73e763b17f971731f6f69d6ffe46c40b944/solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPlugin.java#L156
> In case of basic authentication mechanism, the principal is the userName. 
> Hence it works fine. But in case of kerberos authentication, the user 
> principal also contains the RELM information e.g. instead of foo, it would 
> return f...@example.com. This means if the user changes the authentication 
> mechanism, he would also need to change the user-role mapping in 
> authorization section to use f...@example.com instead of foo. This is not 
> good from usability perspective.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Reopened] (SOLR-10818) backup-collection V2 API doesn't parse params correctly

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man reopened SOLR-10818:
-

Dat: It appears that every *windows* policeman jenkins build since you 
committed this test has failed because you aren't doing anything to properly 
JSON encode the tempDir path before including it in the payload.

This is what {{cat -vet}} on the log from 
https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6693/ shows...

{noformat}
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=V2ApiIntegrationTest -Dtests.method=testCollectionsApi 
-Dtests.seed=2FCBA65E4E932D4F -Dtests.slow=true -Dtests.locale=ar-SY 
-Dtests.timezone=Asia/Karachi -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII^M$
   [junit4] ERROR   0.03s J0 | V2ApiIntegrationTest.testCollectionsApi <<<^M$
   [junit4]> Throwable #1: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: 
Error from server at http://127.0.0.1:62747/solr: 
java.nio.file.InvalidPathException: Illegal char <^H> at index 53: 
C:UsersjenkinsworkspaceLucene-Solr-master-Windowssolr^Huildsolr-core^IestJ0^Iempsolr.handler.V2ApiIntegrationTest_2FCBA65E4E932D4F-001^IempDir-002^M$
   [junit4]> ^Iat 
__randomizedtesting.SeedInfo.seed([2FCBA65E4E932D4F:F35541DFF68A49F1]:0)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:804)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:600)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:470)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:400)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1102)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:843)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)^M$
   [junit4]> ^Iat 
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)^M$
   [junit4]> ^Iat 
org.apache.solr.handler.V2ApiIntegrationTest.testCollectionsApi(V2ApiIntegrationTest.java:141)^M$
{noformat}

AFAICT: The original path, which looked something like 
{{C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\testJ0\temp\solr.handler.V2ApiIntegrationTest_2FCBA65E4E932D4F-001\tIempDir-002}}
 is winding up in the JSON payload verbatim, and the JSON parser in solr is 
treating most of those {{\}} as redundent escape sequences ({{\U}}, {{\j}}, 
etc...) but in the case of {{\b}} and {{\t}} it's treating them as literal 
"BACKSPACE (0x48)" and "HORIZONTAL TAB (0x49)" characters -- and the BACKSPACE 
is causing a InvalidPathException.

You can see this same underlying test bug manifest itself in a slightly diff 
way on linux if you use {{mkdir path\ with\ \'quote/}} and then clone the git 
repo in that directory...

{noformat}
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=V2ApiIntegrationTest -Dtests.method=testCollectionsApi 
-Dtests.seed=D6A7E5ADED6D97B1 -Dtests.slow=true -Dtests.locale=ro 
-Dtests.timezone=Australia/Queensland -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.04s | V2ApiIntegrationTest.testCollectionsApi <<<
   [junit4]> Throwable #1: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: 
Error from server at http://127.0.0.1:46450/solr: 
org.noggit.JSONParser$ParseException: Expected key,value separator ':': 
char=',position=223 
AFTER='andler.V2ApiIntegrationTest_D6A7E5ADED6D97B1-001/tempDir-002'' BEFORE=' 
}}'
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([D6A7E5ADED6D97B1:A39022C5574F30F]:0)
   [junit4]>at 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:804)
{noformat}

> backup-collection V2 API doesn't parse params correctly
> ---
>
> Key: SOLR-10818
> URL: https://issues.apache.org/jira/browse/SOLR-10818
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: v2 API
>Reporter: Varun Thacker
>Assignee: Cao Manh Dat
>
>  tried the backup-collection command and I got an error which seems to 
> indicate that the location param is getting escaped? Also didn't see any 
> tests for backup-collection
> {code}
> ~/solr-6.5.0$ curl -X P

[jira] [Updated] (SOLR-10574) Choose a default configset for Solr 7

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-10574:

Attachment: SOLR-10574-refguide.patch

Here are the documentation changes for SOLR-10574. [~ctargett], can you please 
review?
I'm still working on the changes for SOLR-10272.

> Choose a default configset for Solr 7
> -
>
> Key: SOLR-10574
> URL: https://issues.apache.org/jira/browse/SOLR-10574
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Blocker
> Fix For: master (7.0)
>
> Attachments: SOLR-10574.patch, SOLR-10574.patch, SOLR-10574.patch, 
> SOLR-10574.patch, SOLR-10574-refguide.patch
>
>
> Currently, the data_driven_schema_configs is the default configset when 
> collections are created using the bin/solr script and no configset is 
> specified.
> However, that may not be the best choice. We need to decide which is the best 
> choice, out of the box, considering many users might create collections 
> without knowing about the concept of a configset going forward.
> (See also SOLR-10272)
> Proposed changes:
> # Remove data_driven_schema_configs and basic_configs
> # Introduce a combined configset, {{_default}} based on the above two 
> configsets.
> # Build a "toggleable" data driven functionality into {{_default}}
> Usage:
> # Create a collection (using _default configset)
> # Data driven / schemaless functionality is enabled by default; so just start 
> indexing your documents.
> # If don't want data driven / schemaless, disable this behaviour: {code}
> curl http://host:8983/solr/coll1/config -d '{"set-user-property": 
> {"update.autoCreateFields":"false"}}'
> {code}
> # Create schema fields using schema API, and index documents



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10984) Remove old redirects from web.xml

[Varun Thacker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Thacker updated SOLR-10984:
-
Attachment: SOLR-10984.patch

Tests pass . precommit fails but that's already failing after 196d84b9 and 
Christine pointed it out on the dev list.


I will check the admin ui again. Anything else one should be looking at to 
remove a servlet and redirect rules like this?

> Remove old redirects from web.xml
> -
>
> Key: SOLR-10984
> URL: https://issues.apache.org/jira/browse/SOLR-10984
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
> Fix For: master (7.0)
>
> Attachments: SOLR-10984.patch
>
>
> As found by Varun in 
> https://lists.apache.org/thread.html/c31fa4809c9f5cc5b82eb86601b6be21592f04f623091c6506332015@%3Cdev.lucene.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4119 - Still Failing!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4119/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

6 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Could not find collection:collection2

Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
at 
__randomizedtesting.SeedInfo.seed([10AD9EA1F0837DA6:98F9A17B5E7F105E]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:909)
at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:612)
at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Statem

[jira] [Commented] (SOLR-10814) Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos authentication

[Hrishikesh Gadre (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070799#comment-16070799
 ] 

Hrishikesh Gadre commented on SOLR-10814:
-

[~ichattopadhyaya] [~noble.paul] Thanks for your feedback. 

Note - I have not yet updated the patch as I was wanted to ensure that we all 
agree on the approach. Here is the summary of my thinking so far,
* we need to be backwards compatible with respect to 
RuleBasedAuthorizationPlugin. Hence we should make the switch from principal 
name to user_name configurable.
*  We should use short user name instead of principal as most of the role based 
authorization solutions are using short user names. Here we have two options,

*option (a)* Expose both short user-name and principal

Advantages
(a) No need to worry about backwards incompatibility

Disadvantages
(a) Confusing for the third party integrators (Since similar information is 
available from both these APIs, which one should be used?) Clearly using 
getPrincipal() will require updating authorization metadata every time 
authentication mechanism is changed from/to kerberos. Hence most likely 
everyone will end up using getUserName() and will leave getPrincipal() unused 
for most part.

*option (b)* Expose short user-name and deprecate getPrincipal() method

Advantages
(a) No confusion with respect to which APIs to use.
(b) The API result will be consistent across authentication mechanisms.

Disadvantages
(a) May have backwards incompatibility concerns further down the line (e.g. 
during Solr 8.0 release).

Let me know your thoughts and I will update the patch accordingly.

> Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos 
> authentication
> ---
>
> Key: SOLR-10814
> URL: https://issues.apache.org/jira/browse/SOLR-10814
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>
> Solr allows configuring roles to control user access to the system. This is 
> accomplished through rule-based permission definitions which are assigned to 
> users.
> The authorization framework in Solr passes the information about the request 
> (to be authorized) using an instance of AuthorizationContext class. Currently 
> the only way to extract authenticated user is via getUserPrincipal() method 
> which returns an instance of java.security.Principal class. The 
> RuleBasedAuthorizationPlugin implementation invokes getName() method on the 
> Principal instance to fetch the list of associated roles.
> https://github.com/apache/lucene-solr/blob/2271e73e763b17f971731f6f69d6ffe46c40b944/solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPlugin.java#L156
> In case of basic authentication mechanism, the principal is the userName. 
> Hence it works fine. But in case of kerberos authentication, the user 
> principal also contains the RELM information e.g. instead of foo, it would 
> return f...@example.com. This means if the user changes the authentication 
> mechanism, he would also need to change the user-role mapping in 
> authorization section to use f...@example.com instead of foo. This is not 
> good from usability perspective.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6671) Introduce a solr.data.home as root dir for all data

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070782#comment-16070782
 ] 

Hoss Man edited comment on SOLR-6671 at 6/30/17 10:10 PM:
--

Jan: AFAICT DirectoryFactoryTest.testGetDataHome has failed on every *windows* 
policeman jenkins build since you added this test?

{noformat}
FAILED:  org.apache.solr.core.DirectoryFactoryTest.testGetDataHome

Error Message:
expected:<[/tmp/inst1/]data> but was:<[C:\tmp\inst1\]data>

Stack Trace:
org.junit.ComparisonFailure: expected:<[/tmp/inst1/]data> but 
was:<[C:\tmp\inst1\]data>
at 
__randomizedtesting.SeedInfo.seed([724D123FB68E10C5:EE2BCEE4EE061786]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.junit.Assert.assertEquals(Assert.java:147)
at 
org.apache.solr.core.DirectoryFactoryTest.testGetDataHome(DirectoryFactoryTest.java:58)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

{noformat}


was (Author: hossman):
Jan: AFAICT DirectoryFactoryTest.testGetDataHome has failed on every policeman 
jenkins build since you added this test?

{noformat}
FAILED:  org.apache.solr.core.DirectoryFactoryTest.testGetDataHome

Error Message:
expected:<[/tmp/inst1/]data> but was:<[C:\tmp\inst1\]data>

Stack Trace:
org.junit.ComparisonFailure: expected:<[/tmp/inst1/]data> but 
was:<[C:\tmp\inst1\]data>
at 
__randomizedtesting.SeedInfo.seed([724D123FB68E10C5:EE2BCEE4EE061786]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.junit.Assert.assertEquals(Assert.java:147)
at 
org.apache.solr.core.DirectoryFactoryTest.testGetDataHome(DirectoryFactoryTest.java:58)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

{noformat}

> Introduce a solr.data.home as root dir for all data
> ---
>
> Key: SOLR-6671
> URL: https://issues.apache.org/jira/browse/SOLR-6671
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Affects Versions: 4.10.1
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Fix For: master (7.0)
>
> Attachments: SOLR-6671.patch, SOLR-6671.patch, SOLR-6671.patch, 
> SOLR-6671.patch, SOLR-6671.patch, SOLR-6671.patch, SOLR-6671.patch, 
> SOLR-6671.patch
>
>
> Many users prefer to deploy code, config and data on separate disk locations, 
> so the default of placing the indexes under 
> {{$\{solr.solr.home\}/$\{solr.core.name\}/data}} is not always wanted.
> In a multi-core/collection system, there is not much help in the 
> {{solr.data.dir}} option, as it would set the {{dataDir}} to the same folder 
> for all collections. One workaround, if you don't want to hardcode paths in 
> your {{solrconfig.xml}}, is to specify the {{dataDir}} property in each 
> {{solr.properties}} file.
> A more elegant solution would be to introduce a new Java-option 
> {{solr.data.home}} which would be to data the same as {{solr.solr.home}} is 
> for config. If set, all collections would default their {{dataDir}} as 
> {{$\{solr.data.home\)/$\{solr.core.name\}/data}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Reopened] (SOLR-6671) Introduce a solr.data.home as root dir for all data

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man reopened SOLR-6671:


Jan: AFAICT DirectoryFactoryTest.testGetDataHome has failed on every policeman 
jenkins build since you added this test?

{noformat}
FAILED:  org.apache.solr.core.DirectoryFactoryTest.testGetDataHome

Error Message:
expected:<[/tmp/inst1/]data> but was:<[C:\tmp\inst1\]data>

Stack Trace:
org.junit.ComparisonFailure: expected:<[/tmp/inst1/]data> but 
was:<[C:\tmp\inst1\]data>
at 
__randomizedtesting.SeedInfo.seed([724D123FB68E10C5:EE2BCEE4EE061786]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.junit.Assert.assertEquals(Assert.java:147)
at 
org.apache.solr.core.DirectoryFactoryTest.testGetDataHome(DirectoryFactoryTest.java:58)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

{noformat}

> Introduce a solr.data.home as root dir for all data
> ---
>
> Key: SOLR-6671
> URL: https://issues.apache.org/jira/browse/SOLR-6671
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Affects Versions: 4.10.1
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Fix For: master (7.0)
>
> Attachments: SOLR-6671.patch, SOLR-6671.patch, SOLR-6671.patch, 
> SOLR-6671.patch, SOLR-6671.patch, SOLR-6671.patch, SOLR-6671.patch, 
> SOLR-6671.patch
>
>
> Many users prefer to deploy code, config and data on separate disk locations, 
> so the default of placing the indexes under 
> {{$\{solr.solr.home\}/$\{solr.core.name\}/data}} is not always wanted.
> In a multi-core/collection system, there is not much help in the 
> {{solr.data.dir}} option, as it would set the {{dataDir}} to the same folder 
> for all collections. One workaround, if you don't want to hardcode paths in 
> your {{solrconfig.xml}}, is to specify the {{dataDir}} property in each 
> {{solr.properties}} file.
> A more elegant solution would be to introduce a new Java-option 
> {{solr.data.home}} which would be to data the same as {{solr.solr.home}} is 
> for config. If set, all collections would default their {{dataDir}} as 
> {{$\{solr.data.home\)/$\{solr.core.name\}/data}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10814) Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos authentication

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070780#comment-16070780
 ] 

Ishan Chattopadhyaya commented on SOLR-10814:
-

[~hgadre], actually, looking at your above comment on Ranger, I am less sure 
about not wanting to deprecate.
I'll be able to take a fresh look at it on Tuesday. Thanks for your patience 
and apologies for the delay.

> Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos 
> authentication
> ---
>
> Key: SOLR-10814
> URL: https://issues.apache.org/jira/browse/SOLR-10814
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>
> Solr allows configuring roles to control user access to the system. This is 
> accomplished through rule-based permission definitions which are assigned to 
> users.
> The authorization framework in Solr passes the information about the request 
> (to be authorized) using an instance of AuthorizationContext class. Currently 
> the only way to extract authenticated user is via getUserPrincipal() method 
> which returns an instance of java.security.Principal class. The 
> RuleBasedAuthorizationPlugin implementation invokes getName() method on the 
> Principal instance to fetch the list of associated roles.
> https://github.com/apache/lucene-solr/blob/2271e73e763b17f971731f6f69d6ffe46c40b944/solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPlugin.java#L156
> In case of basic authentication mechanism, the principal is the userName. 
> Hence it works fine. But in case of kerberos authentication, the user 
> principal also contains the RELM information e.g. instead of foo, it would 
> return f...@example.com. This means if the user changes the authentication 
> mechanism, he would also need to change the user-role mapping in 
> authorization section to use f...@example.com instead of foo. This is not 
> good from usability perspective.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10814) Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos authentication

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070777#comment-16070777
 ] 

Ishan Chattopadhyaya commented on SOLR-10814:
-

[~noblepaul], I think the objective is to be able to re-use the authorization 
configs across BasicAuth and Kerberos. Kerberos requires that principals have 
the realm, but basicAuth usernames are simple ones.

I agree with the objectives and the patch in general. However, as I mentioned, 
my major concerns are (a) backward compatibility (maybe the latest patch takes 
care of it) and (b) I don't think we should be deprecating getUserPrincipal(), 
which the authorization plugins have had available till now -- I don't mind 
getUsername() being used in out of the box authz plugins going forward, but I 
don't want to break or forcefully upgrade any third party using plugin that 
uses the full user principal; e.g. Ranger plugin maybe?  [~bosco], any 
thoughts?.

Noble, what do you think on the latter issue?

> Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos 
> authentication
> ---
>
> Key: SOLR-10814
> URL: https://issues.apache.org/jira/browse/SOLR-10814
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>
> Solr allows configuring roles to control user access to the system. This is 
> accomplished through rule-based permission definitions which are assigned to 
> users.
> The authorization framework in Solr passes the information about the request 
> (to be authorized) using an instance of AuthorizationContext class. Currently 
> the only way to extract authenticated user is via getUserPrincipal() method 
> which returns an instance of java.security.Principal class. The 
> RuleBasedAuthorizationPlugin implementation invokes getName() method on the 
> Principal instance to fetch the list of associated roles.
> https://github.com/apache/lucene-solr/blob/2271e73e763b17f971731f6f69d6ffe46c40b944/solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPlugin.java#L156
> In case of basic authentication mechanism, the principal is the userName. 
> Hence it works fine. But in case of kerberos authentication, the user 
> principal also contains the RELM information e.g. instead of foo, it would 
> return f...@example.com. This means if the user changes the authentication 
> mechanism, he would also need to change the user-role mapping in 
> authorization section to use f...@example.com instead of foo. This is not 
> good from usability perspective.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: Solr and HTTP headers

[Uwe Schindler]

Hi Jan,
> Inspired by SOLR-10981 "Allow update to load gzip files” where the proposal
> is to obey the
> Content-Encoding HTTP request header to update a compressed stream, I
> started looking at other
> headers to do things in more industry-standard ways.
> 
> Accept:
> 
>   Advertises which content types, expressed as MIME types, the client is able
> to understand
>   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept
> 
>   Could replace or at least be an alternative to “wt”. Examples:
>   Accept: application/xml
>   Accept: text/csv
> 
>   Issue: Most browsers sends a long accept header, typically
> application/xml,text/html,*/*, and now
>   that json is default for Solr, we’d need to serve JSON if the accept header
> includes “*/*"

That's known under term "Content Negotiation". I come from the scientific / 
library publisher world... We use that every day!

So your described problem is well known and must be solved by some algorithm 
that takes care of browsers. All "Accept" headers have additional scores behind 
the media types. When parsing the Accept header, you split on commas and then 
parse each item and look for the score (in parameter "q"). In addition browser 
gernally send some special mime types that clearly identify them as browsers 😊

One example from the scientific publishing world, where access to digital 
object identifiers is standardized to use the "Content-Negotiation" mechanism 
since approx 10 years, is this one:

Accept: application/rdf+xml;charset=ISO-8859-1;q=0.5, 
application/vnd.citationstyles.csl+json;q=1.0, */*;q=0.1

This tells the webserver that you would like to get the citation of the DOI as 
citeproc-json, but alternatively take it as RDF. The */* is just there because 
you would as a last chance also accept anything else (like HTML).

So the order and scores are important. First order by "q" scores backwards and 
if you have same scores, take the order in list. First wins.

The algorithm is used in the library/scientific publishing world and is well 
understood. E.g. see this DOI (Digital Object Identifier) and their URL to the 
landing page (I work for PANGAEA, too...):

https://doi.pangaea.de/10.1594/PANGAEA.867475

By default it shows the landing page, if visited by a browser, but if you want 
to have the metadata in JSON-LD format, do:

Uwe Schindler@VEGA:~ > curl -H 'Accept: application/ld+json' 
'https://doi.pangaea.de/10.1594/PANGAEA.867475' 
{"@context":"http://schema.org/","@type":"Dataset","identifier":"doi:10.1594/PANGAEA.867475","url":"https://doi.pangaea.de/10.1594/PANGAEA.867475","name":"Response
 of Arctic benthic bacterial deep-sea communities to different detritus 
composition during an ex-situ high pressure 
experiment","creator":[{"@type":"Person","name":"Hoffmann, Katy...]}

If you want to download the data behind the URL (or if it would be a scientific 
paper - the PDF):

curl -H 'Accept: text/tab-separated-values, */*;q=.5' 
'https://doi.pangaea.de/10.1594/PANGAEA.867475'

Here I also added the */* with a lower score. As the server allows to give you 
text/tab-separated-values, it returns it by preference.

If your client accepts BIBTEX citations or Endnote (RIS) ones you can send a 
header, too. So you can fetch the citation of an item in a machine readable 
format the same way - and you can ask the server for any variant - standardized 
across all scientific publishers! Which one you got back is in the response's 
Content-Type 😊

If the server cannot satisfy any of your Accepts, it will send a HTTP error 406:

Uwe Schindler@VEGA:~ > curl -I -H 'Accept: foo/bar' 
'https://doi.pangaea.de/10.1594/PANGAEA.867475'
HTTP/1.1 406 Not Acceptable
Server: PANGAEA/1.0
Date: Fri, 30 Jun 2017 21:49:31 GMT
X-robots-tag: noindex,nofollow,noarchive
Content-length: 139
Content-type: text/html
X-ua-compatible: IE=Edge
X-content-type-options: nosniff
Strict-transport-security: max-age=31536000

The IDF / CrossRef / DataCite organizations (including PANGAEA...) have good 
code that also parses the "Accept" header so that stupid browser with many 
plugins (like Internet Explorer) kill you. So basically you look for specific 
media types and the catch all accept header and if it looks like a browser, 
kill it. E.g. Internet Explorer always sends application/xml with high score.

With this type of content negotiation, you can safely remove the wt=xxx param 
or make it optional.

For compression, you normally do the same (the gzip filter in Jetty does the 
same algorithm), although browser behave well on compressions and you can trust 
the header when the client sends it. The problem with sending data *to* Solr is 
that you don't know what the server accepts because you are sending data 
first...
 
> Accept-Encoding:
> 
>   Advertises which content encoding, usually a compression algorithm, the
> client is able to understand
>   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-
> Encoding

That's usual practise, every 

[jira] [Updated] (LUCENE-7894) IndexWriter suffers temporary short-term amnesia

[Michael McCandless (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless updated LUCENE-7894:
---
Attachment: LUCENE-7894.patch

Simple patch w/ test case showing the issue.  The problem was IW was failing to 
finish flushing a segment with the indexing thread that just wrote all the 
files for the segment.

> IndexWriter suffers temporary short-term amnesia
> 
>
> Key: LUCENE-7894
> URL: https://issues.apache.org/jira/browse/LUCENE-7894
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Blocker
> Fix For: master (7.0)
>
> Attachments: LUCENE-7894.patch
>
>
> Nightly benchmarks had been failing to run for a while due to trunk API 
> changes, but when I finally fixed those, the indexer fails because 
> {{IW.maxDoc}} disagrees with the number of documents indexed after all 
> threads are done indexing.
> It's sort of crazy none of our tests caught this!
> I tracked down the bug; it was caused in LUCENE-7868.  The bug is not as bad 
> as it sounds: documents are not in fact lost, it's just IW's internal 
> accounting for maxDoc that's temporarily incorrect until you do a refresh or 
> commit or close the index.
> I'm marking as blocker for 7.0.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-7894) IndexWriter suffers temporary short-term amnesia

[Michael McCandless (JIRA)]

Michael McCandless created LUCENE-7894:
--

 Summary: IndexWriter suffers temporary short-term amnesia
 Key: LUCENE-7894
 URL: https://issues.apache.org/jira/browse/LUCENE-7894
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Blocker
 Fix For: master (7.0)


Nightly benchmarks had been failing to run for a while due to trunk API 
changes, but when I finally fixed those, the indexer fails because 
{{IW.maxDoc}} disagrees with the number of documents indexed after all threads 
are done indexing.

It's sort of crazy none of our tests caught this!

I tracked down the bug; it was caused in LUCENE-7868.  The bug is not as bad as 
it sounds: documents are not in fact lost, it's just IW's internal accounting 
for maxDoc that's temporarily incorrect until you do a refresh or commit or 
close the index.

I'm marking as blocker for 7.0.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-5822) Create a markdown formatted README file for Lucene/Solr

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated LUCENE-5822:
--
Attachment: LUCENE-5822-addemdum.patch

[~steve_rowe] I have a fix for this, but I'm not sure how to test the smoke 
tester without building a full release, and I don't know how to build a full 
release either. Could commit it and wait for Jenkins to yell at me again I 
guess?

> Create a markdown formatted README file for Lucene/Solr
> ---
>
> Key: LUCENE-5822
> URL: https://issues.apache.org/jira/browse/LUCENE-5822
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Shalin Shekhar Mangar
>Assignee: Mike Drob
> Fix For: master (7.0)
>
> Attachments: LUCENE-5822-addemdum.patch, LUCENE-5822.patch, 
> LUCENE-5822.patch
>
>
> We have a minimal plain text readme file right now. Github is very popular 
> these days and we are even accepting pull requests from there. I think we 
> should add a proper markdown formatted README file which not only talks about 
> the features of Lucene/Solr but also include a short tutorial on both Lucene 
> and Solr.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Reopened] (LUCENE-5822) Create a markdown formatted README file for Lucene/Solr

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob reopened LUCENE-5822:
---

> Create a markdown formatted README file for Lucene/Solr
> ---
>
> Key: LUCENE-5822
> URL: https://issues.apache.org/jira/browse/LUCENE-5822
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Shalin Shekhar Mangar
>Assignee: Mike Drob
> Fix For: master (7.0)
>
> Attachments: LUCENE-5822.patch, LUCENE-5822.patch
>
>
> We have a minimal plain text readme file right now. Github is very popular 
> these days and we are even accepting pull requests from there. I think we 
> should add a proper markdown formatted README file which not only talks about 
> the features of Lucene/Solr but also include a short tutorial on both Lucene 
> and Solr.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Solr and HTTP headers

[Jan Høydahl]

Hi,

Inspired by SOLR-10981 "Allow update to load gzip files” where the proposal is 
to obey the
Content-Encoding HTTP request header to update a compressed stream, I started 
looking at other
headers to do things in more industry-standard ways.

Accept:

  Advertises which content types, expressed as MIME types, the client is able 
to understand 
  https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept

  Could replace or at least be an alternative to “wt”. Examples:
  Accept: application/xml
  Accept: text/csv

  Issue: Most browsers sends a long accept header, typically 
application/xml,text/html,*/*, and now
  that json is default for Solr, we’d need to serve JSON if the accept header 
includes “*/*"
  

Accept-Encoding:

  Advertises which content encoding, usually a compression algorithm, the 
client is able to understand
  https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding

  Could enable compression of large search results. SOLR-856 suggests that this 
is implemented,
  but it does not work. Seems it is only implemented for replication. I’d 
expect this to be useful for
  large /export or /stream requests. Example:
  Accept-Encoding: gzip



What do you think?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5822) Create a markdown formatted README file for Lucene/Solr

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070725#comment-16070725
 ] 

Steve Rowe commented on LUCENE-5822:


Looks like the smoke tester is angry - from 
[https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/799/]:

{noformat}
RuntimeError: file "README.txt" is missing from artifact solr-7.0.0-src.tgz
{noformat}

> Create a markdown formatted README file for Lucene/Solr
> ---
>
> Key: LUCENE-5822
> URL: https://issues.apache.org/jira/browse/LUCENE-5822
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Shalin Shekhar Mangar
>Assignee: Mike Drob
> Fix For: master (7.0)
>
> Attachments: LUCENE-5822.patch, LUCENE-5822.patch
>
>
> We have a minimal plain text readme file right now. Github is very popular 
> these days and we are even accepting pull requests from there. I think we 
> should add a proper markdown formatted README file which not only talks about 
> the features of Lucene/Solr but also include a short tutorial on both Lucene 
> and Solr.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7925) Implement indexing from gzip format file

[Wendy Tao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070693#comment-16070693
 ] 

Wendy Tao commented on SOLR-7925:
-

Hi Jan,

Thank you for keeping posted with new update. I finally switched to index 
database and gave up on indexing gzip files.  Thanks! 

> Implement indexing from gzip format file
> 
>
> Key: SOLR-7925
> URL: https://issues.apache.org/jira/browse/SOLR-7925
> Project: Solr
>  Issue Type: Improvement
>  Components: update
>Affects Versions: 5.2.1
>Reporter: Song　Hyonwoo
>Priority: Minor
>  Labels: patch
> Attachments: SOLR-7925.patch
>
>
> This will support the update of gzipped format file of Json, Xml and CSV.
> The request path will use "update/compress/gzip" instead of "update" with 
> "update.contentType" parameter  and  "Content-Type: application/gzip" as 
> Header field.
> The following is sample request using curl command. (use not --data but 
> --data-binary)
> curl 
> "http://localhost:8080/solr/collection1/update/compress/gzip?update.contentType=application/json&commit=true";
>  -H 'Content-Type: application/gzip' --data-binary @data.json.gz
> To activate this function need to add following request handler information 
> to solrconfig.xml
>class="org.apache.solr.handler.CompressedUpdateRequestHandler">
> 
>   application/gzip
> 
>   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-6.x-Linux (64bit/jdk-9-ea+175) - Build # 3863 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/3863/
Java: 64bit/jdk-9-ea+175 -XX:-UseCompressedOops -XX:+UseSerialGC 
--illegal-access=deny

1 tests failed.
FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
at 
__randomizedtesting.SeedInfo.seed([2A62BD895CEBFC36:A1456E581DED57B2]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:187)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
at 
org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey(ShardSplitTest.java:437)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
o

[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_131) - Build # 6693 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6693/
Java: 32bit/jdk1.8.0_131 -client -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.handler.V2ApiIntegrationTest.testCollectionsApi

Error Message:
Error from server at http://127.0.0.1:62747/solr: 
java.nio.file.InvalidPathException: Illegal char <�> at index 53: 
C:UsersjenkinsworkspaceLucene-Solr-master-Windowssolr�uildsolr-core estJ0 
empsolr.handler.V2ApiIntegrationTest_2FCBA65E4E932D4F-001 empDir-002

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: 
Error from server at http://127.0.0.1:62747/solr: 
java.nio.file.InvalidPathException: Illegal char <�> at index 53: 
C:UsersjenkinsworkspaceLucene-Solr-master-Windowssolr�uildsolr-core  estJ0  
 empsolr.handler.V2ApiIntegrationTest_2FCBA65E4E932D4F-001   empDir-002
at 
__randomizedtesting.SeedInfo.seed([2FCBA65E4E932D4F:F35541DFF68A49F1]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:804)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:600)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:470)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:400)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1102)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:843)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.handler.V2ApiIntegrationTest.testCollectionsApi(V2ApiIntegrationTest.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.random

[JENKINS] Lucene-Solr-SmokeRelease-master - Build # 799 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/799/

No tests ran.

Build Log:
[...truncated 25704 lines...]
prepare-release-no-sign:
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/dist
 [copy] Copying 476 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/dist/lucene
 [copy] Copying 215 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/dist/solr
   [smoker] Java 1.8 JAVA_HOME=/home/jenkins/tools/java/latest1.8
   [smoker] NOTE: output encoding is UTF-8
   [smoker] 
   [smoker] Load release URL 
"file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/dist/"...
   [smoker] 
   [smoker] Test Lucene...
   [smoker]   test basics...
   [smoker]   get KEYS
   [smoker] 0.2 MB in 0.01 sec (23.1 MB/sec)
   [smoker]   check changes HTML...
   [smoker]   download lucene-7.0.0-src.tgz...
   [smoker] 29.5 MB in 0.02 sec (1223.0 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download lucene-7.0.0.tgz...
   [smoker] 69.0 MB in 0.06 sec (1183.8 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download lucene-7.0.0.zip...
   [smoker] 79.3 MB in 0.07 sec (1220.3 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   unpack lucene-7.0.0.tgz...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] test demo with 1.8...
   [smoker]   got 6165 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] check Lucene's javadoc JAR
   [smoker]   unpack lucene-7.0.0.zip...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] test demo with 1.8...
   [smoker]   got 6165 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] check Lucene's javadoc JAR
   [smoker]   unpack lucene-7.0.0-src.tgz...
   [smoker] make sure no JARs/WARs in src dist...
   [smoker] run "ant validate"
   [smoker] run tests w/ Java 8 and testArgs='-Dtests.slow=false'...
   [smoker] test demo with 1.8...
   [smoker]   got 213 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] generate javadocs w/ Java 8...
   [smoker] 
   [smoker] Crawl/parse...
   [smoker] 
   [smoker] Verify...
   [smoker]   confirm all releases have coverage in TestBackwardsCompatibility
   [smoker] find all past Lucene releases...
   [smoker] run TestBackwardsCompatibility..
   [smoker] success!
   [smoker] 
   [smoker] Test Solr...
   [smoker]   test basics...
   [smoker]   get KEYS
   [smoker] 0.2 MB in 0.00 sec (87.1 MB/sec)
   [smoker]   check changes HTML...
   [smoker]   download solr-7.0.0-src.tgz...
   [smoker] 50.2 MB in 0.05 sec (957.1 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download solr-7.0.0.tgz...
   [smoker] 141.6 MB in 0.13 sec (1055.2 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download solr-7.0.0.zip...
   [smoker] 142.6 MB in 0.14 sec (1013.5 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   unpack solr-7.0.0.tgz...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] unpack lucene-7.0.0.tgz...
   [smoker]   **WARNING**: skipping check of 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0/contrib/dataimporthandler-extras/lib/javax.mail-1.5.1.jar:
 it has javax.* classes
   [smoker]   **WARNING**: skipping check of 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0/contrib/dataimporthandler-extras/lib/activation-1.1.1.jar:
 it has javax.* classes
   [smoker] copying unpacked distribution for Java 8 ...
   [smoker] test solr example w/ Java 8...
   [smoker]   start Solr instance 
(log=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0-java8/solr-example.log)...
   [smoker] No process found for Solr node running on port 8983
   [smoker]   Running techproducts example on port 8983 from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0-java8
   [smoker] Creating Solr home directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0-java8/example/techproducts/solr
   [smoker] 
   [smoker] Starting up Solr on port 8983 using command:
   [smoker] "bin/solr" start -p 8983 -s "example/techproducts/solr"
   [smoker] 
   [smoker] Waiting up to 180 seconds to see Solr running on port 8983 [|]  
 [/]   [-]   [\]  
   [smoker] Started Solr server on port 8983 (pid=26236). Happy searching!
   [smo

[jira] [Updated] (SOLR-10991) Support removing top N influential observations in the regress Stream Evaluator

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-10991:
--
Description: 
Influential observations are *outliers* that have a large effect on the slope 
of a regression line. It is very useful to be able to automatically remove 
influential observations prior to running a simple regression.

Syntax:
{code}
regress(colA, colB, 10)
{code}

The function above regresses colA and colB after removing the top 10 
influential observations from the data set.

The approach taken will be to remove each observation one and at a time and 
re-run the regression on the data set minus the observation. After each run the 
difference in model fit will be recorded. After completing the regression runs, 
N observations that had the highest difference of fit will be removed from the 
data set. The final regression will be run without those observations.






  was:
Influential observations are *outliers* that have a large effect on the slope 
of a regression line. It is very useful to be able to automatically remove 
influential observations prior to running a simple regression.

Syntax:
{code}
regress(colA, colB, 10)
{code}

The function above regresses colA and colB after removing the top 10 
influential observations from the data set.







> Support removing top N influential observations in the regress Stream 
> Evaluator
> ---
>
> Key: SOLR-10991
> URL: https://issues.apache.org/jira/browse/SOLR-10991
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: master (7.0)
>
>
> Influential observations are *outliers* that have a large effect on the slope 
> of a regression line. It is very useful to be able to automatically remove 
> influential observations prior to running a simple regression.
> Syntax:
> {code}
> regress(colA, colB, 10)
> {code}
> The function above regresses colA and colB after removing the top 10 
> influential observations from the data set.
> The approach taken will be to remove each observation one and at a time and 
> re-run the regression on the data set minus the observation. After each run 
> the difference in model fit will be recorded. After completing the regression 
> runs, N observations that had the highest difference of fit will be removed 
> from the data set. The final regression will be run without those 
> observations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-master - Build # 1927 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1927/

2 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionAddReplica

Error Message:
Error from server at https://127.0.0.1:50364/solr: delete the collection time 
out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:50364/solr: delete the collection time out:180s
at 
__randomizedtesting.SeedInfo.seed([6B38FFB290F86398:EB189A9C81BB8B3E]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:624)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:470)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:400)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1102)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:843)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:442)
at 
org.apache.solr.cloud.autoscaling.TestPolicyCloud.after(TestPolicyCloud.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:965)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)

[jira] [Updated] (SOLR-10991) Support removing top N influential observations in the regress Stream Evaluator

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-10991:
--
Summary: Support removing top N influential observations in the regress 
Stream Evaluator  (was: Support removing top N influential observations in the 
simple regression Stream Evaluator)

> Support removing top N influential observations in the regress Stream 
> Evaluator
> ---
>
> Key: SOLR-10991
> URL: https://issues.apache.org/jira/browse/SOLR-10991
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: master (7.0)
>
>
> Influential observations are *outliers* that have a large effect on the slope 
> of a regression line. It is very useful to be able to automatically remove 
> influential observations prior to running a simple regression.
> Syntax:
> {code}
> regress(colA, colB, 10)
> {code}
> The function above regresses colA and colB after removing the top 10 
> influential observations from the data set.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7925) Implement indexing from gzip format file

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070624#comment-16070624
 ] 

Jan Høydahl commented on SOLR-7925:
---

See SOLR-10981 which probably has a smoother solution using {{Content-Encoding: 
gzip}} header

> Implement indexing from gzip format file
> 
>
> Key: SOLR-7925
> URL: https://issues.apache.org/jira/browse/SOLR-7925
> Project: Solr
>  Issue Type: Improvement
>  Components: update
>Affects Versions: 5.2.1
>Reporter: Song　Hyonwoo
>Priority: Minor
>  Labels: patch
> Attachments: SOLR-7925.patch
>
>
> This will support the update of gzipped format file of Json, Xml and CSV.
> The request path will use "update/compress/gzip" instead of "update" with 
> "update.contentType" parameter  and  "Content-Type: application/gzip" as 
> Header field.
> The following is sample request using curl command. (use not --data but 
> --data-binary)
> curl 
> "http://localhost:8080/solr/collection1/update/compress/gzip?update.contentType=application/json&commit=true";
>  -H 'Content-Type: application/gzip' --data-binary @data.json.gz
> To activate this function need to add following request handler information 
> to solrconfig.xml
>class="org.apache.solr.handler.CompressedUpdateRequestHandler">
> 
>   application/gzip
> 
>   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10991) Support removing top N influential observations in the simple regression Stream Evaluator

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-10991:
--
Description: 
Influential observations are *outliers* that have a large effect on the slope 
of a regression line. It is very useful to be able to automatically remove 
influential observations prior to running a simple regression.

Syntax:
{code}
regress(colA, colB, 10)
{code}

The function above regresses colA and colB after removing the top 10 
influential observations from the data set.






  was:Influential observations are outliers that have a large effect on the 
slope of a regression line. It is very useful to be able to automatically 
remove influential observations prior to running a simple regression.


> Support removing top N influential observations in the simple regression 
> Stream Evaluator
> -
>
> Key: SOLR-10991
> URL: https://issues.apache.org/jira/browse/SOLR-10991
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: master (7.0)
>
>
> Influential observations are *outliers* that have a large effect on the slope 
> of a regression line. It is very useful to be able to automatically remove 
> influential observations prior to running a simple regression.
> Syntax:
> {code}
> regress(colA, colB, 10)
> {code}
> The function above regresses colA and colB after removing the top 10 
> influential observations from the data set.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10991) Support removing top N influential observations in the simple regression Stream Evaluator

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-10991:
--
Fix Version/s: master (7.0)

> Support removing top N influential observations in the simple regression 
> Stream Evaluator
> -
>
> Key: SOLR-10991
> URL: https://issues.apache.org/jira/browse/SOLR-10991
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: master (7.0)
>
>
> Influential observations are outliers that have a large effect on the slope 
> of a regression line. It is very useful to be able to automatically remove 
> influential observations prior to running a simple regression.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-10991) Support removing top N influential observations in the simple regression Stream Evaluator

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein reassigned SOLR-10991:
-

Assignee: Joel Bernstein

> Support removing top N influential observations in the simple regression 
> Stream Evaluator
> -
>
> Key: SOLR-10991
> URL: https://issues.apache.org/jira/browse/SOLR-10991
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: master (7.0)
>
>
> Influential observations are outliers that have a large effect on the slope 
> of a regression line. It is very useful to be able to automatically remove 
> influential observations prior to running a simple regression.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10991) Support removing top N influential observations in the simple regression Stream Evaluator

[Joel Bernstein (JIRA)]

Joel Bernstein created SOLR-10991:
-

 Summary: Support removing top N influential observations in the 
simple regression Stream Evaluator
 Key: SOLR-10991
 URL: https://issues.apache.org/jira/browse/SOLR-10991
 Project: Solr
  Issue Type: New Feature
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Joel Bernstein


Influential observations are outliers that have a large effect on the slope of 
a regression line. It is very useful to be able to automatically remove 
influential observations prior to running a simple regression.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10981) Allow update to load gzip files

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070591#comment-16070591
 ] 

Jan Høydahl commented on SOLR-10981:


Sorry for confusing content-type and content-encoding. What you propose with 
the encoding sounds like the correct thing to do.
Have not looked closely on the whole patch, but what would it take to be able 
to POST a gzipped csv file using cURL?
{code}
curl -XPOST -H 'Content-Type: text/csv' -H 'Content-Encoding: gzip' 
http://solr.server:8983/solr/coll/update --data-binary @myfile.csv.gz
{code}

> Allow update to load gzip files 
> 
>
> Key: SOLR-10981
> URL: https://issues.apache.org/jira/browse/SOLR-10981
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6
>Reporter: Andrew Lundgren
>  Labels: patch
> Fix For: 4.10.4, 6.6, master (7.0)
>
> Attachments: SOLR-10981.patch, SOLR-10981.patch, SOLR-10981.patch
>
>
> We currently import large CSV files.  We store them in gzip files as they 
> compress at around 80%.
> To import them we must gunzip them and then import them.  After that we no 
> longer need the decompressed files.
> This patch allows directly opening either URL, or local files that are 
> gzipped.
> For URLs, to determine if the file is gzipped, it will check the content 
> encoding=="gzip" or if the file ends in ".gz"
> For files, if the file ends in ".gz" then it will assume the file is gzipped.
> I have tested the patch with 4.10.4, 6.6.0 and master from git.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-10981) Allow update to load gzip files

[Andrew Lundgren (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070510#comment-16070510
 ] 

Andrew Lundgren edited comment on SOLR-10981 at 6/30/17 6:56 PM:
-

Updated code to handle .gzip as well .gz and mixed case.  
Updated FileStream to use suffixes to determine ContentType before looking 
inside file.
Added .cvs extension support to FileStream ContentType
Modified FileStream ContentType so that when it does look inside the file it 
handles whitespaces and end of line.

FileStream and file:// URLStream now behave consistently when application/gzip, 
application/octet-stream, content/unknown are returned.



was (Author: lundgren):
Updated code to handle .gzip as well as mixed case.  
Updated FileStream to use suffixes to determine ContentType before looking 
inside file.
Added .cvs extension support to FileStream ContentType
Modified FileStream ContentType so that when it does look inside the file it 
handles whitespaces and end of line.

FileStream and file:// URLStream now behave consistently when application/gzip, 
application/octet-stream, content/unknown are returned.


> Allow update to load gzip files 
> 
>
> Key: SOLR-10981
> URL: https://issues.apache.org/jira/browse/SOLR-10981
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6
>Reporter: Andrew Lundgren
>  Labels: patch
> Fix For: 4.10.4, 6.6, master (7.0)
>
> Attachments: SOLR-10981.patch, SOLR-10981.patch, SOLR-10981.patch
>
>
> We currently import large CSV files.  We store them in gzip files as they 
> compress at around 80%.
> To import them we must gunzip them and then import them.  After that we no 
> longer need the decompressed files.
> This patch allows directly opening either URL, or local files that are 
> gzipped.
> For URLs, to determine if the file is gzipped, it will check the content 
> encoding=="gzip" or if the file ends in ".gz"
> For files, if the file ends in ".gz" then it will assume the file is gzipped.
> I have tested the patch with 4.10.4, 6.6.0 and master from git.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 938 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/938/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.core.TestDynamicLoading.testDynamicLoading

Error Message:
Could not get expected value  
'org.apache.solr.core.BlobStoreTestRequestHandler' for path 
'overlay/requestHandler/\/test1/class' full output: {   "responseHeader":{ 
"status":0, "QTime":0},   "overlay":{ "znodeVersion":0, 
"runtimeLib":{"colltest":{ "name":"colltest", "version":1,  
from server:  null

Stack Trace:
java.lang.AssertionError: Could not get expected value  
'org.apache.solr.core.BlobStoreTestRequestHandler' for path 
'overlay/requestHandler/\/test1/class' full output: {
  "responseHeader":{
"status":0,
"QTime":0},
  "overlay":{
"znodeVersion":0,
"runtimeLib":{"colltest":{
"name":"colltest",
"version":1,  from server:  null
at 
__randomizedtesting.SeedInfo.seed([3F8ECAB4437C7E8D:E7C3E7E3B4A1DB2D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:556)
at 
org.apache.solr.core.TestDynamicLoading.testDynamicLoading(TestDynamicLoading.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementA

[jira] [Commented] (SOLR-9526) data_driven configs defaults to "strings" for unmapped fields, makes most fields containing "textual content" unsearchable, breaks tutorial examples

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070570#comment-16070570
 ] 

Steve Rowe commented on SOLR-9526:
--

bq. Steve Rowe please fill in your wisdom regarding my question above 

Sure, sorry for the delay, I'll investigate today and let you know what I find. 
 (It's been long enough that I don't remember the situation there.)

> data_driven configs defaults to "strings" for unmapped fields, makes most 
> fields containing "textual content" unsearchable, breaks tutorial examples
> 
>
> Key: SOLR-9526
> URL: https://issues.apache.org/jira/browse/SOLR-9526
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Reporter: Hoss Man
>Assignee: Jan Høydahl
>  Labels: dynamic-schema
> Fix For: master (7.0)
>
> Attachments: SOLR-9526.patch, SOLR-9526.patch, SOLR-9526.patch, 
> SOLR-9526.patch
>
>
> James Pritchett pointed out on the solr-user list that this sample query from 
> the quick start tutorial matched no docs (even though the tutorial text says 
> "The above request returns only one document")...
> http://localhost:8983/solr/gettingstarted/select?wt=json&indent=true&q=name:foundation
> The root problem seems to be that the add-unknown-fields-to-the-schema chain 
> in data_driven_schema_configs is configured with...
> {code}
> strings
> {code}
> ...and the "strings" type uses StrField and is not tokenized.
> 
> Original thread: 
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201609.mbox/%3ccac-n2zrpsspfnk43agecspchc5b-0ff25xlfnzogyuvyg2d...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10032) Create report to assess Solr test quality at a commit point.

[Anshum Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070530#comment-16070530
 ] 

Anshum Gupta commented on SOLR-10032:
-

This is looking really good!

> Create report to assess Solr test quality at a commit point.
> 
>
> Key: SOLR-10032
> URL: https://issues.apache.org/jira/browse/SOLR-10032
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Mark Miller
>Assignee: Mark Miller
> Attachments: Lucene-Solr Master Test Beast Results 
> 01-24-2017-9899cbd031dc3fc37a384b1f9e2b379e90a9a3a6 Level Medium- Running 30 
> iterations, 12 at a time .pdf, Lucene-Solr Master Test Beasults 
> 02-01-2017-bbc455de195c83d9f807980b510fa46018f33b1b Level Medium- Running 30 
> iterations, 10 at a time.pdf, Lucene-Solr Master Test Beasults 
> 02-08-2017-6696eafaae18948c2891ce758c7a2ec09873dab8 Level Medium+- Running 30 
> iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults 
> 02-14-2017- Level Medium+-a1f114f70f3800292c25be08213edf39b3e37f6a Running 30 
> iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults 
> 02%2F17%2F2017-19c8ec2bf1882bed1bb34d0b55198d03f2018838 Level Hard Running 
> 100 iterations, 12 at a time, 8 cores.pdf
>
>
> We have many Jenkins instances blasting tests, some official, some policeman, 
> I and others have or had their own, and the email trail proves the power of 
> the Jenkins cluster to find test fails.
> However, I still have a very hard time with some basic questions:
> what tests are flakey right now? which test fails actually affect devs most? 
> did I break it? was that test already flakey? is that test still flakey? what 
> are our worst tests right now? is that test getting better or worse?
> We really need a way to see exactly what tests are the problem, not because 
> of OS or environmental issues, but more basic test quality issues. Which 
> tests are flakey and how flakey are they at any point in time.
> Reports:
> https://drive.google.com/drive/folders/0ByYyjsrbz7-qa2dOaU1UZDdRVzg?usp=sharing
> 01/24/2017 - 
> https://docs.google.com/spreadsheets/d/1JySta2j2s7A_p16wA1UO-l6c4GsUHBIb4FONS2EzW9k/edit?usp=sharing
> 02/01/2017 - 
> https://docs.google.com/spreadsheets/d/1FndoyHmihaOVL2o_Zns5alpNdAJlNsEwQVoJ4XDWj3c/edit?usp=sharing
> 02/08/2017 - 
> https://docs.google.com/spreadsheets/d/1N6RxH4Edd7ldRIaVfin0si-uSLGyowQi8-7mcux27S0/edit?usp=sharing
> 02/14/2017 - 
> https://docs.google.com/spreadsheets/d/1eZ9_ds_0XyqsKKp8xkmESrcMZRP85jTxSKkNwgtcUn0/edit?usp=sharing
> 02/17/2017 - 
> https://docs.google.com/spreadsheets/d/1LEPvXbsoHtKfIcZCJZ3_P6OHp7S5g2HP2OJgU6B2sAg/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10864) Add static (test only) boolean to PointField indicating 'precisionStep' should be ignored so we can simplify points randomization in our schemas

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070518#comment-16070518
 ] 

ASF subversion and git services commented on SOLR-10864:


Commit f3c851a015e5d8c775f0fb28ffbd8f5725f2f11b in lucene-solr's branch 
refs/heads/master from Chris Hostetter
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f3c851a ]

SOLR-10864: Fixup: Restore the DV vs trie vs points intent of 
TestRandomDVFaceting


> Add static (test only) boolean to PointField indicating 'precisionStep' 
> should be ignored so we can simplify points randomization in our schemas
> 
>
> Key: SOLR-10864
> URL: https://issues.apache.org/jira/browse/SOLR-10864
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Assignee: Hoss Man
> Fix For: master (7.0)
>
> Attachments: SOLR-10864.patch, SOLR-10864.patch, SOLR-10864.patch, 
> SOLR-10864.patch, SOLR-10864.patch, SOLR-10864.patch
>
>
> (I'm spinning this idea out of parent jira SOLR-10807 so that it gets it's 
> own jira# w/ it's own summary for increased visibility/comments)
> In the interest of making it easier & more straight forward to get good 
> randomized test coverage of Points fields, I'd like to add the following to 
> the {{PointField}} class...
> {code}
>  /**
>   * 
>   * The Test framework can set this global variable to instruct PointField 
> that
>   * (on init) it should be tollerant of the precisionStep 
> argument used by TrieFields.
>   * This allows for simple randomization of TrieFields and PointFields w/o 
> extensive duplication
>   * of  declarations.
>   * 
>   *
>   * NOTE: When {@link TrieField} is removed, this boolean must also be 
> removed
>   *
>   * @lucene.internal
>   * @lucene.experimental
>   */
>  public static boolean TEST_HACK_IGNORE_USELESS_TRIEFIELD_ARGS = false;
>  /** 
>   * NOTE: This method can be removed completely when
>   * {@link #TEST_HACK_IGNORE_USELESS_TRIEFIELD_ARGS} is removed 
>   */
>  @Override
>  protected void init(IndexSchema schema, Map args) {
>super.init(schema, args);
>if (TEST_HACK_IGNORE_USELESS_TRIEFIELD_ARGS) {
>  args.remove("precisionStep");
>}
>  }
> {code}
> Then in SolrTestCaseJ4, set 
> {{PointField.TEST_HACK_IGNORE_USELESS_TRIEFIELD_ARGS}} on a class by class 
> basis when randomizing Trie/Points (and unset \@AfterClass).
> (details to follow in comment)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 7x, and 7.0 branches

[Anshum Gupta]

Hi Uwe/Adrien, 

I reverted a bunch of changes to only get rid of the 5x format codec names and 
it brings me to these errors:

 [javac] 
/Users/anshum/workspace/anshumg/lucene-solr/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene60/Lucene60Codec.java:35:
 error: cannot find symbol
[javac] import org.apache.lucene.codecs.lucene50.Lucene50SegmentInfoFormat;
[javac] ^
[javac]   symbol:   class Lucene50SegmentInfoFormat
[javac]   location: package org.apache.lucene.codecs.lucene50
[javac] 
/Users/anshum/workspace/anshumg/lucene-solr/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene60/Lucene60Codec.java:39:
 error: package org.apache.lucene.codecs.lucene53 does not exist
[javac] import org.apache.lucene.codecs.lucene53.Lucene53NormsFormat;
[javac] ^
[javac] 
/Users/anshum/workspace/anshumg/lucene-solr/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene62/Lucene62Codec.java:38:
 error: package org.apache.lucene.codecs.lucene53 does not exist
[javac] import org.apache.lucene.codecs.lucene53.Lucene53NormsFormat;
[javac] ^
[javac] 
/Users/anshum/workspace/anshumg/lucene-solr/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene60/Lucene60Codec.java:59:
 error: cannot find symbol
[javac]   private final SegmentInfoFormat segmentInfosFormat = new 
Lucene50SegmentInfoFormat();
[javac]^
[javac]   symbol:   class Lucene50SegmentInfoFormat
[javac]   location: class Lucene60Codec
[javac] 
/Users/anshum/workspace/anshumg/lucene-solr/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene60/Lucene60Codec.java:171:
 error: cannot find symbol
[javac]   private final NormsFormat normsFormat = new Lucene53NormsFormat();
[javac]   ^
[javac]   symbol:   class Lucene53NormsFormat
[javac]   location: class Lucene60Codec
[javac] 
/Users/anshum/workspace/anshumg/lucene-solr/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene62/Lucene62Codec.java:170:
 error: cannot find symbol
[javac]   private final NormsFormat normsFormat = new Lucene53NormsFormat();
[javac]   ^
[javac]   symbol:   class Lucene53NormsFormat
[javac]   location: class Lucene62Codec
[javac] 6 errors

As you mentioned, the old codecs are still used, but I thought we wouldn’t need 
the 5x named formats. Should we just keep all of this?

I also recreated my fork and applied the patch from the older fork, instead of 
running the upgrade script, only so that I didn’t have to run the upgrade 
script. The upgrade script doesn’t work on anything but the master branch for a 
major version bump.

All my changes are now here: 
https://github.com/anshumg/lucene-solr/tree/upgrade-master-to-8 
 


-Anshum



> On Jun 29, 2017, at 4:18 PM, Uwe Schindler  wrote:
> 
> The problem is that old 7.x indexes still use some codecs named by version 6. 
> They were never updated!
> 
> So backwards codec must keep all stuff in metainf and as classes that the 7.0 
> original index format requires. Maybe create a dummy 7.0 index in branch-7x 
> to have a list of codecs to test.
> 
> Uwe
> 
> Am 30. Juni 2017 00:43:06 MESZ schrieb Anshum Gupta :
> I’ve pushed more changes there, and we have a new set of errors. This is one 
> of them:
> 
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestBackwardsCompatibility 
> -Dtests.method=testUnsupportedOldIndexes -Dtests.seed=8FDA7D3598A2FB46 
> -Dtests.slow=true -Dtests.locale=ar-LB 
> -Dtests.timezone=America/Indiana/Marengo -Dtests.asserts=true 
> -Dtests.file.encoding=UTF-8
>[junit4] ERROR   3.07s | 
> TestBackwardsCompatibility.testUnsupportedOldIndexes <<<
>[junit4]> Throwable #1: java.lang.IllegalArgumentException: Could not 
> load codec 'Lucene60'.  Did you forget to add lucene-backward-codecs.jar?
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([8FDA7D3598A2FB46:74214F1628395C1A]:0)
>[junit4]>  at 
> org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:433)
>[junit4]>  at 
> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:360)
>[junit4]>  at 
> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:295)
>[junit4]>  at 
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:59)
>[junit4]>  at 
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:56)
>[junit4]>  at 
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:694)
>[junit4]>  at 
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:

[jira] [Comment Edited] (SOLR-10981) Allow update to load gzip files

[Andrew Lundgren (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070510#comment-16070510
 ] 

Andrew Lundgren edited comment on SOLR-10981 at 6/30/17 6:22 PM:
-

Updated code to handle .gzip as well as mixed case.  
Updated FileStream to use suffixes to determine ContentType before looking 
inside file.
Added .cvs extension support to FileStream ContentType
Modified FileStream ContentType so that when it does look inside the file it 
handles whitespaces and end of line.

FileStream and file:// URLStream now behave consistently when application/gzip, 
application/octet-stream, content/unknown are returned.



was (Author: lundgren):
Updated code to handle .gzip as well as mixed case.  
Updated FileStream to use suffixes to determine ContentType before looking 
inside file.
Added .cvs extension support to FileStream ContentType
Modified FileStream ContentType so that when it does look inside the file it 
handles whitespaces and end of line.

FileStream and file:// URLStream now behave consistently for certain mime-types.



> Allow update to load gzip files 
> 
>
> Key: SOLR-10981
> URL: https://issues.apache.org/jira/browse/SOLR-10981
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6
>Reporter: Andrew Lundgren
>  Labels: patch
> Fix For: 4.10.4, 6.6, master (7.0)
>
> Attachments: SOLR-10981.patch, SOLR-10981.patch, SOLR-10981.patch
>
>
> We currently import large CSV files.  We store them in gzip files as they 
> compress at around 80%.
> To import them we must gunzip them and then import them.  After that we no 
> longer need the decompressed files.
> This patch allows directly opening either URL, or local files that are 
> gzipped.
> For URLs, to determine if the file is gzipped, it will check the content 
> encoding=="gzip" or if the file ends in ".gz"
> For files, if the file ends in ".gz" then it will assume the file is gzipped.
> I have tested the patch with 4.10.4, 6.6.0 and master from git.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-10981) Allow update to load gzip files

[Andrew Lundgren (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070510#comment-16070510
 ] 

Andrew Lundgren edited comment on SOLR-10981 at 6/30/17 6:21 PM:
-

Updated code to handle .gzip as well as mixed case.  
Updated FileStream to use suffixes to determine ContentType before looking 
inside file.
Added .cvs extension support to FileStream ContentType
Modified FileStream ContentType so that when it does look inside the file it 
handles whitespaces and end of line.

FileStream and file:// URLStream now behave consistently for certain mime-types.




was (Author: lundgren):
Updated code to handle .gzip as well as mixed case.  
Updated FileStream to use suffixes to determine ContentType before looking 
inside file.
Added .cvs extension support to FileStream ContentType
Modified FileStream ContentType so that when it does look inside the file it 
handles whitespaces and end of line.



> Allow update to load gzip files 
> 
>
> Key: SOLR-10981
> URL: https://issues.apache.org/jira/browse/SOLR-10981
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6
>Reporter: Andrew Lundgren
>  Labels: patch
> Fix For: 4.10.4, 6.6, master (7.0)
>
> Attachments: SOLR-10981.patch, SOLR-10981.patch, SOLR-10981.patch
>
>
> We currently import large CSV files.  We store them in gzip files as they 
> compress at around 80%.
> To import them we must gunzip them and then import them.  After that we no 
> longer need the decompressed files.
> This patch allows directly opening either URL, or local files that are 
> gzipped.
> For URLs, to determine if the file is gzipped, it will check the content 
> encoding=="gzip" or if the file ends in ".gz"
> For files, if the file ends in ".gz" then it will assume the file is gzipped.
> I have tested the patch with 4.10.4, 6.6.0 and master from git.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-10981) Allow update to load gzip files

[Andrew Lundgren (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070259#comment-16070259
 ] 

Andrew Lundgren edited comment on SOLR-10981 at 6/30/17 6:20 PM:
-

As this doesn't use the content-type header, it uses the content-encoding 
header, it does not interfere with the existing content-type header usage. 

In the patched ContentStreamBase, line 84 the content type is taken from the 
connection.  As this is not changed by the gzip contentEncoding header on line 
89; code using the content stream is unaffected.  If the contentEncoding is not 
set, then the code will also detect if the file ends with ".gz".  This could be 
dropped, though it seemed a reasonable usage.

In the patched ContentStreamBase, lines 117, 121 the content type of a 
FileStream is determined by the first character found in the stream.  As the 
stream is already opened and the gunzip stream applied over the input stream, 
the code that determines the content type is unaffected.  The FileStream will 
work with any existing format that is gzipped as it determines the content type 
based on the first character of the decompressed stream.  (Attached is a new 
patch that causes this method to use the getStream method on 117 rather than 
open the file itself applying the gzip layer)

I agree that using generic {{Content-Type: application/gzip}} would lead to 
confusion.  To me, the gzip layer is the encoding of the content, not the type 
itself.  By using the encoding type you are able to handle the gzip at a lower 
layer, and keep all of your content-type support untouched.

See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
See: https://www.iana.org/assignments/media-types/media-types.xhtml

The current handling of a FileStream and a file:// URL are inconsistent, as the 
FileStream tries to guess the content type based on the first character.  The 
file:// URL uses mime-types to determine the content.   They seemingly should 
be consistent, though I did not try to make them consistent, as the FileStream 
implementation point's out, it's implementation is buggy.



was (Author: lundgren):
As this doesn't use the content-type header, it uses the content-encoding 
header, it does not interfere with the existing content-type header usage. 

In the patched ContentStreamBase, line 84 the content type is taken from the 
connection.  As this is not changed by the gzip contentEncoding header on line 
89; code using the content stream is unaffected.  If the contentEncoding is not 
set, then the code will also detect if the file ends with ".gz".  This could be 
dropped, though it seemed a reasonable usage.

In the patched ContentStreamBase, lines 117, 121 the content type of a 
FileStream is determined by the first character found in the stream.  As the 
stream is already opened and the gunzip stream applied over the input stream, 
the code that determines the content type is unaffected.  The FileStream will 
work with any existing format that is gzipped as it determines the content type 
based on the first character of the decompressed stream.  (Attached is a new 
patch that causes this method to use the getStream method on 117 rather than 
open the file itself applying the gzip layer)

I agree that using generic {{Content-Type: application/gzip}} would lead to 
confusion.  To me, the gzip layer is the encoding of the content, not the type 
itself.  By using the encoding type you are able to handle the gzip at a lower 
layer, and keep all of your content-type support untouched.

The current handling of a FileStream and a file:// URL are inconsistent, as the 
FileStream tries to guess the content type based on the first character.  The 
file:// URL uses mime-types to determine the content.   They seemingly should 
be consistent, though I did not try to make them consistent, as the FileStream 
implementation point's out, it's implementation is buggy.


> Allow update to load gzip files 
> 
>
> Key: SOLR-10981
> URL: https://issues.apache.org/jira/browse/SOLR-10981
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6
>Reporter: Andrew Lundgren
>  Labels: patch
> Fix For: 4.10.4, 6.6, master (7.0)
>
> Attachments: SOLR-10981.patch, SOLR-10981.patch, SOLR-10981.patch
>
>
> We currently import large CSV files.  We store them in gzip files as they 
> compress at around 80%.
> To import them we must gunzip them and then import them.  After that we no 
> longer need the decompressed files.
> This patch allows directly opening either URL, or local files that are 
> gzipped.
> For URLs, to determine if the file is gzipped, it will check the c

[jira] [Updated] (SOLR-10981) Allow update to load gzip files

[Andrew Lundgren (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lundgren updated SOLR-10981:
---
Attachment: SOLR-10981.patch

Updated code to handle .gzip as well as mixed case.  
Updated FileStream to use suffixes to determine ContentType before looking 
inside file.
Added .cvs extension support to FileStream ContentType
Modified FileStream ContentType so that when it does look inside the file it 
handles whitespaces and end of line.



> Allow update to load gzip files 
> 
>
> Key: SOLR-10981
> URL: https://issues.apache.org/jira/browse/SOLR-10981
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6
>Reporter: Andrew Lundgren
>  Labels: patch
> Fix For: 4.10.4, 6.6, master (7.0)
>
> Attachments: SOLR-10981.patch, SOLR-10981.patch, SOLR-10981.patch
>
>
> We currently import large CSV files.  We store them in gzip files as they 
> compress at around 80%.
> To import them we must gunzip them and then import them.  After that we no 
> longer need the decompressed files.
> This patch allows directly opening either URL, or local files that are 
> gzipped.
> For URLs, to determine if the file is gzipped, it will check the content 
> encoding=="gzip" or if the file ends in ".gz"
> For files, if the file ends in ".gz" then it will assume the file is gzipped.
> I have tested the patch with 4.10.4, 6.6.0 and master from git.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10892) Ref Guide: Move parameters out of tables

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070490#comment-16070490
 ] 

Cassandra Targett commented on SOLR-10892:
--

Making myself a list of pages left to review/modify for my own reference:

* common-query-parameters.adoc
* faceting.adoc
* learning-to-rank.adoc
* performance-statistics-reference.adoc
* spell-checking.adoc
* suggester.adoc
* the-dismax-query-parser.adoc
* the-standard-query-parser.adoc
* working-with-external-files-and-processes.adoc

> Ref Guide: Move parameters out of tables
> 
>
> Key: SOLR-10892
> URL: https://issues.apache.org/jira/browse/SOLR-10892
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>Assignee: Cassandra Targett
> Fix For: master (7.0)
>
>
> We've overused a bit the concept of using tables to explain various config 
> parameters. We have some tables that are massive and try to cram a ton of 
> complex information into a row (see function-queries.adoc), while other 
> tables are only 1 or 2 rows. It's not impossible, but it's also difficult to 
> link directly to parameters when they are in a table
> AsciiDoc format now allows us to use "description lists" or "definition 
> lists" which in many cases might be better. This issue would change many of 
> the current tables to definition lists. However, some of them may remain, 
> depending on how they work within the content.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10032) Create report to assess Solr test quality at a commit point.

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070484#comment-16070484
 ] 

Erick Erickson commented on SOLR-10032:
---

Sweet!

> Create report to assess Solr test quality at a commit point.
> 
>
> Key: SOLR-10032
> URL: https://issues.apache.org/jira/browse/SOLR-10032
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Mark Miller
>Assignee: Mark Miller
> Attachments: Lucene-Solr Master Test Beast Results 
> 01-24-2017-9899cbd031dc3fc37a384b1f9e2b379e90a9a3a6 Level Medium- Running 30 
> iterations, 12 at a time .pdf, Lucene-Solr Master Test Beasults 
> 02-01-2017-bbc455de195c83d9f807980b510fa46018f33b1b Level Medium- Running 30 
> iterations, 10 at a time.pdf, Lucene-Solr Master Test Beasults 
> 02-08-2017-6696eafaae18948c2891ce758c7a2ec09873dab8 Level Medium+- Running 30 
> iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults 
> 02-14-2017- Level Medium+-a1f114f70f3800292c25be08213edf39b3e37f6a Running 30 
> iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults 
> 02%2F17%2F2017-19c8ec2bf1882bed1bb34d0b55198d03f2018838 Level Hard Running 
> 100 iterations, 12 at a time, 8 cores.pdf
>
>
> We have many Jenkins instances blasting tests, some official, some policeman, 
> I and others have or had their own, and the email trail proves the power of 
> the Jenkins cluster to find test fails.
> However, I still have a very hard time with some basic questions:
> what tests are flakey right now? which test fails actually affect devs most? 
> did I break it? was that test already flakey? is that test still flakey? what 
> are our worst tests right now? is that test getting better or worse?
> We really need a way to see exactly what tests are the problem, not because 
> of OS or environmental issues, but more basic test quality issues. Which 
> tests are flakey and how flakey are they at any point in time.
> Reports:
> https://drive.google.com/drive/folders/0ByYyjsrbz7-qa2dOaU1UZDdRVzg?usp=sharing
> 01/24/2017 - 
> https://docs.google.com/spreadsheets/d/1JySta2j2s7A_p16wA1UO-l6c4GsUHBIb4FONS2EzW9k/edit?usp=sharing
> 02/01/2017 - 
> https://docs.google.com/spreadsheets/d/1FndoyHmihaOVL2o_Zns5alpNdAJlNsEwQVoJ4XDWj3c/edit?usp=sharing
> 02/08/2017 - 
> https://docs.google.com/spreadsheets/d/1N6RxH4Edd7ldRIaVfin0si-uSLGyowQi8-7mcux27S0/edit?usp=sharing
> 02/14/2017 - 
> https://docs.google.com/spreadsheets/d/1eZ9_ds_0XyqsKKp8xkmESrcMZRP85jTxSKkNwgtcUn0/edit?usp=sharing
> 02/17/2017 - 
> https://docs.google.com/spreadsheets/d/1LEPvXbsoHtKfIcZCJZ3_P6OHp7S5g2HP2OJgU6B2sAg/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10032) Create report to assess Solr test quality at a commit point.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070480#comment-16070480
 ] 

Mark Miller commented on SOLR-10032:


As part of fully automating everything, I've been experimented with sending 
results to bitballoon rather than cut and pasting into Google sheets. It's dead 
simple to automate and a nicer experience I think. Here is a demo of results 
http://solr-tests.bitballoon.com

> Create report to assess Solr test quality at a commit point.
> 
>
> Key: SOLR-10032
> URL: https://issues.apache.org/jira/browse/SOLR-10032
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Mark Miller
>Assignee: Mark Miller
> Attachments: Lucene-Solr Master Test Beast Results 
> 01-24-2017-9899cbd031dc3fc37a384b1f9e2b379e90a9a3a6 Level Medium- Running 30 
> iterations, 12 at a time .pdf, Lucene-Solr Master Test Beasults 
> 02-01-2017-bbc455de195c83d9f807980b510fa46018f33b1b Level Medium- Running 30 
> iterations, 10 at a time.pdf, Lucene-Solr Master Test Beasults 
> 02-08-2017-6696eafaae18948c2891ce758c7a2ec09873dab8 Level Medium+- Running 30 
> iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults 
> 02-14-2017- Level Medium+-a1f114f70f3800292c25be08213edf39b3e37f6a Running 30 
> iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults 
> 02%2F17%2F2017-19c8ec2bf1882bed1bb34d0b55198d03f2018838 Level Hard Running 
> 100 iterations, 12 at a time, 8 cores.pdf
>
>
> We have many Jenkins instances blasting tests, some official, some policeman, 
> I and others have or had their own, and the email trail proves the power of 
> the Jenkins cluster to find test fails.
> However, I still have a very hard time with some basic questions:
> what tests are flakey right now? which test fails actually affect devs most? 
> did I break it? was that test already flakey? is that test still flakey? what 
> are our worst tests right now? is that test getting better or worse?
> We really need a way to see exactly what tests are the problem, not because 
> of OS or environmental issues, but more basic test quality issues. Which 
> tests are flakey and how flakey are they at any point in time.
> Reports:
> https://drive.google.com/drive/folders/0ByYyjsrbz7-qa2dOaU1UZDdRVzg?usp=sharing
> 01/24/2017 - 
> https://docs.google.com/spreadsheets/d/1JySta2j2s7A_p16wA1UO-l6c4GsUHBIb4FONS2EzW9k/edit?usp=sharing
> 02/01/2017 - 
> https://docs.google.com/spreadsheets/d/1FndoyHmihaOVL2o_Zns5alpNdAJlNsEwQVoJ4XDWj3c/edit?usp=sharing
> 02/08/2017 - 
> https://docs.google.com/spreadsheets/d/1N6RxH4Edd7ldRIaVfin0si-uSLGyowQi8-7mcux27S0/edit?usp=sharing
> 02/14/2017 - 
> https://docs.google.com/spreadsheets/d/1eZ9_ds_0XyqsKKp8xkmESrcMZRP85jTxSKkNwgtcUn0/edit?usp=sharing
> 02/17/2017 - 
> https://docs.google.com/spreadsheets/d/1LEPvXbsoHtKfIcZCJZ3_P6OHp7S5g2HP2OJgU6B2sAg/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-10989) Randomize PointFields and general cleanup in schema files that have completley unused Trie fieldTYpes

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man resolved SOLR-10989.
-
Resolution: Fixed

> Randomize PointFields and general cleanup in schema files that have 
> completley unused Trie fieldTYpes
> -
>
> Key: SOLR-10989
> URL: https://issues.apache.org/jira/browse/SOLR-10989
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Assignee: Hoss Man
> Fix For: master (7.0)
>
>
> while working on parent issue i noticed a schema filed that declared some 
> TrieField fieldTypes that were completely unused by and field or dynamicField 
> declarations.  so i write a quick little perl script to find more of these, 
> manually cleaned up all the ones that seemed like no brainers, and then added 
> the points randomization to replace the remaining Trie fieldtypes (that are 
> used) in these schemas...
> * core/src/test-files/solr/collection1/conf/bad-schema-dup-fieldType.xml
> * core/src/test-files/solr/collection1/conf/schema-binaryfield.xml
> * core/src/test-files/solr/collection1/conf/schema-copyfield-test.xml
> * core/src/test-files/solr/collection1/conf/schema-customfield.xml
> * core/src/test-files/solr/collection1/conf/schema-non-stored-docvalues.xml
> * core/src/test-files/solr/collection1/conf/schema-not-required-unique-key.xml
> * core/src/test-files/solr/collection1/conf/schema-replication1.xml
> * core/src/test-files/solr/collection1/conf/schema-replication2.xml
> * core/src/test-files/solr/collection1/conf/schema-required-fields.xml
> * core/src/test-files/solr/collection1/conf/schema-reversed.xml
> * core/src/test-files/solr/collection1/conf/schema-tokenizer-test.xml
> * core/src/test-files/solr/collection1/conf/schema-version-dv.xml
> * core/src/test-files/solr/collection1/conf/schema-version-indexed.xml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10989) Randomize PointFields and general cleanup in schema files that have completley unused Trie fieldTYpes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070466#comment-16070466
 ] 

ASF subversion and git services commented on SOLR-10989:


Commit 26d2ed7c4ddd2b8c714c2742dc9c42393bdb5d31 in lucene-solr's branch 
refs/heads/master from Chris Hostetter
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=26d2ed7 ]

SOLR-10989: Randomize PointFields and general cleanup in schema files where 
some Trie fields were unused


> Randomize PointFields and general cleanup in schema files that have 
> completley unused Trie fieldTYpes
> -
>
> Key: SOLR-10989
> URL: https://issues.apache.org/jira/browse/SOLR-10989
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Assignee: Hoss Man
> Fix For: master (7.0)
>
>
> while working on parent issue i noticed a schema filed that declared some 
> TrieField fieldTypes that were completely unused by and field or dynamicField 
> declarations.  so i write a quick little perl script to find more of these, 
> manually cleaned up all the ones that seemed like no brainers, and then added 
> the points randomization to replace the remaining Trie fieldtypes (that are 
> used) in these schemas...
> * core/src/test-files/solr/collection1/conf/bad-schema-dup-fieldType.xml
> * core/src/test-files/solr/collection1/conf/schema-binaryfield.xml
> * core/src/test-files/solr/collection1/conf/schema-copyfield-test.xml
> * core/src/test-files/solr/collection1/conf/schema-customfield.xml
> * core/src/test-files/solr/collection1/conf/schema-non-stored-docvalues.xml
> * core/src/test-files/solr/collection1/conf/schema-not-required-unique-key.xml
> * core/src/test-files/solr/collection1/conf/schema-replication1.xml
> * core/src/test-files/solr/collection1/conf/schema-replication2.xml
> * core/src/test-files/solr/collection1/conf/schema-required-fields.xml
> * core/src/test-files/solr/collection1/conf/schema-reversed.xml
> * core/src/test-files/solr/collection1/conf/schema-tokenizer-test.xml
> * core/src/test-files/solr/collection1/conf/schema-version-dv.xml
> * core/src/test-files/solr/collection1/conf/schema-version-indexed.xml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1412 - Still Failing!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1412/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 56202 lines...]
-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/jtidy_tmp
 [echo] Checking for broken links...
 [exec] 
 [exec] Crawl/parse...
 [exec] 
 [exec] Verify...
 [echo] Checking for malformed docs...
 [exec] 
 [exec] 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/docs/solr-core/overview-summary.html
 [exec]   missing description: org.apache.solr.common.cloud
 [exec] 
 [exec] Missing javadocs were found!

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:810: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:669: 
The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:685: 
The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2506:
 exec returned: 1

Total time: 79 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10984) Remove old redirects from web.xml

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070460#comment-16070460
 ] 

Varun Thacker commented on SOLR-10984:
--

The redirect rules comes from SOLR-3337 and we can remove them now.

I removed these mapping locally and the UI at http://localhost:8983/solr/#/ and 
http://localhost:8983/ loads up as expected but no more at 
http://localhost:8983/solr/index  ( earlier it redirected correctly )

{code}
  
RedirectOldAdminUI
/admin/
  
  
RedirectOldAdminUI
/admin
  
  
RedirectOldZookeeper
/zookeeper.jsp
  
  
RedirectOldZookeeper
/zookeeper
  
  
RedirectLogging
/logging
  
{code}

We can remove the RedirectServlet as well I guess

> Remove old redirects from web.xml
> -
>
> Key: SOLR-10984
> URL: https://issues.apache.org/jira/browse/SOLR-10984
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
> Fix For: master (7.0)
>
>
> As found by Varun in 
> https://lists.apache.org/thread.html/c31fa4809c9f5cc5b82eb86601b6be21592f04f623091c6506332015@%3Cdev.lucene.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Release planning for 7.0

[Christine Poerschke (BLOOMBERG/ LONDON)]

Hi Anshum,

SOLR-10962 is turning out bigger (and better) than it seemed at first. So 
please don't delay any branch cutting etc. because of it.

Thanks & Have a good weekend!

Christine

From: dev@lucene.apache.org At: 06/30/17 04:47:47
To: dev@lucene.apache.org
Subject: Re: Release planning for 7.0

Hi Anshum,
I'd like to have SOLR-10282 in for 7.0. It is a low impact new feature that 
helps admins to enable Kerberos more easily using the bin/solr script.
I should be able to have it dev-complete by end of Friday. Let me know if you 
have any objections.
Thanks,
Ishan

On Thu, Jun 29, 2017 at 1:00 AM, Anshum Gupta  wrote:

Hi Christine,

With my current progress, which is much slower that how I'd have liked it to 
be, I think there is still a day before the branches are cut. How far out do 
you think you are with this?

-Anshum


On Wed, Jun 28, 2017 at 9:59 AM Uwe Schindler  wrote:


Hi Anshum,
 
I have a häckidihickhäck workaround for the Hadoop Solr 9 issue. It is already 
committed to master and 6.x branch, so the issue is fixed: 
https://issues.apache.org/jira/browse/SOLR-10966
 
I lowered the Hadoop-Update (https://issues.apache.org/jira/browse/SOLR-10951) 
issue to “Major” level, so it is no longer blocker.
 
Nevertheless, we should fix the startup scripts for Java 9 in master before 
release of Solr 7, because currently the shell scripts fail (on certain 
platforms). And Java 9 is coming soon, so we should really have support because 
the speed improvements are a main reason to move to Java 9 with your Solr 
servers.

 
Uwe
 
-
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de
 

From: Anshum Gupta [mailto:ans...@anshumgupta.net] 
Sent: Sunday, June 25, 2017 7:52 PM


To: dev@lucene.apache.org
Subject: Re: Release planning for 7.0

 

Hi Uwe,

 

+1 on getting SOLR-10951 in before the release but I assume you weren't hinting 
at holding back the branch creation :).

 

I am not well versed with that stuff so it would certainly be optimal for 
someone else to look at that.

 

-Anshum

On Sun, Jun 25, 2017 at 9:58 AM Uwe Schindler  wrote:

Hi,
 
currently we have the following problem:
*The first Java 9 release candidate came out. This one now uses the final 
version format. The string returned by ${java.version} is now plain simple “9” 
– bummer for one single 3rd party library!
*This breaks one of the most basic Hadoop classes, so anything in Solr that 
refers somehow to Hadoop breaks. Of course this is HDFS - but also 
authentication! We should support Java 9, so we should really fix this ASAP!
 
From now on all tests running with Java 9 fail on Jenkins until we fix the 
following:
*Get an Update from Hadoop Guys (2.7.4), with just the stupid check removed 
(the completely useless version checking code snipped already makes its round 
through twitter): https://issues.apache.org/jira/browse/HADOOP-14586
*Or we update at least master/7.0 to latest Hadoop version, which has the bug 
already fixed. Unfortunately this does not work, as there is a bug in the 
Hadoop MiniDFSCluster that hangs on test shutdown. I have no idea how to fix. 
See https://issues.apache.org/jira/browse/SOLR-10951
 
I’d prefer to fix https://issues.apache.org/jira/browse/SOLR-10951 for master 
before release, so I set it as blocker. I am hoping for hely by Mark Miller. If 
the hadoop people have a simple bugfix release for the earlier version, we may 
also be able to fix branch_6x and branch_6_6 (but I disabled them on Jenkins 
anyways).
 
Uwe
 
-
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de
 

From: Anshum Gupta [mailto:ans...@anshumgupta.net] 
Sent: Saturday, June 24, 2017 10:52 PM


To: dev@lucene.apache.org
Subject: Re: Release planning for 7.0

 

I'll create the 7x, and 7.0 branches tomorrow.

 
Ishan, do you mean you would be able to close it by Tuesday? You would have to 
commit to both 7.0, and 7.x, in addition to master, but I think that should be 
ok.

 

We also have SOLR-10803 open at this moment and we'd need to come to a decision 
on that as well in order to move forward with 7.0.

 

P.S: If there are any objections to this plan, kindly let me know.

 

-Anshum
 

On Fri, Jun 23, 2017 at 5:03 AM Ishan Chattopadhyaya 
 wrote:

Hi Anshum,


> I will send out an email a day before cutting the branch, as well as once the 
> branch is in place.

I'm right now on travel, and unable to finish SOLR-10574 until Monday (possibly 
Tuesday).

Regards,

Ishan

 

On Tue, Jun 20, 2017 at 5:08 PM, Anshum Gupta  wrote:

From my understanding, there's not really a 'plan' but some intention to 
release a 6.7 at some time if enough people need it, right? In that case I 
wouldn't hold back anything for a 6x line release and cut the 7x, and 7.0 
branches around, but not before the coming weekend. I will send out an email a 
day before cutting the branch, as well as once the branch is in place.

 

If anyone has any objections to that, do 

[jira] [Commented] (SOLR-10990) QueryComponent.process breakup (for readability)

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070393#comment-16070393
 ] 

Christine Poerschke commented on SOLR-10990:


Created 
[jira/solr-10990|https://github.com/apache/lucene-solr/tree/jira/solr-10990] 
working branch with proposed change - quick link to the shorter method: 
https://github.com/apache/lucene-solr/blob/jira/solr-10990/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L300-L376

> QueryComponent.process breakup (for readability)
> 
>
> Key: SOLR-10990
> URL: https://issues.apache.org/jira/browse/SOLR-10990
> Project: Solr
>  Issue Type: Wish
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
>
> The method is currently very long i.e. 
> https://github.com/apache/lucene-solr/blob/e2521b2a8baabdaf43b92192588f51e042d21e97/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L300-L565
>  and breaking it up along logical lines (ids, grouped distributed first 
> phase, grouped distributed second phase, undistributed grouped, ungrouped) 
> would make it more readable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10990) QueryComponent.process breakup (for readability)

[Christine Poerschke (JIRA)]

Christine Poerschke created SOLR-10990:
--

 Summary: QueryComponent.process breakup (for readability)
 Key: SOLR-10990
 URL: https://issues.apache.org/jira/browse/SOLR-10990
 Project: Solr
  Issue Type: Wish
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Christine Poerschke
Assignee: Christine Poerschke
Priority: Minor


The method is currently very long i.e. 
https://github.com/apache/lucene-solr/blob/e2521b2a8baabdaf43b92192588f51e042d21e97/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L300-L565
 and breaking it up along logical lines (ids, grouped distributed first phase, 
grouped distributed second phase, undistributed grouped, ungrouped) would make 
it more readable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10989) Randomize PointFields and general cleanup in schema files that have completley unused Trie fieldTYpes

[Hoss Man (JIRA)]

Hoss Man created SOLR-10989:
---

 Summary: Randomize PointFields and general cleanup in schema files 
that have completley unused Trie fieldTYpes
 Key: SOLR-10989
 URL: https://issues.apache.org/jira/browse/SOLR-10989
 Project: Solr
  Issue Type: Sub-task
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Hoss Man
Assignee: Hoss Man


while working on parent issue i noticed a schema filed that declared some 
TrieField fieldTypes that were completely unused by and field or dynamicField 
declarations.  so i write a quick little perl script to find more of these, 
manually cleaned up all the ones that seemed like no brainers, and then added 
the points randomization to replace the remaining Trie fieldtypes (that are 
used) in these schemas...


* core/src/test-files/solr/collection1/conf/bad-schema-dup-fieldType.xml
* core/src/test-files/solr/collection1/conf/schema-binaryfield.xml
* core/src/test-files/solr/collection1/conf/schema-copyfield-test.xml
* core/src/test-files/solr/collection1/conf/schema-customfield.xml
* core/src/test-files/solr/collection1/conf/schema-non-stored-docvalues.xml
* core/src/test-files/solr/collection1/conf/schema-not-required-unique-key.xml
* core/src/test-files/solr/collection1/conf/schema-replication1.xml
* core/src/test-files/solr/collection1/conf/schema-replication2.xml
* core/src/test-files/solr/collection1/conf/schema-required-fields.xml
* core/src/test-files/solr/collection1/conf/schema-reversed.xml
* core/src/test-files/solr/collection1/conf/schema-tokenizer-test.xml
* core/src/test-files/solr/collection1/conf/schema-version-dv.xml
* core/src/test-files/solr/collection1/conf/schema-version-indexed.xml




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10988) SolrJ: Possible Stackoverflow in DocumentObjectBinder

[Danny Trunk (JIRA)]

Danny Trunk created SOLR-10988:
--

 Summary: SolrJ: Possible Stackoverflow in DocumentObjectBinder
 Key: SOLR-10988
 URL: https://issues.apache.org/jira/browse/SOLR-10988
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: SolrJ
Affects Versions: 6.6
Reporter: Danny Trunk
Priority: Minor


If a bean contains a list of itself flagged as child documents a Stackoverflow 
occurs. See external issue URL for detailed information.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10987) Solr Cloud (5 nodes and 70 million documents) going down, when the overseer node becomes unreachable. Issue Started Recently

[RAHAT BHALLA (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

RAHAT BHALLA updated SOLR-10987:

Summary: Solr Cloud (5 nodes and 70 million documents) going down, when the 
overseer node becomes unreachable. Issue Started Recently  (was: Solr Cloud (5 
nodes and 70 million documents) going down, when the overseer node becomes 
unreachable. Started Recently)

> Solr Cloud (5 nodes and 70 million documents) going down, when the overseer 
> node becomes unreachable. Issue Started Recently
> 
>
> Key: SOLR-10987
> URL: https://issues.apache.org/jira/browse/SOLR-10987
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 6.1
> Environment: *The following is the usage on each of the Solr Nodes:*
> Tasks: 254 total,   1 running, 252 sleeping,   0 stopped,   1 zombie
> %Cpu(s):  0.4 us,  0.3 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 
> st
> KiB Mem : 20392276 total,  4169296 free,  2917012 used, 13305968 buff/cache
> KiB Swap:  5111804 total,  5111636 free,  168 used. 16058184 avail Mem
>   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
> 21250 solr  20   0 23.599g 1.184g 228440 S   2.0  6.1  59:55.91 java
> *Solr is running on 5 machines with similar configuration:*
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):4
> On-line CPU(s) list:   0-3
> Thread(s) per core:1
> Core(s) per socket:2
> Socket(s): 2
> NUMA node(s):  1
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 62
> Model name:Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
> Stepping:  4
> CPU MHz:   2799.033
> BogoMIPS:  5600.00
> Hypervisor vendor: VMware
> Virtualization type:   full
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0-3
>Reporter: RAHAT BHALLA
>  Labels: assistance, critical, customer, impacting, issue, need, 
> production
>
> We host a Solr Cloud of 5 Nodes for Solr Instances and 3 Zookeeper nodes to 
> maintain the cloud. We have over 70 million docs spread across 13 collections 
> with 40K more documents being added every day almost near time within spans 
> of 5 to 6 minutes.
> The System was working as expected and as required for th elast 7 months 
> until suddenly we saw the following exception and all of our instances went 
> offline. We restarted the instances and the cloud ran smoothly for three days 
> before it came crashing down again.
> *Exception It gives before it goes down is as follows:*
> 3542285 ERROR 
> (OverseerCollectionConfigSetProcessor-98221003671470081-prod-solr-node01:9080_solr-n_000106)
>  [   ] o.a.s.c.OverseerTaskProcessor
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for /overseer_elect/leader
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
> at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> at 
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
> at 
> org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
> at 
> org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
> at java.lang.Thread.run(Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10276) Update ZK leader election so that leader notices if its leadership is revoked

[Scott Blum (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070340#comment-16070340
 ] 

Scott Blum commented on SOLR-10276:
---

Please add a patch!  This is super useful operationally.

> Update ZK leader election so that leader notices if its leadership is revoked
> -
>
> Key: SOLR-10276
> URL: https://issues.apache.org/jira/browse/SOLR-10276
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 5.5.3
>Reporter: Joshua Humphries
>Assignee: Scott Blum
>Priority: Minor
>  Labels: leader, zookeeper
>
> When we have an issue with a solr node, it would be nice to revoke its 
> leadership of one or more shard or to revoke its role as overseer without 
> actually restarting the node. (Restarting the node tends to spam the overseer 
> queue since we have a very large number of cores per node.)
> Operationally, it would be nice if one could just delete the leader's 
> election node (e.g. its ephemeral sequential node that indicates it as 
> current leader) and to have it notice the change and stop behaving as leader.
> Currently, once a node becomes leader, it isn't watching ZK for any changes 
> that could revoke its leadership. I am proposing that, upon being elected 
> leader, it use a ZK watch to monitor its own election node. If its own 
> election node is deleted, it then relinquishes leadership (e.g. calls 
> ElectionContext#cancelElection() and then re-joins the election).
> I have a patch with tests that I can contribute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-10276) Update ZK leader election so that leader notices if its leadership is revoked

[Scott Blum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Blum reassigned SOLR-10276:
-

Assignee: Scott Blum

> Update ZK leader election so that leader notices if its leadership is revoked
> -
>
> Key: SOLR-10276
> URL: https://issues.apache.org/jira/browse/SOLR-10276
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 5.5.3
>Reporter: Joshua Humphries
>Assignee: Scott Blum
>Priority: Minor
>  Labels: leader, zookeeper
>
> When we have an issue with a solr node, it would be nice to revoke its 
> leadership of one or more shard or to revoke its role as overseer without 
> actually restarting the node. (Restarting the node tends to spam the overseer 
> queue since we have a very large number of cores per node.)
> Operationally, it would be nice if one could just delete the leader's 
> election node (e.g. its ephemeral sequential node that indicates it as 
> current leader) and to have it notice the change and stop behaving as leader.
> Currently, once a node becomes leader, it isn't watching ZK for any changes 
> that could revoke its leadership. I am proposing that, upon being elected 
> leader, it use a ZK watch to monitor its own election node. If its own 
> election node is deleted, it then relinquishes leadership (e.g. calls 
> ElectionContext#cancelElection() and then re-joins the election).
> I have a patch with tests that I can contribute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-5822) Create a markdown formatted README file for Lucene/Solr

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob resolved LUCENE-5822.
---
   Resolution: Fixed
 Assignee: Mike Drob
Fix Version/s: master (7.0)
Lucene Fields:   (was: New)

Thanks, Jason!

> Create a markdown formatted README file for Lucene/Solr
> ---
>
> Key: LUCENE-5822
> URL: https://issues.apache.org/jira/browse/LUCENE-5822
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Shalin Shekhar Mangar
>Assignee: Mike Drob
> Fix For: master (7.0)
>
> Attachments: LUCENE-5822.patch, LUCENE-5822.patch
>
>
> We have a minimal plain text readme file right now. Github is very popular 
> these days and we are even accepting pull requests from there. I think we 
> should add a proper markdown formatted README file which not only talks about 
> the features of Lucene/Solr but also include a short tutorial on both Lucene 
> and Solr.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5822) Create a markdown formatted README file for Lucene/Solr

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070330#comment-16070330
 ] 

ASF subversion and git services commented on LUCENE-5822:
-

Commit 2e1c5063a5cdd76098730265dc551c43ad8526ae in lucene-solr's branch 
refs/heads/master from [~gerlowskija]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2e1c506 ]

LUCENE-5822 Add markdown-compatible README.md

Signed-off-by: Mike Drob 


> Create a markdown formatted README file for Lucene/Solr
> ---
>
> Key: LUCENE-5822
> URL: https://issues.apache.org/jira/browse/LUCENE-5822
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Shalin Shekhar Mangar
> Attachments: LUCENE-5822.patch, LUCENE-5822.patch
>
>
> We have a minimal plain text readme file right now. Github is very popular 
> these days and we are even accepting pull requests from there. I think we 
> should add a proper markdown formatted README file which not only talks about 
> the features of Lucene/Solr but also include a short tutorial on both Lucene 
> and Solr.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10987) Solr Cloud (5 nodes and 70 million documents) going down, when the overseer node becomes unreachable. Started Recently

[RAHAT BHALLA (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

RAHAT BHALLA updated SOLR-10987:

Component/s: SolrCloud

> Solr Cloud (5 nodes and 70 million documents) going down, when the overseer 
> node becomes unreachable. Started Recently
> --
>
> Key: SOLR-10987
> URL: https://issues.apache.org/jira/browse/SOLR-10987
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 6.1
> Environment: *The following is the usage on each of the Solr Nodes:*
> Tasks: 254 total,   1 running, 252 sleeping,   0 stopped,   1 zombie
> %Cpu(s):  0.4 us,  0.3 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 
> st
> KiB Mem : 20392276 total,  4169296 free,  2917012 used, 13305968 buff/cache
> KiB Swap:  5111804 total,  5111636 free,  168 used. 16058184 avail Mem
>   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
> 21250 solr  20   0 23.599g 1.184g 228440 S   2.0  6.1  59:55.91 java
> *Solr is running on 5 machines with similar configuration:*
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):4
> On-line CPU(s) list:   0-3
> Thread(s) per core:1
> Core(s) per socket:2
> Socket(s): 2
> NUMA node(s):  1
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 62
> Model name:Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
> Stepping:  4
> CPU MHz:   2799.033
> BogoMIPS:  5600.00
> Hypervisor vendor: VMware
> Virtualization type:   full
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0-3
>Reporter: RAHAT BHALLA
>  Labels: assistance, critical, customer, impacting, issue, need, 
> production
>
> We host a Solr Cloud of 5 Nodes for Solr Instances and 3 Zookeeper nodes to 
> maintain the cloud. We have over 70 million docs spread across 13 collections 
> with 40K more documents being added every day almost near time within spans 
> of 5 to 6 minutes.
> The System was working as expected and as required for th elast 7 months 
> until suddenly we saw the following exception and all of our instances went 
> offline. We restarted the instances and the cloud ran smoothly for three days 
> before it came crashing down again.
> *Exception It gives before it goes down is as follows:*
> 3542285 ERROR 
> (OverseerCollectionConfigSetProcessor-98221003671470081-prod-solr-node01:9080_solr-n_000106)
>  [   ] o.a.s.c.OverseerTaskProcessor
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for /overseer_elect/leader
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
> at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> at 
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
> at 
> org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
> at 
> org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
> at java.lang.Thread.run(Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10987) Solr Cloud (5 nodes and 70 million documents) going down, when the overseer node becomes unreachable. Started Recently

[RAHAT BHALLA (JIRA)]

RAHAT BHALLA created SOLR-10987:
---

 Summary: Solr Cloud (5 nodes and 70 million documents) going down, 
when the overseer node becomes unreachable. Started Recently
 Key: SOLR-10987
 URL: https://issues.apache.org/jira/browse/SOLR-10987
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 6.1
 Environment: *The following is the usage on each of the Solr Nodes:*

Tasks: 254 total,   1 running, 252 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.4 us,  0.3 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 20392276 total,  4169296 free,  2917012 used, 13305968 buff/cache
KiB Swap:  5111804 total,  5111636 free,  168 used. 16058184 avail Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
21250 solr  20   0 23.599g 1.184g 228440 S   2.0  6.1  59:55.91 java



*Solr is running on 5 machines with similar configuration:*

Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):4
On-line CPU(s) list:   0-3
Thread(s) per core:1
Core(s) per socket:2
Socket(s): 2
NUMA node(s):  1
Vendor ID: GenuineIntel
CPU family:6
Model: 62
Model name:Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Stepping:  4
CPU MHz:   2799.033
BogoMIPS:  5600.00
Hypervisor vendor: VMware
Virtualization type:   full
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  25600K
NUMA node0 CPU(s): 0-3



Reporter: RAHAT BHALLA


We host a Solr Cloud of 5 Nodes for Solr Instances and 3 Zookeeper nodes to 
maintain the cloud. We have over 70 million docs spread across 13 collections 
with 40K more documents being added every day almost near time within spans of 
5 to 6 minutes.

The System was working as expected and as required for th elast 7 months until 
suddenly we saw the following exception and all of our instances went offline. 
We restarted the instances and the cloud ran smoothly for three days before it 
came crashing down again.

*Exception It gives before it goes down is as follows:*

3542285 ERROR 
(OverseerCollectionConfigSetProcessor-98221003671470081-prod-solr-node01:9080_solr-n_000106)
 [   ] o.a.s.c.OverseerTaskProcessor
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = 
ConnectionLoss for /overseer_elect/leader
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
at 
org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
at 
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
at java.lang.Thread.run(Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10986) TestScoreJoinQPScore.testDeleteByScoreJoinQuery() failure: mismatch: '0'!='1' @ response/numFound

[Steve Rowe (JIRA)]

Steve Rowe created SOLR-10986:
-

 Summary: TestScoreJoinQPScore.testDeleteByScoreJoinQuery() 
failure: mismatch: '0'!='1' @ response/numFound
 Key: SOLR-10986
 URL: https://issues.apache.org/jira/browse/SOLR-10986
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Steve Rowe


Reproduces for me on branch_6x but not on master, from 
[https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/3861/] - {{git bisect}} 
blames commit {{c215c78}} on SOLR-9217:

{noformat}
Checking out Revision 9947a811e83cc0f848f9ddaa37a4137f19efff1a 
(refs/remotes/origin/branch_6x)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestScoreJoinQPScore -Dtests.method=testDeleteByScoreJoinQuery 
-Dtests.seed=6DE98178CA5DE220 -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=el-GR -Dtests.timezone=Asia/Vientiane -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.02s J1 | TestScoreJoinQPScore.testDeleteByScoreJoinQuery 
<<<
   [junit4]> Throwable #1: java.lang.RuntimeException: mismatch: '0'!='1' @ 
response/numFound
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([6DE98178CA5DE220:7A8B1D8F401EA807]:0)
   [junit4]>at 
org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:989)
   [junit4]>at 
org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:936)
   [junit4]>at 
org.apache.solr.search.join.TestScoreJoinQPScore.testDeleteByScoreJoinQuery(TestScoreJoinQPScore.java:125)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{t_description=BlockTreeOrds(blocksize=128), 
title_stemmed=PostingsFormat(name=Memory doPackFST= false), 
price_s=BlockTreeOrds(blocksize=128), name=BlockTreeOrds(blocksize=128), 
id=BlockTreeOrds(blocksize=128), 
text=PostingsFormat(name=LuceneVarGapFixedInterval), 
movieId_s=BlockTreeOrds(blocksize=128), title=PostingsFormat(name=Memory 
doPackFST= false), title_lettertok=BlockTreeOrds(blocksize=128), 
productId_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))},
 docValues:{}, maxPointsInLeafNode=166, maxMBSortInHeap=7.4808509338680995, 
sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=el-GR, 
timezone=Asia/Vientiane
   [junit4]   2> NOTE: Linux 4.10.0-21-generic i386/Oracle Corporation 
1.8.0_131 (32-bit)/cpus=8,threads=1,free=159538432,total=510918656
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10981) Allow update to load gzip files

[Andrew Lundgren (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lundgren updated SOLR-10981:
---
Attachment: SOLR-10981.patch

Added tests around checking content type. Fixed bug in checking content type of 
FileStream for gzipped streams.  Added comments in test of FileStream vs 
file:// URL Stream.

> Allow update to load gzip files 
> 
>
> Key: SOLR-10981
> URL: https://issues.apache.org/jira/browse/SOLR-10981
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6
>Reporter: Andrew Lundgren
>  Labels: patch
> Fix For: 4.10.4, 6.6, master (7.0)
>
> Attachments: SOLR-10981.patch, SOLR-10981.patch
>
>
> We currently import large CSV files.  We store them in gzip files as they 
> compress at around 80%.
> To import them we must gunzip them and then import them.  After that we no 
> longer need the decompressed files.
> This patch allows directly opening either URL, or local files that are 
> gzipped.
> For URLs, to determine if the file is gzipped, it will check the content 
> encoding=="gzip" or if the file ends in ".gz"
> For files, if the file ends in ".gz" then it will assume the file is gzipped.
> I have tested the patch with 4.10.4, 6.6.0 and master from git.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10981) Allow update to load gzip files

[Andrew Lundgren (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070259#comment-16070259
 ] 

Andrew Lundgren commented on SOLR-10981:


As this doesn't use the content-type header, it uses the content-encoding 
header, it does not interfere with the existing content-type header usage. 

In the patched ContentStreamBase, line 84 the content type is taken from the 
connection.  As this is not changed by the gzip contentEncoding header on line 
89; code using the content stream is unaffected.  If the contentEncoding is not 
set, then the code will also detect if the file ends with ".gz".  This could be 
dropped, though it seemed a reasonable usage.

In the patched ContentStreamBase, lines 117, 121 the content type of a 
FileStream is determined by the first character found in the stream.  As the 
stream is already opened and the gunzip stream applied over the input stream, 
the code that determines the content type is unaffected.  The FileStream will 
work with any existing format that is gzipped as it determines the content type 
based on the first character of the decompressed stream.  (Attached is a new 
patch that causes this method to use the getStream method on 117 rather than 
open the file itself applying the gzip layer)

I agree that using generic {{Content-Type: application/gzip}} would lead to 
confusion.  To me, the gzip layer is the encoding of the content, not the type 
itself.  By using the encoding type you are able to handle the gzip at a lower 
layer, and keep all of your content-type support untouched.

The current handling of a FileStream and a file:// URL are inconsistent, as the 
FileStream tries to guess the content type based on the first character.  The 
file:// URL uses mime-types to determine the content.   They seemingly should 
be consistent, though I did not try to make them consistent, as the FileStream 
implementation point's out, it's implementation is buggy.


> Allow update to load gzip files 
> 
>
> Key: SOLR-10981
> URL: https://issues.apache.org/jira/browse/SOLR-10981
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6
>Reporter: Andrew Lundgren
>  Labels: patch
> Fix For: 4.10.4, 6.6, master (7.0)
>
> Attachments: SOLR-10981.patch
>
>
> We currently import large CSV files.  We store them in gzip files as they 
> compress at around 80%.
> To import them we must gunzip them and then import them.  After that we no 
> longer need the decompressed files.
> This patch allows directly opening either URL, or local files that are 
> gzipped.
> For URLs, to determine if the file is gzipped, it will check the content 
> encoding=="gzip" or if the file ends in ".gz"
> For files, if the file ends in ".gz" then it will assume the file is gzipped.
> I have tested the patch with 4.10.4, 6.6.0 and master from git.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10827) factor out abstract FilteringSolrMetricReporter

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070233#comment-16070233
 ] 

Christine Poerschke commented on SOLR-10827:


ant precommit (sans SOLR-10954) passes, solr core tests pass.

If there are no objections or concerns I will commit this patch early next week.

> factor out abstract FilteringSolrMetricReporter
> ---
>
> Key: SOLR-10827
> URL: https://issues.apache.org/jira/browse/SOLR-10827
> Project: Solr
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-10827.patch
>
>
> Currently multiple SolrMetricReporter classes have their own local filter 
> settings, a common setting somewhere will reduce code duplication for 
> existing, future and custom reporters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4118 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4118/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForHashRouter

Error Message:
Collection not found: routeFieldColl

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: routeFieldColl
at 
__randomizedtesting.SeedInfo.seed([9B9A39DAE7A4D0D5:33ACA70778C53B8F]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1120)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:803)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at 
org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
at 
org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForHashRouter(CustomCollectionTest.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.ca

[jira] [Commented] (SOLR-10778) Ant precommit task WARNINGS about unclosed resources

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070216#comment-16070216
 ] 

David Smiley commented on SOLR-10778:
-

That is useful Christine.  But if it's long term then I think a message to the 
dev list would be better than JIRA.  Hopefully this sort of thing is short term 
so that we can get to a point where we can fail precommit if it occurs?

> Ant precommit task WARNINGS about unclosed resources
> 
>
> Key: SOLR-10778
> URL: https://issues.apache.org/jira/browse/SOLR-10778
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: clients - java
>Affects Versions: 4.6
>Reporter: Andrew Musselman
>Priority: Minor
> Attachments: dated-warnings, dated-warnings.log, notclosed.txt
>
>
> During precommit we are seeing lots of warnings about resources that aren't 
> being closed, which could pose problems based on chat amongst the team. Log 
> snippet for example:
> [mkdir] Created dir: 
> /var/folders/5p/6b46rm_94dzc5m8d4v56tds4gp/T/ecj1165341501
>  [ecj-lint] Compiling 419 source files to 
> /var/folders/5p/6b46rm_94dzc5m8d4v56tds4gp/T/ecj1165341501
>  [ecj-lint] --
>  [ecj-lint] 1. WARNING in 
> /path/to/lucene-solr/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttpSolrClient.java
>  (at line 920)
>  [ecj-lint] new LBHttpSolrClient(httpSolrClientBuilder, httpClient, 
> solrServerUrls) :
>  [ecj-lint] 
> ^^^
>  [ecj-lint] Resource leak: '' is never closed
>  [ecj-lint] --
>  [ecj-lint] --
>  [ecj-lint] 2. WARNING in 
> /path/to/lucene-solr/solr/solrj/src/java/org/apache/solr/client/solrj/impl/StreamingBinaryResponseParser.java
>  (at line 49)
>  [ecj-lint] JavaBinCodec codec = new JavaBinCodec() {
>  [ecj-lint]  ^
>  [ecj-lint] Resource leak: 'codec' is never closed
>  [ecj-lint] --
>  [ecj-lint] --
>  [ecj-lint] 3. WARNING in 
> /path/to/lucene-solr/solr/solrj/src/java/org/apache/solr/client/solrj/request/JavaBinUpdateRequestCodec.java
>  (at line 90)
>  [ecj-lint] JavaBinCodec codec = new JavaBinCodec();
>  [ecj-lint]  ^
>  [ecj-lint] Resource leak: 'codec' is never closed
>  [ecj-lint] --
>  [ecj-lint] 4. WARNING in 
> /path/to/lucene-solr/solr/solrj/src/java/org/apache/solr/client/solrj/request/JavaBinUpdateRequestCodec.java
>  (at line 113)
>  [ecj-lint] JavaBinCodec codec = new JavaBinCodec() {
>  [ecj-lint]  ^
>  [ecj-lint] Resource leak: 'codec' is never closed



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10962) replicationHandler's reserveCommitDuration configurable in SolrCloud mode

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070214#comment-16070214
 ] 

Christine Poerschke commented on SOLR-10962:


Thanks for suggesting the 
{{core.solrConfig.luceneMatchVersion.onOrAfter(Version.LUCENE_7_0_0)}} clause, 
that totally makes sense.

bq. ... a well-known name in Config API to update these settings in the same 
way that we can update autoCommit.maxTime etc. ...

Good idea, let's explore that.

The replication handler already has an existing top-level attribute 
_maxNumberOfBackups_ - presumably that would transition over to Config API at 
the same time as the (currently not top-level) _reserveCommitDuration_ 
attribute.

> replicationHandler's reserveCommitDuration configurable in SolrCloud mode
> -
>
> Key: SOLR-10962
> URL: https://issues.apache.org/jira/browse/SOLR-10962
> Project: Solr
>  Issue Type: New Feature
>  Components: replication (java)
>Reporter: Ramsey Haddad
>Priority: Minor
> Attachments: SOLR-10962.patch, SOLR-10962.patch, SOLR-10962.patch
>
>
> With SolrCloud mode, when doing replication via IndexFetcher, we occasionally 
> see the Fetch fail and then get restarted from scratch in cases where an 
> Index file is deleted after fetch manifest is computed and before the fetch 
> actually transfers the file. The risk of this happening can be reduced with a 
> higher value of reserveCommitDuration. However, the current configuration 
> only allows this value to be adjusted for "master" mode. This change allows 
> the value to also be changed when using "SolrCloud" mode.
> https://lucene.apache.org/solr/guide/6_6/index-replication.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Linux (32bit/jdk1.8.0_131) - Build # 3861 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/3861/
Java: 32bit/jdk1.8.0_131 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  
org.apache.solr.search.join.TestScoreJoinQPScore.testDeleteByScoreJoinQuery

Error Message:
mismatch: '0'!='1' @ response/numFound

Stack Trace:
java.lang.RuntimeException: mismatch: '0'!='1' @ response/numFound
at 
__randomizedtesting.SeedInfo.seed([6DE98178CA5DE220:7A8B1D8F401EA807]:0)
at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:989)
at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:936)
at 
org.apache.solr.search.join.TestScoreJoinQPScore.testDeleteByScoreJoinQuery(TestScoreJoinQPScore.java:125)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13222 lines...]
   [junit4] Suite: org.apache.solr.search.join.TestScoreJoinQPScore
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr

[jira] [Commented] (SOLR-10415) Within solr-core, debug/trace level logging should use parameterized log messages

[Michael Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070195#comment-16070195
 ] 

Michael Braun commented on SOLR-10415:
--

Breaking this up into sub-tickets with patches to make this easier - see 
[SOLR-10985]

> Within solr-core, debug/trace level logging should use parameterized log 
> messages
> -
>
> Key: SOLR-10415
> URL: https://issues.apache.org/jira/browse/SOLR-10415
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Michael Braun
>Priority: Trivial
>
> Noticed in several samplings of an active Solr that several debug statements 
> were taking decently measurable time because of the time of the .toString 
> even when the log.debug() statement would not output because it was 
> effectively INFO or higher. Using parameterized logging statements, ie 
> 'log.debug("Blah {}", o)' will avoid incurring that cost.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10985) Clean up debug logging statements in solr-core's search package

[Michael Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Braun updated SOLR-10985:
-
Flags: Patch

> Clean up debug logging statements in solr-core's search package
> ---
>
> Key: SOLR-10985
> URL: https://issues.apache.org/jira/browse/SOLR-10985
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Michael Braun
>Priority: Trivial
> Attachments: SOLR-10985.patch
>
>
> Clean up debug-level statements in solr-core's search package that are not 
> using parameterized logging and/or are manually calling toString() on method 
> arguments.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10985) Clean up debug logging statements in solr-core's search package

[Michael Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Braun updated SOLR-10985:
-
Attachment: SOLR-10985.patch

> Clean up debug logging statements in solr-core's search package
> ---
>
> Key: SOLR-10985
> URL: https://issues.apache.org/jira/browse/SOLR-10985
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Michael Braun
>Priority: Trivial
> Attachments: SOLR-10985.patch
>
>
> Clean up debug-level statements in solr-core's search package that are not 
> using parameterized logging and/or are manually calling toString() on method 
> arguments.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10985) Clean up debug logging statements in solr-core's search package

[Michael Braun (JIRA)]

Michael Braun created SOLR-10985:


 Summary: Clean up debug logging statements in solr-core's search 
package
 Key: SOLR-10985
 URL: https://issues.apache.org/jira/browse/SOLR-10985
 Project: Solr
  Issue Type: Sub-task
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Michael Braun
Priority: Trivial


Clean up debug-level statements in solr-core's search package that are not 
using parameterized logging and/or are manually calling toString() on method 
arguments.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10032) Create report to assess Solr test quality at a commit point.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070191#comment-16070191
 ] 

Mark Miller commented on SOLR-10032:


I actually bailed on spending more time with report generation for a bit. The 
process was still a bit too cumbersome - it really needs to be one command to 
kick off and then everything else is automatic so that it can be run regularly 
with ease. I finally came up with a plan to automate from test running to table 
generation and publishing, so I have spent the time instead finishing up full 
automation. I still have a bit to do, but I'll be done soon and then will run a 
report on a regular schedule.

I've also spent a little time planning out using Docker for parallel test runs 
instead of the built in project support. This will enable the possibility of 
supporting other projects and there are some nice benefits in truly isolating 
parallel test runs.

> Create report to assess Solr test quality at a commit point.
> 
>
> Key: SOLR-10032
> URL: https://issues.apache.org/jira/browse/SOLR-10032
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Mark Miller
>Assignee: Mark Miller
> Attachments: Lucene-Solr Master Test Beast Results 
> 01-24-2017-9899cbd031dc3fc37a384b1f9e2b379e90a9a3a6 Level Medium- Running 30 
> iterations, 12 at a time .pdf, Lucene-Solr Master Test Beasults 
> 02-01-2017-bbc455de195c83d9f807980b510fa46018f33b1b Level Medium- Running 30 
> iterations, 10 at a time.pdf, Lucene-Solr Master Test Beasults 
> 02-08-2017-6696eafaae18948c2891ce758c7a2ec09873dab8 Level Medium+- Running 30 
> iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults 
> 02-14-2017- Level Medium+-a1f114f70f3800292c25be08213edf39b3e37f6a Running 30 
> iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults 
> 02%2F17%2F2017-19c8ec2bf1882bed1bb34d0b55198d03f2018838 Level Hard Running 
> 100 iterations, 12 at a time, 8 cores.pdf
>
>
> We have many Jenkins instances blasting tests, some official, some policeman, 
> I and others have or had their own, and the email trail proves the power of 
> the Jenkins cluster to find test fails.
> However, I still have a very hard time with some basic questions:
> what tests are flakey right now? which test fails actually affect devs most? 
> did I break it? was that test already flakey? is that test still flakey? what 
> are our worst tests right now? is that test getting better or worse?
> We really need a way to see exactly what tests are the problem, not because 
> of OS or environmental issues, but more basic test quality issues. Which 
> tests are flakey and how flakey are they at any point in time.
> Reports:
> https://drive.google.com/drive/folders/0ByYyjsrbz7-qa2dOaU1UZDdRVzg?usp=sharing
> 01/24/2017 - 
> https://docs.google.com/spreadsheets/d/1JySta2j2s7A_p16wA1UO-l6c4GsUHBIb4FONS2EzW9k/edit?usp=sharing
> 02/01/2017 - 
> https://docs.google.com/spreadsheets/d/1FndoyHmihaOVL2o_Zns5alpNdAJlNsEwQVoJ4XDWj3c/edit?usp=sharing
> 02/08/2017 - 
> https://docs.google.com/spreadsheets/d/1N6RxH4Edd7ldRIaVfin0si-uSLGyowQi8-7mcux27S0/edit?usp=sharing
> 02/14/2017 - 
> https://docs.google.com/spreadsheets/d/1eZ9_ds_0XyqsKKp8xkmESrcMZRP85jTxSKkNwgtcUn0/edit?usp=sharing
> 02/17/2017 - 
> https://docs.google.com/spreadsheets/d/1LEPvXbsoHtKfIcZCJZ3_P6OHp7S5g2HP2OJgU6B2sAg/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10778) Ant precommit task WARNINGS about unclosed resources

[Christine Poerschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christine Poerschke updated SOLR-10778:
---
Attachment: dated-warnings
dated-warnings.log

We obviously cannot take care of all the warnings overnight but we can 
incrementally take care of existing warnings and watch out for new warnings 
appearing. It's difficult though to spot new warnings amongst the copious 'ant 
precommit' output.

With all of that in mind I've created a little _dated-warnings_ shell script 
that correlates warnings with git commit history. Attaching both the script and 
its current output, for convenience also here copied/pasted from the script the 
warnings for June 2017 (so far):

* 2017-06-21 
http://www.github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/ReadersAndUpdates.java#L843
* 2017-06-21 
http://www.github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FrozenBufferedUpdates.java#L186
* 2017-06-21 
http://www.github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FrozenBufferedUpdates.java#L144
* 2017-06-20 
http://www.github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/core/DirectoryFactoryTest.java#L53
* 2017-06-16 
http://www.github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/Utils.java#L110
* 2017-06-16 
http://www.github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/CommandOperation.java#L248
* 2017-06-16 
http://www.github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/util/TestUtils.java#L186

If folks think this is useful then I'd be happy to periodically re-run the 
script and share results here.


> Ant precommit task WARNINGS about unclosed resources
> 
>
> Key: SOLR-10778
> URL: https://issues.apache.org/jira/browse/SOLR-10778
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: clients - java
>Affects Versions: 4.6
>Reporter: Andrew Musselman
>Priority: Minor
> Attachments: dated-warnings, dated-warnings.log, notclosed.txt
>
>
> During precommit we are seeing lots of warnings about resources that aren't 
> being closed, which could pose problems based on chat amongst the team. Log 
> snippet for example:
> [mkdir] Created dir: 
> /var/folders/5p/6b46rm_94dzc5m8d4v56tds4gp/T/ecj1165341501
>  [ecj-lint] Compiling 419 source files to 
> /var/folders/5p/6b46rm_94dzc5m8d4v56tds4gp/T/ecj1165341501
>  [ecj-lint] --
>  [ecj-lint] 1. WARNING in 
> /path/to/lucene-solr/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttpSolrClient.java
>  (at line 920)
>  [ecj-lint] new LBHttpSolrClient(httpSolrClientBuilder, httpClient, 
> solrServerUrls) :
>  [ecj-lint] 
> ^^^
>  [ecj-lint] Resource leak: '' is never closed
>  [ecj-lint] --
>  [ecj-lint] --
>  [ecj-lint] 2. WARNING in 
> /path/to/lucene-solr/solr/solrj/src/java/org/apache/solr/client/solrj/impl/StreamingBinaryResponseParser.java
>  (at line 49)
>  [ecj-lint] JavaBinCodec codec = new JavaBinCodec() {
>  [ecj-lint]  ^
>  [ecj-lint] Resource leak: 'codec' is never closed
>  [ecj-lint] --
>  [ecj-lint] --
>  [ecj-lint] 3. WARNING in 
> /path/to/lucene-solr/solr/solrj/src/java/org/apache/solr/client/solrj/request/JavaBinUpdateRequestCodec.java
>  (at line 90)
>  [ecj-lint] JavaBinCodec codec = new JavaBinCodec();
>  [ecj-lint]  ^
>  [ecj-lint] Resource leak: 'codec' is never closed
>  [ecj-lint] --
>  [ecj-lint] 4. WARNING in 
> /path/to/lucene-solr/solr/solrj/src/java/org/apache/solr/client/solrj/request/JavaBinUpdateRequestCodec.java
>  (at line 113)
>  [ecj-lint] JavaBinCodec codec = new JavaBinCodec() {
>  [ecj-lint]  ^
>  [ecj-lint] Resource leak: 'codec' is never closed



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7892) LatLonDocValuesField methods should be clearly marked as slow

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070129#comment-16070129
 ] 

Michael McCandless commented on LUCENE-7892:


+1

> LatLonDocValuesField methods should be clearly marked as slow
> -
>
> Key: LUCENE-7892
> URL: https://issues.apache.org/jira/browse/LUCENE-7892
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
>
> It is very trappy that LatLonDocValuesField has stuff like 
> newBoxQuery/newDistanceQuery.
> Users bring this up on the user list and are confused as to why the resulting 
> queries are slow.
> Here, we hurt the typical use case, to try to slightly speed up an esoteric 
> one (sparse stuff). Its a terrible tradeoff for the API.
> If we truly must have such slow methods in the public API, then they should 
> have {{slow}} in their name.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7871) false positive match BlockJoinSelector[SortedDV] when child value is absent

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070127#comment-16070127
 ] 

Mikhail Khludnev commented on LUCENE-7871:
--

..going, going

> false positive match BlockJoinSelector[SortedDV] when child value is absent 
> 
>
> Key: LUCENE-7871
> URL: https://issues.apache.org/jira/browse/LUCENE-7871
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Mikhail Khludnev
> Attachments: LUCENE-7871.patch, LUCENE-7871.patch, LUCENE-7871.patch, 
> LUCENE-7871.patch, LUCENE-7871.patch
>
>
> * fix false positive match for SortedSetDV
> * make {{children}} an iterator instead of bitset.
> see [the 
> comment|https://issues.apache.org/jira/browse/LUCENE-7407?focusedCommentId=16042640&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16042640]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+175) - Build # 20022 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20022/
Java: 64bit/jdk-9-ea+175 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC 
--illegal-access=deny

1 tests failed.
FAILED:  org.apache.solr.cloud.ReplaceNodeTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
at 
__randomizedtesting.SeedInfo.seed([27DC30A61F472DF8:AF880F7CB1BB4000]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.ReplaceNodeTest.test(ReplaceNodeTest.java:95)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 1694 lines...]
   [junit4] JVM J0: stderr was not empty, see: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/temp/junit4-J0-20170630_123117_8292490547773883025225.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) 
   [junit4] Java HotSpot(TM) 64-Bit Server VM w

Re:[1/2] lucene-solr:master: SOLR-1095: Refactor code to standardize replica assignment

[Christine Poerschke (BLOOMBERG/ LONDON)]

The addition of the new ReplicaPosition.java file here appears to have broken 
(the documentation-lint part of) precommit.

 [exec]   missing description: org.apache.solr.common.cloud
 [exec]
 [exec] Missing javadocs were found!

- Original Message -
From: dev@lucene.apache.org
To: comm...@lucene.apache.org
At: 06/30/17 07:21:06

Repository: lucene-solr
Updated Branches:
  refs/heads/master 0159d494f -> 15118d40c


SOLR-1095: Refactor code to standardize replica assignment


Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/196d84b9
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/196d84b9
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/196d84b9

Branch: refs/heads/master
Commit: 196d84b9e08730e9af225450217032cf70d52b5a
Parents: 0159d49
Author: Noble Paul 
Authored: Fri Jun 30 15:49:40 2017 +0930
Committer: Noble Paul 
Committed: Fri Jun 30 15:49:40 2017 +0930

--
 .../src/java/org/apache/solr/cloud/Assign.java  | 131 ---
 .../apache/solr/cloud/CreateCollectionCmd.java  |  31 ++---
 .../java/org/apache/solr/cloud/RestoreCmd.java  |  30 +++--
 .../org/apache/solr/cloud/SplitShardCmd.java|  13 +-
 .../apache/solr/cloud/rule/ReplicaAssigner.java |  68 --
 .../solr/common/cloud/ReplicaPosition.java  |  55 
 .../apache/solr/cloud/rule/RuleEngineTest.java  |   8 +-
 7 files changed, 234 insertions(+), 102 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/196d84b9/solr/core/src/java/org/apache/solr/cloud/Assign.java
--
diff --git a/solr/core/src/java/org/apache/solr/cloud/Assign.java 
b/solr/core/src/java/org/apache/solr/cloud/Assign.java
index 9f21245..7c9752d 100644
--- a/solr/core/src/java/org/apache/solr/cloud/Assign.java
+++ b/solr/core/src/java/org/apache/solr/cloud/Assign.java
@@ -25,10 +25,14 @@ import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
+import java.util.Random;
 import java.util.Set;
+import java.util.function.Supplier;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
+import java.util.stream.Collectors;
 
+import com.google.common.collect.ImmutableMap;
 import org.apache.solr.client.solrj.cloud.autoscaling.Policy;
 import org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper;
 import org.apache.solr.client.solrj.impl.CloudSolrClient;
@@ -40,7 +44,9 @@ import org.apache.solr.common.SolrException;
 import org.apache.solr.common.cloud.ClusterState;
 import org.apache.solr.common.cloud.DocCollection;
 import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.ReplicaPosition;
 import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.common.cloud.ZkNodeProps;
 import org.apache.solr.common.cloud.ZkStateReader;
 import org.apache.solr.common.util.StrUtils;
 import org.apache.solr.common.util.Utils;
@@ -49,6 +55,11 @@ import org.apache.zookeeper.KeeperException;
 
 import static java.util.Collections.singletonMap;
 import static org.apache.solr.client.solrj.cloud.autoscaling.Policy.POLICY;
+import static 
org.apache.solr.cloud.OverseerCollectionMessageHandler.CREATE_NODE_SET;
+import static 
org.apache.solr.cloud.OverseerCollectionMessageHandler.CREATE_NODE_SET_EMPTY;
+import static 
org.apache.solr.cloud.OverseerCollectionMessageHandler.CREATE_NODE_SET_SHUFFLE;
+import static 
org.apache.solr.cloud.OverseerCollectionMessageHandler.CREATE_NODE_SET_SHUFFLE_DEFAULT;
+import static org.apache.solr.common.cloud.DocCollection.SNITCH;
 import static org.apache.solr.common.cloud.ZkStateReader.CORE_NAME_PROP;
 import static org.apache.solr.common.cloud.ZkStateReader.MAX_SHARDS_PER_NODE;
 import static 
org.apache.solr.common.cloud.ZkStateReader.SOLR_AUTOSCALING_CONF_PATH;
@@ -119,7 +130,7 @@ public class Assign {
 returnShardId = shardIdNames.get(0);
 return returnShardId;
   }
-  
+
   public static String buildCoreName(String collectionName, String shard, 
Replica.Type type, int replicaNum) {
 // TODO: Adding the suffix is great for debugging, but may be an issue if 
at some point we want to support a way to change replica type
 return String.format(Locale.ROOT, "%s_%s_replica_%s%s", collectionName, 
shard, type.name().substring(0,1).toLowerCase(Locale.ROOT), replicaNum);
@@ -141,6 +152,91 @@ public class Assign {
   else return replicaName;
 }
   }
+  public static List getLiveOrLiveAndCreateNodeSetList(final 
Set liveNodes, final ZkNodeProps message, final Random random) {
+// TODO: add smarter options that look at the current number of cores per
+// node?
+// for now we just go random (except when createNodeSet and 
createNodeSet.shuffle=false are passed in)
+
+List no

[JENKINS-EA] Lucene-Solr-master-Windows (32bit/jdk-9-ea+173) - Build # 6692 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6692/
Java: 32bit/jdk-9-ea+173 -server -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  org.apache.solr.handler.V2ApiIntegrationTest.testCollectionsApi

Error Message:
Error from server at http://127.0.0.1:61024/solr: 
java.nio.file.InvalidPathException: Illegal char <�> at index 53: 
C:UsersjenkinsworkspaceLucene-Solr-master-Windowssolr�uildsolr-core estJ1 
empsolr.handler.V2ApiIntegrationTest_D3463383BCBCD45E-001 empDir-002

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: 
Error from server at http://127.0.0.1:61024/solr: 
java.nio.file.InvalidPathException: Illegal char <�> at index 53: 
C:UsersjenkinsworkspaceLucene-Solr-master-Windowssolr�uildsolr-core  estJ1  
 empsolr.handler.V2ApiIntegrationTest_D3463383BCBCD45E-001   empDir-002
at 
__randomizedtesting.SeedInfo.seed([D3463383BCBCD45E:FD8D40204A5B0E0]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:805)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:600)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:470)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:400)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1102)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:843)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:774)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.handler.V2ApiIntegrationTest.testCollectionsApi(V2ApiIntegrationTest.java:141)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No

[jira] [Moved] (LUCENE-7893) SynonymGraphFilterFactory proximity search error

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley moved SOLR-10980 to LUCENE-7893:
-

Affects Version/s: (was: 6.6)
   6.6
 Security: (was: Public)
Lucene Fields: New
  Key: LUCENE-7893  (was: SOLR-10980)
  Project: Lucene - Core  (was: Solr)

> SynonymGraphFilterFactory proximity search error
> 
>
> Key: LUCENE-7893
> URL: https://issues.apache.org/jira/browse/LUCENE-7893
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 6.6
>Reporter: Diogo Guilherme Leão Edelmuth
>
> There seems to be an issue when doing proximity searches that include terms 
> that have multi-word synonyms.
> Example:
> consider there's is configured in synonyms.txt
> (
> grand mother, grandmother
> grandfather, granddad
> )
> and there's an indexed field with: (My mother and my grandmother went...)
> Proximity search with: ("mother grandmother"~8)
> won't return the file, while ("father grandfather"~8) does return the 
> analogous file.
> I am not a developer of Solr, so pardon if I am wrong, but I ran it with 
> debug=query and saw that when proximity searches are done with multi-term 
> synonyms, the called function is spanNearQuery: 
> "parsedquery":"SpanNearQuery(spanNear([laudo:mother,
> spanOr([laudo:grand mother, laudo:grandmother])],*0*, true))"
> while proximity searches with one-term synonyms are executed with:
> "MultiPhraseQuery(laudo:\"father (grandfather granddad)\"~10)"
> Note that the SpanNearQuery is called with a slope parameter of 0, no matter 
> what is passed after the tilde. So if I search the exact phrase it does match.
> Here is my field-type, just in case:
>  class="solr.TextField" positionIncrementGap="100">
> 
> 
> 
>  words="lang/stopwords_pt.txt" ignoreCase="true"/>
> 
> 
> 
>  class="solr.LowerCaseFilterFactory"/>
>  words="lang/stopwords_pt.txt" ignoreCase="true"/> class="solr.ASCIIFoldingFilterFactory" preserveOriginal="true"/>
>  ignoreCase="true" synonyms="synonyms_radex.txt"/>
> 
> 
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6807) Make handleSelect=false by default and deprecate StandardRequestHandler

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070006#comment-16070006
 ] 

David Smiley commented on SOLR-6807:


Yes, pretty much.  I added this particular fix as a bug in CHANGES.txt.  It 
seems like a fairly minor bug (i.e. it's not essential if this checking even 
happens) but I don't know enough about this aspect of Solr to be sure.

> Make handleSelect=false by default and deprecate StandardRequestHandler
> ---
>
> Key: SOLR-6807
> URL: https://issues.apache.org/jira/browse/SOLR-6807
> Project: Solr
>  Issue Type: Task
>Affects Versions: 4.10.2
>Reporter: Alexandre Rafalovitch
>Assignee: David Smiley
>Priority: Minor
>  Labels: solrconfig.xml
> Fix For: master (7.0)
>
> Attachments: 
> SOLR_6807__fix__stateVer__check_to_not_depend_on_handleSelect_setting.patch, 
> SOLR_6807_handleSelect_false.patch, SOLR_6807_handleSelect_false.patch, 
> SOLR_6807_handleSelect_false.patch, SOLR_6807_test_files.patch
>
>
> In the solrconfig.xml, we have a long explanation on the legacy 
> ** section. Since we are cleaning up 
> legacy stuff for version 5, is it safe now to flip handleSelect's default to 
> be *false* and therefore remove both the attribute and the whole section 
> explaining it?
> Then, a section in Reference Guide or even a blog post can explain what to do 
> for the old clients that still need it. But it does not seem to be needed 
> anymore for the new users. And possibly cause confusing now that we have 
> implicit, explicit and overlay handlers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-10299) Provide search for online Ref Guide

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-10299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069939#comment-16069939
 ] 

Jan Høydahl edited comment on SOLR-10299 at 6/30/17 12:05 PM:
--

A downside of building the search index from HTML is that you get search hits 
from the left-hand TOC, the menu header and the navigation in footer, so a 
search for any word in the toc such as "analysers" will retrieve *all* pages 
(with the best on top though). A workaround for this if we let our ant build 
generate the index, is to introduce a new build target {{build-site-notoc}} or 
similar that produces clean HTML without the toc, menu bar and footer?

*UPDATE* I tested. Built the site using {{_layouts/default_print.html}} and 
uploaded the index. Now we get much better precision! And the index also shrunk 
from 605k to 375k since the URL prefix is now in index.html and not in 
search-index.js :)


was (Author: janhoy):
A downside of building the search index from HTML is that you get search hits 
from the left-hand TOC, the menu header and the navigation in footer, so a 
search for any word in the toc such as "analysers" will retrieve *all* pages 
(with the best on top though). A workaround for this if we let our ant build 
generate the index, is to introduce a new build target {{build-site-notoc}} or 
similar that produces clean HTML without the toc, menu bar and footer?

> Provide search for online Ref Guide
> ---
>
> Key: SOLR-10299
> URL: https://issues.apache.org/jira/browse/SOLR-10299
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>
> The POC to move the Ref Guide off Confluence did not address providing 
> full-text search of the page content. Not because it's hard or impossible, 
> but because there were plenty of other issues to work on.
> The current HTML page design provides a title index, but to replicate the 
> current Confluence experience, the online version(s) need to provide a 
> full-text search experience.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10299) Provide search for online Ref Guide

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-10299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069939#comment-16069939
 ] 

Jan Høydahl commented on SOLR-10299:


A downside of building the search index from HTML is that you get search hits 
from the left-hand TOC, the menu header and the navigation in footer, so a 
search for any word in the toc such as "analysers" will retrieve *all* pages 
(with the best on top though). A workaround for this if we let our ant build 
generate the index, is to introduce a new build target {{build-site-notoc}} or 
similar that produces clean HTML without the toc, menu bar and footer?

> Provide search for online Ref Guide
> ---
>
> Key: SOLR-10299
> URL: https://issues.apache.org/jira/browse/SOLR-10299
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Reporter: Cassandra Targett
>
> The POC to move the Ref Guide off Confluence did not address providing 
> full-text search of the page content. Not because it's hard or impossible, 
> but because there were plenty of other issues to work on.
> The current HTML page design provides a title index, but to replicate the 
> current Confluence experience, the online version(s) need to provide a 
> full-text search experience.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+175) - Build # 20021 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20021/
Java: 64bit/jdk-9-ea+175 -XX:-UseCompressedOops -XX:+UseSerialGC 
--illegal-access=deny

1 tests failed.
FAILED:  org.apache.solr.cloud.UnloadDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:41247//unloadcollection_shard1_replica2: 
ClusterState says we are the leader 
(http://127.0.0.1:41247/unloadcollection_shard1_replica2), but locally we don't 
think so. Request came from null

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:41247//unloadcollection_shard1_replica2: 
ClusterState says we are the leader 
(http://127.0.0.1:41247/unloadcollection_shard1_replica2), but locally we don't 
think so. Request came from null
at 
__randomizedtesting.SeedInfo.seed([AFFEF86BDA7ACCFC:27AAC7B17486A104]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:624)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
at 
org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:270)
at 
org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:67)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrots

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1411 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1411/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 56206 lines...]
-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/jtidy_tmp
 [echo] Checking for broken links...
 [exec] 
 [exec] Crawl/parse...
 [exec] 
 [exec] Verify...
 [echo] Checking for malformed docs...
 [exec] 
 [exec] 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/docs/solr-core/overview-summary.html
 [exec]   missing description: org.apache.solr.common.cloud
 [exec] 
 [exec] Missing javadocs were found!

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:810: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:669: 
The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:685: 
The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2506:
 exec returned: 1

Total time: 73 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10982) Deprecate FieldCache

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069772#comment-16069772
 ] 

Uwe Schindler commented on SOLR-10982:
--

Thanks Tomás! The proposal looks good!

> Deprecate FieldCache
> 
>
> Key: SOLR-10982
> URL: https://issues.apache.org/jira/browse/SOLR-10982
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
> Environment: 
>Reporter: Tomás Fernández Löbbe
>
> Extracting this idea suggested by [~thetaphi] in SOLR-10803. The proposal is 
> to:
> # Enable DocValues by default for numeric/string/date fields. (SOLR-10808)
> # Have a merge policy that can generate the DocValues at merge time if a 
> field doesn’t have them but should according to the schema (SOLR-10046). Make 
> this Merge Policy the default.
> # When using an index created with 7.x (maybe using the new metadata added by 
> [~jpountz] recently) and something tries to access FieldCache (e.g. for 
> sorting or faceting or functions), it should fail the query. 
> # Remove FieldCache in 8.0



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6441) MoreLikeThis support for stopwords as in Lucene

[Goran Miskovic (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069770#comment-16069770
 ] 

Goran Miskovic commented on SOLR-6441:
--

Hi [~mikemccand]

It appears that [~jeroens] patch was not merged although the resolution was set 
to done and you closed the ticket.

Could you provide more information, please?

Thanks,
Goran

> MoreLikeThis support for stopwords as in Lucene
> ---
>
> Key: SOLR-6441
> URL: https://issues.apache.org/jira/browse/SOLR-6441
> Project: Solr
>  Issue Type: Improvement
>  Components: MoreLikeThis
>Affects Versions: 4.10
>Reporter: Jeroen Steggink
>Priority: Minor
>  Labels: difficulty-easy, impact-low, workaround-exists
> Fix For: 4.10.1, 5.0
>
> Attachments: SOLR-6441.patch, SOLR-6441.patch
>
>
> In the Lucene implementation of MoreLikeThis, it's possible to add a list of 
> stopwords which are considered "uninteresting" and are ignored.
> It would be a great addition to the MoreLikeThisHandler to be able to specify 
> a list of stopwords.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-master - Build # 1926 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1926/

1 tests failed.
FAILED:  org.apache.solr.update.AutoCommitTest.testMaxTime

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
at 
__randomizedtesting.SeedInfo.seed([A04985DB95B8E33A:3ABDF8390B227F06]:0)
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:878)
at 
org.apache.solr.update.AutoCommitTest.testMaxTime(AutoCommitTest.java:270)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//result[@numFound=1]
xml response was: 

00


request was:q=id:529&qt=&start=0&rows=20&version=2.2
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:871)
... 40 more




Build Log:
[...truncated 10888 lines...]
   [junit4] Suite: org.apache.solr.update.AutoCommitTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkin