[jira] [Commented] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459460#comment-16459460
 ] 

Erick Erickson commented on LUCENE-7976:


OK, I think this is getting quite close. The place I'm most uncomfortable is in 
findForcedMerges. See the TODO around line 778 plus the fact that there's a 
bunch of special handling depending on whether we're forceMerging to 1 segment, 
a max count or respecting max segments size. I want to make one more pass 
though it, there _ought_ to be a more organized way of doing this.

Also, when giving a maximum number of segments, we calculate the ideal segment 
size and then I increase it by 25% on the theory that segments won't fit 
perfectly, so allow some extra space in hopes that all the segments will be fit 
into the max segment count the first time through.

However, there are still edge cases I think where that won't necessarily work 
on the first pass, especially if there are very many segments. In that case, at 
the very end there's a loop essentially saying "go through as many iterations 
as necessary increasing the max segment size by 25% each time until you can fit 
them all in the required number of segments". This really means that in this 
case you could rewrite the entire index twice. Is that OK? I don't want to 
spend a lot of time on this case though, it seems to me that if you specify 
this you'll have to live with this edge case.

[~mikemccand] There's another departure from the old process here. If there are 
multiple passes for forceMerge, I keep returning null in until there aren't any 
current merges running involving the original segments. Is there any real point 
in trying to create another merge specification if there are merges from 
previous passes going on? This is around line 684.

I beasted all of Mikes failures 120 times along with TestTieredMergePolicy and 
no failures. All tests pass and precommit worked.

Then, of course I made one tiny change so I'll have to go 'round that testing 
again. I also have to make another couple of runs at counting the total bytes 
written to see if something crept in.

That said, I think this is the last major rearrangement I want to do. If my 
additional testing succeeds and there are no objections, I'll probably commit 
sometime this weekend.

Thanks to all who've looked at this!

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
>  (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO 
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with 
> >> smaller segments to bring the resulting segment up to 5G. If no smaller 
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). 
> >> It would be rewritten into a single segment removing all deleted docs no 
> >> matter how big it is to start. The 100G example above would be rewritten 
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the 
> default 

[jira] [Updated] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

[Erick Erickson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson updated LUCENE-7976:
---
Attachment: LUCENE-7976.patch

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
>  (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO 
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with 
> >> smaller segments to bring the resulting segment up to 5G. If no smaller 
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). 
> >> It would be rewritten into a single segment removing all deleted docs no 
> >> matter how big it is to start. The 100G example above would be rewritten 
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the 
> default would be the same behavior we see now. As it stands now, though, 
> there's no way to recover from an optimize/forceMerge except to re-index from 
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the 
> wild" with 10s of  shards replicated 3 or more times. And that doesn't even 
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A 
> new merge policy is certainly an alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Weekly BadApple report:

[Erick Erickson]

In order to reduce some of the make-work, I collect failed tests Fri->Mon
so the BadApple'd tests on Thursday don't clutter things up.

BadApple candidates: I'll BadApple these on Thursday unless there are objections

   
org.apache.solr.client.solrj.io.stream.MathExpressionTest.testGammaDistribution
   org.apache.solr.cloud.AddReplicaTest.test
   org.apache.solr.cloud.CreateRoutedAliasTest.testCollectionNamesMustBeAbsent
   org.apache.solr.cloud.CreateRoutedAliasTest.testTimezoneAbsoluteDate
   org.apache.solr.cloud.CreateRoutedAliasTest.testV1
   org.apache.solr.cloud.CreateRoutedAliasTest.testV2
   org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingDVOnly
   org.apache.solr.cloud.LIRRollingUpdatesTest.testNewLeaderAndMixedReplicas
   org.apache.solr.cloud.LIRRollingUpdatesTest.testNewLeaderOldReplica
   org.apache.solr.cloud.LIRRollingUpdatesTest.testOldLeaderAndMixedReplicas
   
org.apache.solr.cloud.LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
   org.apache.solr.cloud.OverseerRolesTest.testOverseerRole
   
org.apache.solr.cloud.SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection
   org.apache.solr.cloud.TestPullReplica.testKillLeader
   org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test
   org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeLost
   org.apache.solr.cloud.autoscaling.NodeAddedTriggerTest.testRestoreState
   
org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication
   org.apache.solr.schema.TestCloudSchemaless.test
   org.apache.solr.search.TestStressRecovery.testStressRecovery
   org.apache.solr.update.TestInPlaceUpdatesDistrib.test


I believe these are being worked on, so I will _not_ BadApple them
even though they fail.
What about the other autoscaling tests?

   org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration
   org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration
   org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger


Number of AwaitsFix: 21 Number of BadApples: 90

*AwaitsFix Annotations:


Lucene AwaitsFix
GeoPolygonTest.java
   testLUCENE8276_case3()
   //@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/LUCENE-8276;)

GeoPolygonTest.java
   testLUCENE8280()
   //@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/LUCENE-8280;)

GeoPolygonTest.java
   testLUCENE8281()
   //@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/LUCENE-8281;)

RandomGeoPolygonTest.java
   testCompareBigPolygons()
   //@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/LUCENE-8281;)

RandomGeoPolygonTest.java
   testCompareSmallPolygons()
   //@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/LUCENE-8281;)

TestControlledRealTimeReopenThread.java
   testCRTReopen()
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-5737;)

TestICUNormalizer2CharFilter.java
   testRandomStrings()
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-5595;)

TestICUTokenizerCJK.java
   TestICUTokenizerCJK suite
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/LUCENE-8222;)

TestMoreLikeThis.java
   testMultiFieldShouldReturnPerFieldBooleanQuery()
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-7161;)

UIMABaseAnalyzerTest.java
   testRandomStrings()
   @Test @AwaitsFix(bugUrl =
"https://issues.apache.org/jira/browse/LUCENE-3869;)

UIMABaseAnalyzerTest.java
   testRandomStringsWithConfigurationParameters()
   @Test @AwaitsFix(bugUrl =
"https://issues.apache.org/jira/browse/LUCENE-3869;)

UIMATypeAwareAnalyzerTest.java
   testRandomStrings()
   @Test @AwaitsFix(bugUrl =
"https://issues.apache.org/jira/browse/LUCENE-3869;)


Solr AwaitsFix
ReplaceNodeNoTargetTest.java
   ReplaceNodeNoTargetTest suite
   @LuceneTestCase.AwaitsFix(bugUrl =
"https://issues.apache.org/jira/browse/SOLR-11067;)

TestCollapseQParserPlugin.java
   testStringCollapse()
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-11974;)

TestImpersonationWithHadoopAuth.java
   testForwarding()
   @AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/HADOOP-9893;)

TestLTRReRankingPipeline.java
   testDifferentTopN()
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/SOLR-11134;)

TestMinMaxOnMultiValuedField.java
   testDoubleFieldCache()
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709;)

TestMinMaxOnMultiValuedField.java
   testFloatFieldCache()
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709;)

TestMinMaxOnMultiValuedField.java
   testIntFieldCache()
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709;)

TestMinMaxOnMultiValuedField.java
   testLongFieldCache()
   @AwaitsFix(bugUrl = "https://issues.apache.org/jira/browse/LUCENE-6709;)



*BadApple Annotations:

Lucene BadApple

Solr BadApple
AddReplicaTest.java
   test()

[jira] [Commented] (SOLR-12278) Ignore very large document on indexing

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459430#comment-16459430
 ] 

Cao Manh Dat commented on SOLR-12278:
-

Patch for this ticket passed the precommit. Thank [~dsmiley] for your review.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12278) Ignore very large document on indexing

[Cao Manh Dat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cao Manh Dat updated SOLR-12278:

Attachment: SOLR-12278.patch

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12297) Create a good SolrClient for SolrCloud.

[Mark Miller (JIRA)]

Mark Miller created SOLR-12297:
--

 Summary: Create a good SolrClient for SolrCloud.
 Key: SOLR-12297
 URL: https://issues.apache.org/jira/browse/SOLR-12297
 Project: Solr
  Issue Type: New Feature
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Mark Miller


Blocking or async support as well as HTTP2 compatible with multiplexing.

Once it supports enough and is stable, replace internal usage, allowing async, 
and eventually move to HTTP2 connector and allow multiplexing. Could support 
HTTP1.1 and HTTP2 on different ports depending on state of the world then.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Windows (64bit/jdk-11-ea+5) - Build # 7297 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7297/
Java: 64bit/jdk-11-ea+5 -XX:+UseCompressedOops -XX:+UseG1GC

20 tests failed.
FAILED:  org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingDVOnly

Error Message:
Unexpected number of elements in the group for intGSF: 6

Stack Trace:
java.lang.AssertionError: Unexpected number of elements in the group for 
intGSF: 6
at 
__randomizedtesting.SeedInfo.seed([4F2663511E674E29:D49D0D09533F7C77]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingDVOnly(DocValuesNotIndexedTest.java:379)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:841)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
found:2[index.20180430204049960, index.20180430204051756, index.properties, 
replication.properties, snapshot_metadata]

Stack Trace:
java.lang.AssertionError: found:2[index.20180430204049960, 

[jira] [Created] (SOLR-12296) Test fail in TimeRoutedAliasUpdateProcessorTest RE "We need to create a new time routed collection but for unknown reasons were unable to do so."

[David Smiley (JIRA)]

David Smiley created SOLR-12296:
---

 Summary: Test fail in TimeRoutedAliasUpdateProcessorTest RE "We 
need to create a new time routed collection but for unknown reasons were unable 
to do so."
 Key: SOLR-12296
 URL: https://issues.apache.org/jira/browse/SOLR-12296
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: SolrCloud
Reporter: David Smiley
Assignee: David Smiley


Two test fails came up recently with this exception:
{{o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: We need to 
create a new time routed collection but for unknown reasons were unable to do 
so.}}
It's a message in TimeRoutedAliasUpdateProcessor.

thetaphi/Lucene-Solr-7.x-MacOSX/602
thetaphi/Lucene-Solr-master-Solaris/1820




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-12278) Ignore very large document on indexing

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459412#comment-16459412
 ] 

Cao Manh Dat edited comment on SOLR-12278 at 5/1/18 3:28 AM:
-

[~dsmiley] some problem of that approach
 * we have to modify all other parsers,
 * each parser has its own set of parameters, which make the size of a 
SolrInputDocument quite different with the number of bytes from the input (ie: 
SOLR-6304)
 * what happens if the users have some processor in the middle which 
significantly enriches the SolrInputDocument

In short vision, IgnoreLargeDocumentProcessor might handy for users who need to 
filter large documents and work accurately in all cases.


was (Author: caomanhdat):
[~dsmiley] some problem of that approach
 * we have to modify all other parsers,
 * each parser has its own set of parameters, which make the size of a 
SolrInputDocument quite different with the number of bytes from the input (ie: 
SOLR-6304)
 * what happens if the users have some processor in the middle which 
significantly enriches the SolrInputDocument

In short vision, IgnoreLargeDocumentProcessor might handy for users who need to 
filter large documents.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12278) Ignore very large document on indexing

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459417#comment-16459417
 ] 

David Smiley commented on SOLR-12278:
-

Sure, the URP is still useful.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-12278) Ignore very large document on indexing

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459412#comment-16459412
 ] 

Cao Manh Dat edited comment on SOLR-12278 at 5/1/18 3:16 AM:
-

[~dsmiley] some problem of that approach
 * we have to modify all other parsers,
 * each parser has its own set of parameters, which make the size of a 
SolrInputDocument quite different with the number of bytes from the input (ie: 
SOLR-6304)
 * what happens if the users have some processor in the middle which 
significantly enriches the SolrInputDocument

In short vision, IgnoreLargeDocumentProcessor might handy for users who need to 
filter large documents.


was (Author: caomanhdat):
[~dsmiley] some problem of that approach
 * we have to modify all other parsers,
 * each parser has its own set of parameters, which make the size of a 
SolrInputDocument quite different with the number of bytes of the input (ie: 
SOLR-6304)
 * what happens if the users have some processor in the middle which enriches 
the SolrInputDocument

In short vision, IgnoreLargeDocumentProcessor might handy for users who need to 
filter large documents.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-12278) Ignore very large document on indexing

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459412#comment-16459412
 ] 

Cao Manh Dat edited comment on SOLR-12278 at 5/1/18 3:13 AM:
-

[~dsmiley] some problem of that approach
 * we have to modify all other parsers,
 * each parser has its own set of parameters, which make the size of a 
SolrInputDocument quite different with the number of bytes of the input (ie: 
SOLR-6304)
 * what happens if the users have some processor in the middle which enriches 
the SolrInputDocument

In short vision, IgnoreLargeDocumentProcessor might handy for users who need to 
filter large documents.


was (Author: caomanhdat):
[~dsmiley] problem of that approach is we have to modify all other parsers, not 
mention that each parser has its worn set of parameters.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12278) Ignore very large document on indexing

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459412#comment-16459412
 ] 

Cao Manh Dat commented on SOLR-12278:
-

[~dsmiley] problem of that approach is we have to modify all other parsers, not 
mention that each parser has its worn set of parameters.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 588 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/588/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest

Error Message:
Expected data dir and instance dir of core_node5 is deleted

Stack Trace:
java.util.concurrent.TimeoutException: Expected data dir and instance dir of 
core_node5 is deleted
at 
__randomizedtesting.SeedInfo.seed([2F12178DC25DD4F0:E22C8C7E8DFCA212]:0)
at org.apache.solr.util.TimeOut.waitFor(TimeOut.java:66)
at 
org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest(DeleteInactiveReplicaTest.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14250 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteInactiveReplicaTest
   [junit4]   2> 2138355 INFO  
(SUITE-DeleteInactiveReplicaTest-seed#[2F12178DC25DD4F0]-worker) [] 

[jira] [Updated] (SOLR-12295) Time Routed Aliases: Delete obsolete collections Async

[Gus Heck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gus Heck updated SOLR-12295:

Attachment: SOLR-12295.patch

> Time Routed Aliases: Delete obsolete collections Async
> --
>
> Key: SOLR-12295
> URL: https://issues.apache.org/jira/browse/SOLR-12295
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Reporter: Gus Heck
>Priority: Major
> Attachments: SOLR-12295.patch
>
>
> Presently TimeRoutedAliasUpdaateProcessor calls the MaintainRoutedAliasCmd 
> command to handle the book keeping around creation of new and deletion of 
> obsolete collections (the collections are representing time segments within 
> the time routed alias). While we generally need to wait for collection 
> creation before proceeding, since the incoming request may  be placing a 
> document in the collection to be created, deletions need only be initiated 
> and do not need to impede processing of updates.
> In stress testing in SOLR-11949 the delays caused by synchronous updates were 
> sufficient to fail the test via an apparent deadlock (actual locked threads 
> could not be located however). With async deletes provided in this patch the 
> stress test has passed for several consecutive executions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12295) Time Routed Aliases: Delete obsolete collections Async

[Gus Heck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gus Heck updated SOLR-12295:

Attachment: (was: SOLR-12030.patch)

> Time Routed Aliases: Delete obsolete collections Async
> --
>
> Key: SOLR-12295
> URL: https://issues.apache.org/jira/browse/SOLR-12295
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Reporter: Gus Heck
>Priority: Major
>
> Presently TimeRoutedAliasUpdaateProcessor calls the MaintainRoutedAliasCmd 
> command to handle the book keeping around creation of new and deletion of 
> obsolete collections (the collections are representing time segments within 
> the time routed alias). While we generally need to wait for collection 
> creation before proceeding, since the incoming request may  be placing a 
> document in the collection to be created, deletions need only be initiated 
> and do not need to impede processing of updates.
> In stress testing in SOLR-11949 the delays caused by synchronous updates were 
> sufficient to fail the test via an apparent deadlock (actual locked threads 
> could not be located however). With async deletes provided in this patch the 
> stress test has passed for several consecutive executions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12295) Time Routed Aliases: Delete obsolete collections Async

[Gus Heck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gus Heck updated SOLR-12295:

Attachment: (was: SOLR-12295)

> Time Routed Aliases: Delete obsolete collections Async
> --
>
> Key: SOLR-12295
> URL: https://issues.apache.org/jira/browse/SOLR-12295
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Reporter: Gus Heck
>Priority: Major
>
> Presently TimeRoutedAliasUpdaateProcessor calls the MaintainRoutedAliasCmd 
> command to handle the book keeping around creation of new and deletion of 
> obsolete collections (the collections are representing time segments within 
> the time routed alias). While we generally need to wait for collection 
> creation before proceeding, since the incoming request may  be placing a 
> document in the collection to be created, deletions need only be initiated 
> and do not need to impede processing of updates.
> In stress testing in SOLR-11949 the delays caused by synchronous updates were 
> sufficient to fail the test via an apparent deadlock (actual locked threads 
> could not be located however). With async deletes provided in this patch the 
> stress test has passed for several consecutive executions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12295) Time Routed Aliases: Delete obsolete collections Async

[Gus Heck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gus Heck updated SOLR-12295:

Attachment: SOLR-12030.patch

> Time Routed Aliases: Delete obsolete collections Async
> --
>
> Key: SOLR-12295
> URL: https://issues.apache.org/jira/browse/SOLR-12295
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Reporter: Gus Heck
>Priority: Major
>
> Presently TimeRoutedAliasUpdaateProcessor calls the MaintainRoutedAliasCmd 
> command to handle the book keeping around creation of new and deletion of 
> obsolete collections (the collections are representing time segments within 
> the time routed alias). While we generally need to wait for collection 
> creation before proceeding, since the incoming request may  be placing a 
> document in the collection to be created, deletions need only be initiated 
> and do not need to impede processing of updates.
> In stress testing in SOLR-11949 the delays caused by synchronous updates were 
> sufficient to fail the test via an apparent deadlock (actual locked threads 
> could not be located however). With async deletes provided in this patch the 
> stress test has passed for several consecutive executions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12295) Time Routed Aliases: Delete obsolete collections Async

[Gus Heck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gus Heck updated SOLR-12295:

Attachment: SOLR-12295

> Time Routed Aliases: Delete obsolete collections Async
> --
>
> Key: SOLR-12295
> URL: https://issues.apache.org/jira/browse/SOLR-12295
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Reporter: Gus Heck
>Priority: Major
> Attachments: SOLR-12295
>
>
> Presently TimeRoutedAliasUpdaateProcessor calls the MaintainRoutedAliasCmd 
> command to handle the book keeping around creation of new and deletion of 
> obsolete collections (the collections are representing time segments within 
> the time routed alias). While we generally need to wait for collection 
> creation before proceeding, since the incoming request may  be placing a 
> document in the collection to be created, deletions need only be initiated 
> and do not need to impede processing of updates.
> In stress testing in SOLR-11949 the delays caused by synchronous updates were 
> sufficient to fail the test via an apparent deadlock (actual locked threads 
> could not be located however). With async deletes provided in this patch the 
> stress test has passed for several consecutive executions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12295) Time Routed Aliases: Delete obsolete collections Async

[Gus Heck (JIRA)]

Gus Heck created SOLR-12295:
---

 Summary: Time Routed Aliases: Delete obsolete collections Async
 Key: SOLR-12295
 URL: https://issues.apache.org/jira/browse/SOLR-12295
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: UpdateRequestProcessors
Reporter: Gus Heck


Presently TimeRoutedAliasUpdaateProcessor calls the MaintainRoutedAliasCmd 
command to handle the book keeping around creation of new and deletion of 
obsolete collections (the collections are representing time segments within the 
time routed alias). While we generally need to wait for collection creation 
before proceeding, since the incoming request may  be placing a document in the 
collection to be created, deletions need only be initiated and do not need to 
impede processing of updates.

In stress testing in SOLR-11949 the delays caused by synchronous updates were 
sufficient to fail the test via an apparent deadlock (actual locked threads 
could not be located however). With async deletes provided in this patch the 
stress test has passed for several consecutive executions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1831 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1831/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore

Error Message:
ObjectTracker found 7 object(s) that were not released!!! [MMapDirectory, 
MMapDirectory, MMapDirectory, InternalHttpClient, SolrCore, MMapDirectory, 
InternalHttpClient] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MMapDirectory  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:526)  
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:369) 
 at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) 
 at 
org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1159)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 at java.lang.Thread.run(Thread.java:748)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MMapDirectory  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:354)  at 
org.apache.solr.core.SolrCore.initIndex(SolrCore.java:732)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:957)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:866)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1051)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:961)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:90)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:356)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)  
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)  
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:393)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:339)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) 
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
  at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)  
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
 at org.eclipse.jetty.server.Server.handle(Server.java:530)  at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)  at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)  at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
  at 

[JENKINS] Lucene-Solr-NightlyTests-7.3 - Build # 20 - Still Failing

[Apache Jenkins Server]

Error processing tokens: Error while parsing action 
'Text/ZeroOrMore/FirstOf/Token/DelimitedToken/DelimitedToken_Action3' at input 
position (line 76, pos 4):
)"}
   ^

java.lang.OutOfMemoryError: Java heap space

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-7.3-Linux (64bit/jdk-11-ea+5) - Build # 153 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.3-Linux/153/
Java: 64bit/jdk-11-ea+5 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection

Error Message:
Delete action failed!

Stack Trace:
java.lang.AssertionError: Delete action failed!
at 
__randomizedtesting.SeedInfo.seed([17CE91AA0ACE2B23:4ADA3C53BA19285]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.SolrCloudExampleTest.doTestDeleteAction(SolrCloudExampleTest.java:193)
at 
org.apache.solr.cloud.SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection(SolrCloudExampleTest.java:169)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS] Lucene-Solr-7.x-MacOSX (64bit/jdk1.8.0) - Build # 607 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-MacOSX/607/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

3 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger

Error Message:
waitFor not elapsed but produced an event

Stack Trace:
java.lang.AssertionError: waitFor not elapsed but produced an event
at 
__randomizedtesting.SeedInfo.seed([2448CBB76C0732CC:4783FD35F5C841E1]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at 
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger(IndexSizeTriggerTest.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: 
DocCollection(testSplitIntegration_collection//clusterstate.json/56)={   

[jira] [Commented] (LUCENE-8273) Add a ConditionalTokenFilter

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459103#comment-16459103
 ] 

Steve Rowe commented on LUCENE-8273:


bq. {{if}} is a keyword, unfortunately, so that won't work. Maybe 
{{ifCondition}} instead?

Shorter and almost as good?: {{when}} (used e.g. by Mockito)

> Add a ConditionalTokenFilter
> 
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch, LUCENE-8273.patch, 
> LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265.  It would be useful to be able to wrap a TokenFilter 
> in such a way that it could optionally be bypassed based on the current state 
> of the TokenStream.  This could be used to, for example, only apply 
> WordDelimiterFilter to terms that contain hyphens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8278) UAX29URLEmailTokenizer is not detecting some tokens as URL type

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459037#comment-16459037
 ] 

Steve Rowe commented on LUCENE-8278:


Hmm, "example.co" and "example.info" in the above test succeed, so the problem 
here *is* somehow related to TLD spelling. 

> UAX29URLEmailTokenizer is not detecting some tokens as URL type
> ---
>
> Key: LUCENE-8278
> URL: https://issues.apache.org/jira/browse/LUCENE-8278
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Junte Zhang
>Priority: Minor
>
> We are using the UAX29URLEmailTokenizer so we can use the token types in our 
> plugins.
> However, I noticed that the tokenizer is not detecting certain URLs as  
> but  instead.
> Examples that are not working:
>  * example.com is 
>  * example.net is 
> But:
>  * https://example.com is 
>  * as is https://example.net
> Examples that work:
>  * example.ch is 
>  * example.co.uk is 
>  * example.nl is 
> I have checked this JIRA, and could not find an issue. I have tested this on 
> Lucene (Solr) 6.4.1 and 7.3.
> Could someone confirm my findings and advise what I could do to (help) 
> resolve this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-2332) TikaEntityProcessor retrieves only File Names from Zip extraction

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl resolved SOLR-2332.
---
Resolution: Implemented

> TikaEntityProcessor retrieves only File Names from Zip extraction
> -
>
> Key: SOLR-2332
> URL: https://issues.apache.org/jira/browse/SOLR-2332
> Project: Solr
>  Issue Type: Bug
>  Components: contrib - DataImportHandler
>Reporter: Jayendra Patil
>Priority: Major
> Attachments: SOLR-2332.patch, solr-word.zip
>
>
> Extraction of Zip files using TikaEntityProcessor results in only names of 
> file.
> It does not extract the contents of the Files in the Zip



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-2416) Solr Cell fails to index Zip file contents

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl resolved SOLR-2416.
---
Resolution: Implemented

> Solr Cell fails to index Zip file contents
> --
>
> Key: SOLR-2416
> URL: https://issues.apache.org/jira/browse/SOLR-2416
> Project: Solr
>  Issue Type: Bug
>  Components: contrib - DataImportHandler, contrib - Solr Cell (Tika 
> extraction)
>Affects Versions: 1.4.1
>Reporter: Jayendra Patil
>Priority: Major
> Fix For: 6.0
>
> Attachments: SOLR-2416_ExtractingDocumentLoader.patch, SOLR-4216.patch
>
>
> Working with the latest Solr Trunk code and seems the Tika handlers for Solr 
> Cell (ExtractingDocumentLoader.java) and Data Import handler 
> (TikaEntityProcessor.java) fails to index the zip file contents again.
> It just indexes the file names again.
> This issue was addressed some time back, late last year, but seems to have 
> reappeared with the latest code.
> Jira for the Data Import handler part with the patch and the testcase - 
> https://issues.apache.org/jira/browse/SOLR-2332.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4597 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4597/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseSerialGC

5 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration

Error Message:
did not finish processing in time

Stack Trace:
java.lang.AssertionError: did not finish processing in time
at 
__randomizedtesting.SeedInfo.seed([916CFE1DFE0BF0BD:C2D5BCAD1C1A6547]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration(IndexSizeTriggerTest.java:404)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: 
DocCollection(testSplitIntegration_collection//clusterstate.json/66)={   
"replicationFactor":"2",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   

[jira] [Updated] (SOLR-11277) Add auto hard commit setting based on tlog size

[Rupa Shankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rupa Shankar updated SOLR-11277:

Attachment: SOLR-11277.patch

> Add auto hard commit setting based on tlog size
> ---
>
> Key: SOLR-11277
> URL: https://issues.apache.org/jira/browse/SOLR-11277
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Rupa Shankar
>Assignee: Anshum Gupta
>Priority: Major
> Attachments: SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, 
> SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, 
> max_size_auto_commit.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When indexing documents of variable sizes and at variable schedules, it can 
> be hard to estimate the optimal auto hard commit maxDocs or maxTime settings. 
> We’ve had some occurrences of really huge tlogs, resulting in serious issues, 
> so in an attempt to avoid this, it would be great to have a “maxSize” setting 
> based on the tlog size on disk. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8278) UAX29URLEmailTokenizer is not detecting some tokens as URL type

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459032#comment-16459032
 ] 

Steve Rowe commented on LUCENE-8278:


Confirming there is an issue, but I don't think the spellings of "example.com" 
and "example.net" are the problem though; more likely this is related to an 
end-of-input issue.  This test added to {{TestUAX29URLEmailTokenizer}} fails 
for me:
  
{noformat}
  public void testExampleURLs() throws Exception {
Analyzer analyzer = new Analyzer() {
  @Override protected TokenStreamComponents createComponents(String 
fieldName) {
return new TokenStreamComponents(new 
UAX29URLEmailTokenizer(newAttributeFactory()));
  }};

// A trailing space allows these to succeed
BaseTokenStreamTestCase.assertAnalyzesTo(analyzer, "example.com ", new 
String[]{"example.com"}, new String[]{""});
BaseTokenStreamTestCase.assertAnalyzesTo(analyzer, "example.net ", new 
String[]{"example.net"}, new String[]{""});

// These fail
BaseTokenStreamTestCase.assertAnalyzesTo(analyzer, "example.com", new 
String[]{"example.com"}, new String[]{""});
BaseTokenStreamTestCase.assertAnalyzesTo(analyzer, "example.net", new 
String[]{"example.net"}, new String[]{""});
  }
{noformat}

So there is an issue here with no-scheme end-of-input URLs not being recognized 
as type {{}}.

> UAX29URLEmailTokenizer is not detecting some tokens as URL type
> ---
>
> Key: LUCENE-8278
> URL: https://issues.apache.org/jira/browse/LUCENE-8278
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Junte Zhang
>Priority: Minor
>
> We are using the UAX29URLEmailTokenizer so we can use the token types in our 
> plugins.
> However, I noticed that the tokenizer is not detecting certain URLs as  
> but  instead.
> Examples that are not working:
>  * example.com is 
>  * example.net is 
> But:
>  * https://example.com is 
>  * as is https://example.net
> Examples that work:
>  * example.ch is 
>  * example.co.uk is 
>  * example.nl is 
> I have checked this JIRA, and could not find an issue. I have tested this on 
> Lucene (Solr) 6.4.1 and 7.3.
> Could someone confirm my findings and advise what I could do to (help) 
> resolve this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2332) TikaEntityProcessor retrieves only File Names from Zip extraction

[Markus Schuch (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459006#comment-16459006
 ] 

Markus Schuch commented on SOLR-2332:
-

I just tested ZIP data import with 7.3.0 and i can confirm that due to the new 
default behavior of Tika 1.15+ the TikaEntityProcessor extracts the text of the 
embedded documents as well and not only the file names as stated in the issue 
description.

So this was fixed with SOLR-10335.

> TikaEntityProcessor retrieves only File Names from Zip extraction
> -
>
> Key: SOLR-2332
> URL: https://issues.apache.org/jira/browse/SOLR-2332
> Project: Solr
>  Issue Type: Bug
>  Components: contrib - DataImportHandler
>Reporter: Jayendra Patil
>Priority: Major
> Attachments: SOLR-2332.patch, solr-word.zip
>
>
> Extraction of Zip files using TikaEntityProcessor results in only names of 
> file.
> It does not extract the contents of the Files in the Zip



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2416) Solr Cell fails to index Zip file contents

[Markus Schuch (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458950#comment-16458950
 ] 

Markus Schuch commented on SOLR-2416:
-

I just tested ZIP extraction with 7.3.0 and i can confirm that due to the new 
default behavior of Tika 1.15+ the Extracting Request Handler extracts the text 
of the embedded documents as well and not only the file names as stated in the 
issue description.

So this was fixed with SOLR-10335.

> Solr Cell fails to index Zip file contents
> --
>
> Key: SOLR-2416
> URL: https://issues.apache.org/jira/browse/SOLR-2416
> Project: Solr
>  Issue Type: Bug
>  Components: contrib - DataImportHandler, contrib - Solr Cell (Tika 
> extraction)
>Affects Versions: 1.4.1
>Reporter: Jayendra Patil
>Priority: Major
> Fix For: 6.0
>
> Attachments: SOLR-2416_ExtractingDocumentLoader.patch, SOLR-4216.patch
>
>
> Working with the latest Solr Trunk code and seems the Tika handlers for Solr 
> Cell (ExtractingDocumentLoader.java) and Data Import handler 
> (TikaEntityProcessor.java) fails to index the zip file contents again.
> It just indexes the file names again.
> This issue was addressed some time back, late last year, but seems to have 
> reappeared with the latest code.
> Jira for the Data Import handler part with the patch and the testcase - 
> https://issues.apache.org/jira/browse/SOLR-2332.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12282) JSONResponseWriter should return Content-Type application/javascript for JSONP requests

[Markus Schuch (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458887#comment-16458887
 ] 

Markus Schuch commented on SOLR-12282:
--

Good point. Browser support for CORS looks way better than last time i checked 
([https://caniuse.com/#feat=cors).]

> JSONResponseWriter should return Content-Type application/javascript for 
> JSONP requests
> ---
>
> Key: SOLR-12282
> URL: https://issues.apache.org/jira/browse/SOLR-12282
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Response Writers
>Reporter: Markus Schuch
>Priority: Minor
> Attachments: SOLR-12282.patch
>
>
> The JSONResponseWriter handles two purposes:
>  * writing JSON responses (wt=json)
>  * writing JSONP responses, namely JSON responses wrapped by a JavaScript 
> function (wt=json=myFunction)
> The response writer returns the same Content-Type in both cases. (default: 
> application/json)
> But for JSONP the correct Content-Type would be "application/javascript".
> The response type is configurable, but it is currently not possible to return 
> the correct Content-Type in both cases with one configuration at the same 
> time.
> The attached patch changes the default Content-Type to 
> {{application/javascript; charset=utf-8}} for responses with wrapper 
> functions (JSONP). If param {{content-type}} is configured, this Content-Type 
> will be returned for both JSON and JSONP to ensure backward compatibility.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12284) WordBreakSolrSpellChecker incorrectly adds parenthesis when breaking words

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458885#comment-16458885
 ] 

ASF subversion and git services commented on SOLR-12284:


Commit 162a077ef3da3e7a4f8aab6c5e693c0fc812dfdf in lucene-solr's branch 
refs/heads/branch_7x from jdyer1
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=162a077 ]

SOLR-12284: Stop adding parenthesis to word-break suggestions, unless query 
uses boolean operators.


> WordBreakSolrSpellChecker incorrectly adds parenthesis when breaking words
> --
>
> Key: SOLR-12284
> URL: https://issues.apache.org/jira/browse/SOLR-12284
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: spellchecker
>Affects Versions: 7.3
>Reporter: James Dyer
>Assignee: James Dyer
>Priority: Minor
> Attachments: SOLR-12284.patch
>
>
> When using WordBreakSolrSpellChecker to break single words into multiple, the 
> collation queries include parenthesis around the original term.  In some 
> cases, this causes required terms to become optional and users get spurious 
> nonsensical collation results.
> For instance, if I search: +eward +smith 
> ...If +ward +smith is a match, it might give a collation like: (+e +ward) 
> +smith
> ...This requires either the "e" or the "ward" to exist, but not both.  But 
> users are more likely to want both terms to be required, so it would be 
> better if it was not adding parenthesis.
> This might be the cause of SOLR-5995 and [this SO 
> issue|https://stackoverflow.com/questions/23849747/solr-wordbreak-spellchecker-breaking-words-into-letters-excessive-breaking]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Windows (32bit/jdk1.8.0_144) - Build # 572 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/572/
Java: 32bit/jdk1.8.0_144 -server -XX:+UseSerialGC

21 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration

Error Message:
did not finish processing in time

Stack Trace:
java.lang.AssertionError: did not finish processing in time
at 
__randomizedtesting.SeedInfo.seed([AD0ECBD6AB373D86:FEB789664926A87C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration(IndexSizeTriggerTest.java:404)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration

Error Message:
did not finish processing in time

Stack Trace:
java.lang.AssertionError: did not finish processing in time
at 

[jira] [Commented] (SOLR-12284) WordBreakSolrSpellChecker incorrectly adds parenthesis when breaking words

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458881#comment-16458881
 ] 

ASF subversion and git services commented on SOLR-12284:


Commit d92b891f95f3185e5d23dc89a23831e633ab64e5 in lucene-solr's branch 
refs/heads/master from jdyer1
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d92b891 ]

SOLR-12284: Stop adding parenthesis to word-break suggestions, unless query 
uses boolean operators.


> WordBreakSolrSpellChecker incorrectly adds parenthesis when breaking words
> --
>
> Key: SOLR-12284
> URL: https://issues.apache.org/jira/browse/SOLR-12284
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: spellchecker
>Affects Versions: 7.3
>Reporter: James Dyer
>Assignee: James Dyer
>Priority: Minor
> Attachments: SOLR-12284.patch
>
>
> When using WordBreakSolrSpellChecker to break single words into multiple, the 
> collation queries include parenthesis around the original term.  In some 
> cases, this causes required terms to become optional and users get spurious 
> nonsensical collation results.
> For instance, if I search: +eward +smith 
> ...If +ward +smith is a match, it might give a collation like: (+e +ward) 
> +smith
> ...This requires either the "e" or the "ward" to exist, but not both.  But 
> users are more likely to want both terms to be required, so it would be 
> better if it was not adding parenthesis.
> This might be the cause of SOLR-5995 and [this SO 
> issue|https://stackoverflow.com/questions/23849747/solr-wordbreak-spellchecker-breaking-words-into-letters-excessive-breaking]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12286) Wrap log instances in "if (LOG.isLevelEnabled) { log... }

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458793#comment-16458793
 ] 

Shawn Heisey commented on SOLR-12286:
-

Generally speaking, I'm all for using SLF4J parameterization where we can.  It 
makes the code much cleaner.  The fact that it also has performance benefits is 
a nice bonus.

Building messages with slow operations like string concatenation is not 
necessarily evil -- for things that get logged once or infrequently, the time 
difference is minimal.  But it's probably not a good general pattern to use.

It's interesting that the page Mark linked to advises against using the 
"isLevelEnabled" check in most cases.  I would tend to agree with this for INFO 
and higher levels, but it did seem like a good idea to me to leave the check 
for DEBUG and lower.  We can follow their advice, and only do that wrapping 
when the log message or its parameters are built by code that might run slowly.

On the forbidden API stuff Erick mentioned: How about we create alternate and 
much more sensitive build targets for things like forbidden APIs?  Build 
targets that would have to be manually called.  Detecting string usage that we 
consider sub-optimal could help us write better code, but until the codebase is 
sufficiently cleansed, I think flagging those usages with precommit would hold 
up general development.


> Wrap log instances in "if (LOG.isLevelEnabled) { log... }
> -
>
> Key: SOLR-12286
> URL: https://issues.apache.org/jira/browse/SOLR-12286
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I've been playing around with the question "are objects generated when 
> logging if I use pattern." and "it depends" (tm). I'll attach a test 
> program for anyone to use. Attach VisualVM to it and sample memory when 
> various patterns are enabled.
> The nub of it is this: with the log level set at ERROR, no messages from any 
> of these are printed, yet the number of objects created is vastly different:
> {code}
>   while (true) {
>   // "test" is an instance of a class with an overridden toString() 
> method.
>   // These generate a zillion spurious objects.
>   logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
>   logger.info("This is a test {}", rand.nextInt());
>   logger.info("This is really bad" + test);
>   
>   // These don't generate spurious objects
>   logger.info("This is a test {} {}", test, "whatever");
>   logger.info("This is really bad" + "whatever");
>   }
> {code}
> Simply generating a new random number doesn't create zillions of objects.
> I don't particularly care _why_ the differences exist, the take-away is that 
> if we simply establish the rule "wrap log.level() messages with if..." then 
> we'll avoid the problems altogether. That's _much_ easier to both remember 
> and enforce than trying to understand and remember when pattern A is 
> acceptable and pattern B is not.
> Maybe we could even make this a precommit failure?
> Solr has enough "interesting" things happen re: GC that we don't need to 
> needlessly add to GC pressure.
> Parameterizing is still a good idea as in SOLR-10415 (I'll link)
> Here's the full program, there's not a lot of sophistication here:
> {code}
> import org.apache.logging.log4j.Level;
> import org.apache.logging.log4j.Logger;
> import org.apache.logging.log4j.LogManager;
> import org.apache.logging.log4j.core.LoggerContext;
> import org.apache.logging.log4j.core.config.Configuration;
> import org.apache.logging.log4j.core.config.LoggerConfig;
> import java.util.Random;
> public class Log4J2Test {
>   // Define a static logger variable so that it references the
>   // Logger instance named "MyApp".
>   private static final Logger logger = LogManager.getLogger(Log4J2Test.class);
>   static Random rand = new Random();
>   public static void main(final String... args) {
> // Set up a simple configuration that logs on the console.
> logger.trace("Entering application.");
> LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
> Configuration config = ctx.getConfiguration();
> LoggerConfig loggerConfig = 
> config.getLoggerConfig(LogManager.ROOT_LOGGER_NAME);
> loggerConfig.setLevel(Level.ERROR);
> ctx.updateLoggers();
> Test test = new Test();
> logger.error("Ensure something comes out.");
> while (true) {
>   if (logger.isInfoEnabled()) {
> // These generate a zillion objects.
> logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
> logger.info("This is a test {}", rand.nextInt());
> 

[jira] [Commented] (SOLR-12286) Wrap log instances in "if (LOG.isLevelEnabled) { log... }

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458777#comment-16458777
 ] 

David Smiley commented on SOLR-12286:
-

bq. Now maybe the right thing to do is enhance the forbidden APIs to fail the 
known bad logging patterns: any log.whatever calls with plus signs outside of 
quotes and any log.whatever that has a toString() in it etc. That'd probably 
take care of 99+% of my concerns and the remaining fraction of a percent isn't 
worth the trouble.

+1 that'd be great but it appears that forbidden-apis only supports disallowing 
particular method calls, not general bytecode pattern detection.  Am I right 
[~thetaphi]?

> Wrap log instances in "if (LOG.isLevelEnabled) { log... }
> -
>
> Key: SOLR-12286
> URL: https://issues.apache.org/jira/browse/SOLR-12286
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I've been playing around with the question "are objects generated when 
> logging if I use pattern." and "it depends" (tm). I'll attach a test 
> program for anyone to use. Attach VisualVM to it and sample memory when 
> various patterns are enabled.
> The nub of it is this: with the log level set at ERROR, no messages from any 
> of these are printed, yet the number of objects created is vastly different:
> {code}
>   while (true) {
>   // "test" is an instance of a class with an overridden toString() 
> method.
>   // These generate a zillion spurious objects.
>   logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
>   logger.info("This is a test {}", rand.nextInt());
>   logger.info("This is really bad" + test);
>   
>   // These don't generate spurious objects
>   logger.info("This is a test {} {}", test, "whatever");
>   logger.info("This is really bad" + "whatever");
>   }
> {code}
> Simply generating a new random number doesn't create zillions of objects.
> I don't particularly care _why_ the differences exist, the take-away is that 
> if we simply establish the rule "wrap log.level() messages with if..." then 
> we'll avoid the problems altogether. That's _much_ easier to both remember 
> and enforce than trying to understand and remember when pattern A is 
> acceptable and pattern B is not.
> Maybe we could even make this a precommit failure?
> Solr has enough "interesting" things happen re: GC that we don't need to 
> needlessly add to GC pressure.
> Parameterizing is still a good idea as in SOLR-10415 (I'll link)
> Here's the full program, there's not a lot of sophistication here:
> {code}
> import org.apache.logging.log4j.Level;
> import org.apache.logging.log4j.Logger;
> import org.apache.logging.log4j.LogManager;
> import org.apache.logging.log4j.core.LoggerContext;
> import org.apache.logging.log4j.core.config.Configuration;
> import org.apache.logging.log4j.core.config.LoggerConfig;
> import java.util.Random;
> public class Log4J2Test {
>   // Define a static logger variable so that it references the
>   // Logger instance named "MyApp".
>   private static final Logger logger = LogManager.getLogger(Log4J2Test.class);
>   static Random rand = new Random();
>   public static void main(final String... args) {
> // Set up a simple configuration that logs on the console.
> logger.trace("Entering application.");
> LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
> Configuration config = ctx.getConfiguration();
> LoggerConfig loggerConfig = 
> config.getLoggerConfig(LogManager.ROOT_LOGGER_NAME);
> loggerConfig.setLevel(Level.ERROR);
> ctx.updateLoggers();
> Test test = new Test();
> logger.error("Ensure something comes out.");
> while (true) {
>   if (logger.isInfoEnabled()) {
> // These generate a zillion objects.
> logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
> logger.info("This is a test {}", rand.nextInt());
> logger.info("This is really bad" + test);
> // These generates no spurious objects
> logger.info("This is a test {} {}", test, "whatever");
> logger.info("This is really bad" + "whatever");
>   }
> }
>   }
> }
> class Test {
>   static Random rand = new Random();
>   public String toString() {
> return "This is some random string" + rand.nextInt();
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9.0.4) - Build # 1826 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1826/
Java: 64bit/jdk-9.0.4 -XX:+UseCompressedOops -XX:+UseParallelGC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestSQLHandler

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.TestSQLHandler: 
1) Thread[id=2686, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestSQLHandler] at 
java.base@9.0.4/java.lang.Thread.sleep(Native Method) at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 at java.base@9.0.4/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.handler.TestSQLHandler: 
   1) Thread[id=2686, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestSQLHandler]
at java.base@9.0.4/java.lang.Thread.sleep(Native Method)
at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@9.0.4/java.lang.Thread.run(Thread.java:844)
at __randomizedtesting.SeedInfo.seed([BBCD5A5C0DEC9BC8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestSQLHandler

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.TestSQLHandler: 
1) Thread[id=2715, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestSQLHandler] at 
java.base@9.0.4/java.lang.Thread.sleep(Native Method) at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 at java.base@9.0.4/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.handler.TestSQLHandler: 
   1) Thread[id=2715, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestSQLHandler]
at java.base@9.0.4/java.lang.Thread.sleep(Native Method)
at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@9.0.4/java.lang.Thread.run(Thread.java:844)
at __randomizedtesting.SeedInfo.seed([BBCD5A5C0DEC9BC8]:0)


FAILED:  org.apache.solr.handler.TestSQLHandler.doTest

Error Message:
--> https://127.0.0.1:45465/collection1_shard2_replica_n41:Failed to execute 
sqlQuery 'select id, field_i, str_s from collection1 where (text='()' OR 
text='') AND text='' order by field_i desc' against JDBC connection 
'jdbc:calcitesolr:'. Error while executing SQL "select id, field_i, str_s from 
collection1 where (text='()' OR text='') AND text='' order by 
field_i desc": java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: --> 
https://127.0.0.1:45465/collection1_shard2_replica_n41/:id must have DocValues 
to use this feature.

Stack Trace:
java.io.IOException: --> 
https://127.0.0.1:45465/collection1_shard2_replica_n41:Failed to execute 
sqlQuery 'select id, field_i, str_s from collection1 where (text='()' OR 
text='') AND text='' order by field_i desc' against JDBC connection 
'jdbc:calcitesolr:'.
Error while executing SQL "select id, field_i, str_s from collection1 where 
(text='()' OR text='') AND text='' order by field_i desc": 
java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: --> 
https://127.0.0.1:45465/collection1_shard2_replica_n41/:id must have DocValues 
to use this feature.
at 
__randomizedtesting.SeedInfo.seed([BBCD5A5C0DEC9BC8:1C89E2F860578871]:0)
at 
org.apache.solr.client.solrj.io.stream.SolrStream.read(SolrStream.java:222)
at 
org.apache.solr.handler.TestSQLHandler.getTuples(TestSQLHandler.java:2522)
at 
org.apache.solr.handler.TestSQLHandler.testBasicSelect(TestSQLHandler.java:124)
at org.apache.solr.handler.TestSQLHandler.doTest(TestSQLHandler.java:82)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at 

Re: Team please help

[Greg Solovyev]

Sujeet, what do you mean by migrating? E.g., are you moving your data from
Cloudera CDH to Azure HDI? Are migrating your application code written on
top of Cloudera CDH to run on top of Azure HDI? As far as I know, Azure HDI
does not include Solr, so if your application on top of Cloudera CDH is
using Solr, it won't run on HDI.
Greg

On Sat, Apr 28, 2018 at 5:45 PM Sujeet Singh 
wrote:

> Adding Dev
>
>
>
> *From:* Sujeet Singh
> *Sent:* Sunday, April 29, 2018 12:14 AM
> *To:* 'solr-u...@lucene.apache.org'
> *Subject:* Team please help
>
>
>
> Team I am facing an issue right now. I am working ahead to migrate
> cloudera to HDI Azure. Now cloudera has Solr implementation and using the
> below jar
>
> search-mr-1.0.0-cdh5.7.0-job.jar
> org.apache.solr.hadoop.MapReduceIndexerTool
>
>
>
> While looking into all option I found “solr-map-reduce-4.9.0.jar” and
> tried using it with class “org.apache.solr.hadoop.MapReduceIndexerTool”. I
> tried adding lib details in solrconfig.xml but did not worked . Getting
> error
>
> “Caused by: java.lang.ClassNotFoundException:
> org.apache.solr.morphlines.solr.DocumentLoader”
>
>
>
> Please let me know the right way to use MapReduceIndexerTool class.
>
>
>
> Regards,
> --
>
> *Sujeet Singh* | Sr. Software Analyst | cloudmoyo | *E.*
> sujeet.si...@cloudmoyo.com | *M.* +91 9860586055 <+91%2098605%2086055>
>
> [image: CloudMoyo Logo]
> 
> [image:
> https://icertisportalstorage.blob.core.windows.net/siteasset/icon-linkedin.png]
> [image:
> https://icertisportalstorage.blob.core.windows.net/siteasset/icon-fb.png]
> [image:
> https://icertisportalstorage.blob.core.windows.net/siteasset/icon-twitter.png]
> 
> www.cloudmoyo.com
>
>
>

[jira] [Commented] (SOLR-12286) Wrap log instances in "if (LOG.isLevelEnabled) { log... }

[Michael Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458734#comment-16458734
 ] 

Michael Braun commented on SOLR-12286:
--

[~erickerickson] I think the int example depends also on what number it is / if 
it goes beyond the Integer cache bounds for the second case. 

> Wrap log instances in "if (LOG.isLevelEnabled) { log... }
> -
>
> Key: SOLR-12286
> URL: https://issues.apache.org/jira/browse/SOLR-12286
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I've been playing around with the question "are objects generated when 
> logging if I use pattern." and "it depends" (tm). I'll attach a test 
> program for anyone to use. Attach VisualVM to it and sample memory when 
> various patterns are enabled.
> The nub of it is this: with the log level set at ERROR, no messages from any 
> of these are printed, yet the number of objects created is vastly different:
> {code}
>   while (true) {
>   // "test" is an instance of a class with an overridden toString() 
> method.
>   // These generate a zillion spurious objects.
>   logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
>   logger.info("This is a test {}", rand.nextInt());
>   logger.info("This is really bad" + test);
>   
>   // These don't generate spurious objects
>   logger.info("This is a test {} {}", test, "whatever");
>   logger.info("This is really bad" + "whatever");
>   }
> {code}
> Simply generating a new random number doesn't create zillions of objects.
> I don't particularly care _why_ the differences exist, the take-away is that 
> if we simply establish the rule "wrap log.level() messages with if..." then 
> we'll avoid the problems altogether. That's _much_ easier to both remember 
> and enforce than trying to understand and remember when pattern A is 
> acceptable and pattern B is not.
> Maybe we could even make this a precommit failure?
> Solr has enough "interesting" things happen re: GC that we don't need to 
> needlessly add to GC pressure.
> Parameterizing is still a good idea as in SOLR-10415 (I'll link)
> Here's the full program, there's not a lot of sophistication here:
> {code}
> import org.apache.logging.log4j.Level;
> import org.apache.logging.log4j.Logger;
> import org.apache.logging.log4j.LogManager;
> import org.apache.logging.log4j.core.LoggerContext;
> import org.apache.logging.log4j.core.config.Configuration;
> import org.apache.logging.log4j.core.config.LoggerConfig;
> import java.util.Random;
> public class Log4J2Test {
>   // Define a static logger variable so that it references the
>   // Logger instance named "MyApp".
>   private static final Logger logger = LogManager.getLogger(Log4J2Test.class);
>   static Random rand = new Random();
>   public static void main(final String... args) {
> // Set up a simple configuration that logs on the console.
> logger.trace("Entering application.");
> LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
> Configuration config = ctx.getConfiguration();
> LoggerConfig loggerConfig = 
> config.getLoggerConfig(LogManager.ROOT_LOGGER_NAME);
> loggerConfig.setLevel(Level.ERROR);
> ctx.updateLoggers();
> Test test = new Test();
> logger.error("Ensure something comes out.");
> while (true) {
>   if (logger.isInfoEnabled()) {
> // These generate a zillion objects.
> logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
> logger.info("This is a test {}", rand.nextInt());
> logger.info("This is really bad" + test);
> // These generates no spurious objects
> logger.info("This is a test {} {}", test, "whatever");
> logger.info("This is really bad" + "whatever");
>   }
> }
>   }
> }
> class Test {
>   static Random rand = new Random();
>   public String toString() {
> return "This is some random string" + rand.nextInt();
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12286) Wrap log instances in "if (LOG.isLevelEnabled) { log... }

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458730#comment-16458730
 ] 

Erick Erickson commented on SOLR-12286:
---

Here 's where keeping track of exactly what log message does or does not 
generate objects is a pain. 

"eoe" is just an int. 

This one generates useless objects that are then GCd
logger.info("silly string {} ", eoe++);

This one does not
logger.info("silly string {} ", eoe);



> Wrap log instances in "if (LOG.isLevelEnabled) { log... }
> -
>
> Key: SOLR-12286
> URL: https://issues.apache.org/jira/browse/SOLR-12286
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I've been playing around with the question "are objects generated when 
> logging if I use pattern." and "it depends" (tm). I'll attach a test 
> program for anyone to use. Attach VisualVM to it and sample memory when 
> various patterns are enabled.
> The nub of it is this: with the log level set at ERROR, no messages from any 
> of these are printed, yet the number of objects created is vastly different:
> {code}
>   while (true) {
>   // "test" is an instance of a class with an overridden toString() 
> method.
>   // These generate a zillion spurious objects.
>   logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
>   logger.info("This is a test {}", rand.nextInt());
>   logger.info("This is really bad" + test);
>   
>   // These don't generate spurious objects
>   logger.info("This is a test {} {}", test, "whatever");
>   logger.info("This is really bad" + "whatever");
>   }
> {code}
> Simply generating a new random number doesn't create zillions of objects.
> I don't particularly care _why_ the differences exist, the take-away is that 
> if we simply establish the rule "wrap log.level() messages with if..." then 
> we'll avoid the problems altogether. That's _much_ easier to both remember 
> and enforce than trying to understand and remember when pattern A is 
> acceptable and pattern B is not.
> Maybe we could even make this a precommit failure?
> Solr has enough "interesting" things happen re: GC that we don't need to 
> needlessly add to GC pressure.
> Parameterizing is still a good idea as in SOLR-10415 (I'll link)
> Here's the full program, there's not a lot of sophistication here:
> {code}
> import org.apache.logging.log4j.Level;
> import org.apache.logging.log4j.Logger;
> import org.apache.logging.log4j.LogManager;
> import org.apache.logging.log4j.core.LoggerContext;
> import org.apache.logging.log4j.core.config.Configuration;
> import org.apache.logging.log4j.core.config.LoggerConfig;
> import java.util.Random;
> public class Log4J2Test {
>   // Define a static logger variable so that it references the
>   // Logger instance named "MyApp".
>   private static final Logger logger = LogManager.getLogger(Log4J2Test.class);
>   static Random rand = new Random();
>   public static void main(final String... args) {
> // Set up a simple configuration that logs on the console.
> logger.trace("Entering application.");
> LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
> Configuration config = ctx.getConfiguration();
> LoggerConfig loggerConfig = 
> config.getLoggerConfig(LogManager.ROOT_LOGGER_NAME);
> loggerConfig.setLevel(Level.ERROR);
> ctx.updateLoggers();
> Test test = new Test();
> logger.error("Ensure something comes out.");
> while (true) {
>   if (logger.isInfoEnabled()) {
> // These generate a zillion objects.
> logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
> logger.info("This is a test {}", rand.nextInt());
> logger.info("This is really bad" + test);
> // These generates no spurious objects
> logger.info("This is a test {} {}", test, "whatever");
> logger.info("This is really bad" + "whatever");
>   }
> }
>   }
> }
> class Test {
>   static Random rand = new Random();
>   public String toString() {
> return "This is some random string" + rand.nextInt();
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12293) Updates need to use their own connection pool to maintain connection reuse and prevent spurious recoveries.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458711#comment-16458711
 ] 

Mark Miller commented on SOLR-12293:


Patch attached to fix this. A connection manager and thread pool only for 
sending literal updates.

> Updates need to use their own connection pool to maintain connection reuse 
> and prevent spurious recoveries.
> ---
>
> Key: SOLR-12293
> URL: https://issues.apache.org/jira/browse/SOLR-12293
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Attachments: SOLR-12293.patch
>
>
> Currently the pool is shared too broadly - for example during replication we 
> don't guarantee we read the full streams when downloading index files and we 
> don't necessarily want to, emptying the stream for a huge file due to error 
> or abort is too expensive. We can't have these connections pollute the update 
> connection pool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12293) Updates need to use their own connection pool to maintain connection reuse and prevent spurious recoveries.

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-12293:
---
Attachment: SOLR-12293.patch

> Updates need to use their own connection pool to maintain connection reuse 
> and prevent spurious recoveries.
> ---
>
> Key: SOLR-12293
> URL: https://issues.apache.org/jira/browse/SOLR-12293
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Attachments: SOLR-12293.patch
>
>
> Currently the pool is shared too broadly - for example during replication we 
> don't guarantee we read the full streams when downloading index files and we 
> don't necessarily want to, emptying the stream for a huge file due to error 
> or abort is too expensive. We can't have these connections pollute the update 
> connection pool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12286) Wrap log instances in "if (LOG.isLevelEnabled) { log... }

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458706#comment-16458706
 ] 

Erick Erickson commented on SOLR-12286:
---

bq.  Short lived objects are dealt with efficiently by the garbage collector.

My turn for a "visceral -1" ;) I vastly prefer not generating useless work 
regardless of how efficient cleanup is afterwards. 

bq. BTW rand.nextInt() has side-effects; most code you'd put in a log statement 
does not.  This is likely a factor.

Well, that's largely my point. It's not obvious at all that rand.nextInt() 
would have this side effect on logging, at least not to me. What about side 
effect N+1? That said I'll concede this one.

The parameterized versions of log messages do help a lot and should be used. 

And this pattern generates the string object and throws it away for fairly 
obvious reasons:
logger.info("This is a test {} {}", test.toString(), "whatever");

Peruse the current code and you'll see a bunch of the log.info("message" + 
"another message") patterns. I count 300 without even straining. My take is 
that we _haven't_ done a very good job of paying attention to this. Why do you 
think we'll do a better job in future? Who's going to police this?

Then there's also another 300 or so debug and trace level messages with the 
same pattern.

Admittedly the above 600 messages are simple greps, perhaps some of them _are_ 
wrapped in if statements.

This all came to light with some work with JFR.

Now maybe the right thing to do is enhance the forbidden APIs to fail the known 
bad logging patterns: any log.whatever calls with plus signs outside of quotes 
and any log.whatever that has a toString() in it etc. That'd probably take care 
of 99+% of my concerns and the remaining fraction of a percent isn't worth the 
trouble.

If that's the consensus we can probably close this or change it to "add 
forbidden API check for log messages that create spurious objects". It still 
requires that we parameterize all the log.whatever calls.

> Wrap log instances in "if (LOG.isLevelEnabled) { log... }
> -
>
> Key: SOLR-12286
> URL: https://issues.apache.org/jira/browse/SOLR-12286
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I've been playing around with the question "are objects generated when 
> logging if I use pattern." and "it depends" (tm). I'll attach a test 
> program for anyone to use. Attach VisualVM to it and sample memory when 
> various patterns are enabled.
> The nub of it is this: with the log level set at ERROR, no messages from any 
> of these are printed, yet the number of objects created is vastly different:
> {code}
>   while (true) {
>   // "test" is an instance of a class with an overridden toString() 
> method.
>   // These generate a zillion spurious objects.
>   logger.info("This is a test {} {} {}", test, rand.nextInt(), 
> "whatever");
>   logger.info("This is a test {}", rand.nextInt());
>   logger.info("This is really bad" + test);
>   
>   // These don't generate spurious objects
>   logger.info("This is a test {} {}", test, "whatever");
>   logger.info("This is really bad" + "whatever");
>   }
> {code}
> Simply generating a new random number doesn't create zillions of objects.
> I don't particularly care _why_ the differences exist, the take-away is that 
> if we simply establish the rule "wrap log.level() messages with if..." then 
> we'll avoid the problems altogether. That's _much_ easier to both remember 
> and enforce than trying to understand and remember when pattern A is 
> acceptable and pattern B is not.
> Maybe we could even make this a precommit failure?
> Solr has enough "interesting" things happen re: GC that we don't need to 
> needlessly add to GC pressure.
> Parameterizing is still a good idea as in SOLR-10415 (I'll link)
> Here's the full program, there's not a lot of sophistication here:
> {code}
> import org.apache.logging.log4j.Level;
> import org.apache.logging.log4j.Logger;
> import org.apache.logging.log4j.LogManager;
> import org.apache.logging.log4j.core.LoggerContext;
> import org.apache.logging.log4j.core.config.Configuration;
> import org.apache.logging.log4j.core.config.LoggerConfig;
> import java.util.Random;
> public class Log4J2Test {
>   // Define a static logger variable so that it references the
>   // Logger instance named "MyApp".
>   private static final Logger logger = LogManager.getLogger(Log4J2Test.class);
>   static Random rand = new Random();
>   public static void main(final String... args) {
> // Set up a simple configuration that logs on the console.
> logger.trace("Entering 

[jira] [Commented] (SOLR-12278) Ignore very large document on indexing

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458678#comment-16458678
 ] 

David Smiley commented on SOLR-12278:
-

Me:
bq. imagine if FastInputStream had methods to track the overall bytes written 

it already does -- position().  position() might need to move to 
DataInputInputStream or probably better -- JavaBinCodec could cast (but not 
complain if it's not of type FastInputStream).

This approach would also provide a way to bail out earlier -- before we read 
stuff beyond the limit into heap.  That would take more LOC though.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12290) Do not close any servlet streams and improve our servlet stream closing prevention code for users and devs.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458676#comment-16458676
 ] 

Mark Miller commented on SOLR-12290:


As I've been working on this, I realized that we don't ensure reading full 
streams when we get files for replication, whether we close the streams early 
or not.

I filed SOLR-12293. We need to isolate our update connection pool from these 
other random uses.

> Do not close any servlet streams and improve our servlet stream closing 
> prevention code for users and devs.
> ---
>
> Key: SOLR-12290
> URL: https://issues.apache.org/jira/browse/SOLR-12290
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-12290.patch, SOLR-12290.patch, SOLR-12290.patch, 
> SOLR-12290.patch
>
>
> Original Summary:
> When you fetch a file for replication we close the request output stream 
> after writing the file which ruins the connection for reuse.
> We can't close response output streams, we need to reuse these connections. 
> If we do close them, clients are hit with connection problems when they try 
> and reuse the connection from their pool.
> New Summary:
> At some point the above was addressed during refactoring. We should remove 
> these neutered closes and review our close shield code.
> If you are here to track down why this is done:
> Connection reuse requires that we read all streams and do not close them - 
> instead the container itself must manage request and response streams. If we 
> allow them to be closed, not only do we lose some connection reuse, but we 
> can cause spurious client errors that can cause expensive recoveries for no 
> reason. The spec allows us to count on the container to manage streams. It's 
> our job simply to not close them and to always read them fully, from client 
> and server. 
> Java itself can help with always reading the streams fully up to some small 
> default amount of unread stream slack, but that is very dangerous to count 
> on, so we always manually eat up anything on the streams our normal logic 
> ends up not reading for whatever reason.
> We also cannot call abort without ruining the connection or sendError. These 
> should be options of very last resort (requiring a blood sacrifice) or when 
> shutting down.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8284) Add MultiTermsIntervalsSource

[Matt Weber (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458673#comment-16458673
 ] 

Matt Weber commented on LUCENE-8284:


[~romseygeek] [~jimczi]

Since these expand terms per-segment the terms are not available when creating 
the {{IntervalWeight}} and thus result in a null {{simScorer}} if these are the 
only sources.  I currently picked using a constant {{1.0f}} in this case.  Not 
sure if this is the best approach or not.

> Add MultiTermsIntervalsSource
> -
>
> Key: LUCENE-8284
> URL: https://issues.apache.org/jira/browse/LUCENE-8284
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Matt Weber
>Priority: Minor
> Attachments: LUCENE-8284.patch
>
>
> Add support for creating an {{IntervalsSource}} from multi-term expansions 
> such as wildcards, regular expressions, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8284) Add MultiTermsIntervalsSource

[Matt Weber (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Weber updated LUCENE-8284:
---
Attachment: LUCENE-8284.patch

> Add MultiTermsIntervalsSource
> -
>
> Key: LUCENE-8284
> URL: https://issues.apache.org/jira/browse/LUCENE-8284
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Matt Weber
>Priority: Minor
> Attachments: LUCENE-8284.patch
>
>
> Add support for creating an {{IntervalsSource}} from multi-term expansions 
> such as wildcards, regular expressions, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8284) Add MultiTermsIntervalsSource

[Matt Weber (JIRA)]

Matt Weber created LUCENE-8284:
--

 Summary: Add MultiTermsIntervalsSource
 Key: LUCENE-8284
 URL: https://issues.apache.org/jira/browse/LUCENE-8284
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Matt Weber


Add support for creating an {{IntervalsSource}} from multi-term expansions such 
as wildcards, regular expressions, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1830 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1830/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

15 tests failed.
FAILED:  org.apache.solr.cloud.AssignBackwardCompatibilityTest.test

Error Message:
Expected 4 active replicas null Live Nodes: [127.0.0.1:40241_solr, 
127.0.0.1:45716_solr, 127.0.0.1:48502_solr, 127.0.0.1:56544_solr] Last 
available state: null

Stack Trace:
java.lang.AssertionError: Expected 4 active replicas
null
Live Nodes: [127.0.0.1:40241_solr, 127.0.0.1:45716_solr, 127.0.0.1:48502_solr, 
127.0.0.1:56544_solr]
Last available state: null
at 
__randomizedtesting.SeedInfo.seed([7EB65480721DD8AB:F6E26B5ADCE1B553]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:278)
at 
org.apache.solr.cloud.AssignBackwardCompatibilityTest.test(AssignBackwardCompatibilityTest.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 

[jira] [Commented] (SOLR-12278) Ignore very large document on indexing

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458656#comment-16458656
 ] 

David Smiley commented on SOLR-12278:
-

{quote}If you want to reject large requests, configure Jetty to reject large 
requests.
{quote}
If it's streaming (Content-Length header is \-1) then Jetty won't know. And 
even for gigabytes of streaming data through it's not necessarily a problem at 
all -- it could be lots of reasonably sized documents.

A few hours ago I wondered if we could solve this in a lower level more 
efficient way.  It's a shame to measure this stuff after it's been deserialized 
when Solr is in fact reading the bytes and knows how many bytes were read when 
the SolrInputDocument finishes.  To this end, imagine if FastInputStream had 
methods to track the overall bytes written and then JavaBinCodec could use that 
information.  Granted an on-the-wire bytes != Java heap bytes... and of course 
what if someone sends the data in CSV or XML or JSON... but nonetheless it 
could be a bonus feature of JavaBin that would be exceptionally cheap to 
calculate and very little LOC.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8998) JSON Facet API child roll-ups

[Mikhail Khludnev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Khludnev updated SOLR-8998:
---
Description: 
The JSON Facet API currently has the ability to map between parents and 
children ( see http://yonik.com/solr-nested-objects/ )
This issue is about adding a true rollup ability where parents would take on 
derived values from their children.  The most important part (and the most 
difficult part) will be the external API.

[~mkhludnev] says
{quote}
The bottom line is to introduce {{uniqueBlock(\_root_)}} aggregation, which is 
supposed to be faster than {{unique(\_root_)}} and purposed for blocked index 
only. For now it it supports singlevalue string fields, docValues usually make 
sense.   
{quote}


  was:
The JSON Facet API currently has the ability to map between parents and 
children ( see http://yonik.com/solr-nested-objects/ )
This issue is about adding a true rollup ability where parents would take on 
derived values from their children.  The most important part (and the most 
difficult part) will be the external API.



> JSON Facet API child roll-ups
> -
>
> Key: SOLR-8998
> URL: https://issues.apache.org/jira/browse/SOLR-8998
> Project: Solr
>  Issue Type: New Feature
>  Components: Facet Module
>Reporter: Yonik Seeley
>Priority: Major
> Attachments: SOLR-8998.patch, SOLR-8998.patch, SOLR-8998.patch, 
> SOLR_8998.patch, SOLR_8998.patch, SOLR_8998.patch
>
>
> The JSON Facet API currently has the ability to map between parents and 
> children ( see http://yonik.com/solr-nested-objects/ )
> This issue is about adding a true rollup ability where parents would take on 
> derived values from their children.  The most important part (and the most 
> difficult part) will be the external API.
> [~mkhludnev] says
> {quote}
> The bottom line is to introduce {{uniqueBlock(\_root_)}} aggregation, which 
> is supposed to be faster than {{unique(\_root_)}} and purposed for blocked 
> index only. For now it it supports singlevalue string fields, docValues 
> usually make sense.   
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12294) System collection - Lazy loading mechanism not working for custom UpdateProcessors

[Johannes Brucher (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Brucher updated SOLR-12294:

Description: 
Hi all,

I'm facing an issue regarding custom code inside a .system-collection and 
starting up a Solr Cloud cluster.

I thought, like its stated in the documentation, that in case using the .system 
collection custom code is lazy loaded, because it can happen that a collection 
that uses custom code is initialized before the .system collection is up and 
running.

I did all the necessary configuration and while debugging, I can see that the 
custom code is wrapped via a PluginBag$LazyPluginHolder. So far its seems good, 
but I still get Exceptions when starting the Solr Cloud with the following 
errors:

SolrException: Blob loading failed: .no active replica available for .system 
collection...

In my case I'm using custom code for a couple of UpdateProcessors. So it seems, 
that this lazy mechanism is not working well for UpdateProcessors.

Inside the calzz LazyPluginHolder the comment says:

"A class that loads plugins Lazily. When the get() method is invoked the Plugin 
is initialized and returned."

When a core is initialized and you have a custom UpdateProcessor, the 
get-method is invoked directly and the lazy loading mechanism tries to get the 
custom class from the MemClassLoader, but in most scenarios the system 
collection is not up and the above Exception is thrown...

So maybe it’s the case that for UpdateProcessors while initializing a core, the 
routine is not implemented optimal for the lazy loading mechanism?

 

Here are the steps to reproduce the issue:
 # Unpack solr 7.3.0
 1.1 Add SOLR_OPTS="$SOLR_OPTS -Denable.runtime.lib=true" to solr.in.sh
 1.2 Start Solr with -c option
 # Setup .system collection:
 2.1 Upload custom test jar --> curl -X POST -H 'Content-Type: 
application/octet-stream' --data-binary @/update-processor-0.0.1-SNAPSHOT.jar http:///solr/.system/blob/test_blob
 2.2 Alter maxShardsPerNode --> 
.../admin/collections?action=MODIFYCOLLECTION=.system=2
 2.2 Add Replica to .system collection --> 
.../admin/collections?action=ADDREPLICA=.system=shard1
 # Setup test collection:
 3.1 Upload test conf to ZK --> ./zkcli.sh -zkhost  -cmd upconfig 
-confdir  -confname test_conf
 3.2 Create a test1 collection with commented UP-chain inside solrconfig.xml 
via Admin UI
 3.3 Add blob to test collection --> curl http:///solr/test1/config 
-H 'Content-type:application/json' -d '\{"add-runtimelib": { 
"name":"test_blob", "version":1 }}'
 3.4 Uncomment the UP-chain and upload test-conf again --> ./zkcli.sh -zkhost 
 -cmd upconfig -confdir  -confname test_conf
 3.5 Reload test1 collection
 3.6 Everything should work as expected now (no erros are shown)
 # Restart SOLR
 4.1 Now you can see: SolrException: Blob loading failed: No active replica 
available for .system collection

Expected: The lazy loading mechanism should work for UpdateProcessors inside 
core init routine, but it isn't due to above Exception.

Sometimes you are lucky and the test1 collection will be initialize after the 
.system collection. But in ~90% of the time this isn't the case...

Let me know if you need further details here,

 

Johannes

  was:
Hi all,

I'm facing an issue regarding custom code inside a .system-collection and 
starting up a Solr Cloud cluster.

I thought, like its stated in the documentation, that in case using the .system 
collection custom code is lazy loaded, because it can happen that a collection 
that uses custom code is initialized before the .system collection is up and 
running.

I did all the necessary configuration and while debugging, I can see that the 
custom code is wrapped via a PluginBag$LazyPluginHolder. So far its seems good, 
but I still get Exceptions when starting the Solr Cloud with the following 
errors:

SolrException: Blob loading failed: .no active replica available for .system 
collection...

In my case I'm using custom code for a couple of UpdateProcessors. So it seems, 
that this lazy mechanism is not working well for UpdateProcessors.

Inside the calzz LazyPluginHolder the comment says:

"A class that loads plugins Lazily. When the get() method is invoked the Plugin 
is initialized and returned."

When a core is initialized and you have a custom UpdateProcessor, the 
get-method is invoked directly and the lazy loading mechanism tries to get the 
custom class from the MemClassLoader, but in most scenarios the system 
collection is not up and the above Exception is thrown...

So maybe it’s the case that for UpdateProcessors while initializing a core, the 
routine is not implemented optimal for the lazy loading mechanism?

 

Here are the steps to reproduce the issue:
 # Unpack solr 7.3.0
 1.1 Add SOLR_OPTS="$SOLR_OPTS -Denable.runtime.lib=true" to solr.in.sh
 1.2 Start Solr with -c option
 # Setup .system collection:
 2.1 Upload custom test jar --> curl 

[jira] [Updated] (SOLR-12294) System collection - Lazy loading mechanism not working for custom UpdateProcessors

[Johannes Brucher (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Brucher updated SOLR-12294:

Attachment: no_active_replica_available.png

> System collection - Lazy loading mechanism not working for custom 
> UpdateProcessors
> --
>
> Key: SOLR-12294
> URL: https://issues.apache.org/jira/browse/SOLR-12294
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Affects Versions: 7.3
>Reporter: Johannes Brucher
>Priority: Critical
> Attachments: no_active_replica_available.png, schema.xml, 
> solrconfig.xml, update-processor-0.0.1-SNAPSHOT.jar
>
>
> Hi all,
> I'm facing an issue regarding custom code inside a .system-collection and 
> starting up a Solr Cloud cluster.
> I thought, like its stated in the documentation, that in case using the 
> .system collection custom code is lazy loaded, because it can happen that a 
> collection that uses custom code is initialized before the .system collection 
> is up and running.
> I did all the necessary configuration and while debugging, I can see that the 
> custom code is wrapped via a PluginBag$LazyPluginHolder. So far its seems 
> good, but I still get Exceptions when starting the Solr Cloud with the 
> following errors:
> SolrException: Blob loading failed: .no active replica available for .system 
> collection...
> In my case I'm using custom code for a couple of UpdateProcessors. So it 
> seems, that this lazy mechanism is not working well for UpdateProcessors.
> Inside the calzz LazyPluginHolder the comment says:
> "A class that loads plugins Lazily. When the get() method is invoked the 
> Plugin is initialized and returned."
> When a core is initialized and you have a custom UpdateProcessor, the 
> get-method is invoked directly and the lazy loading mechanism tries to get 
> the custom class from the MemClassLoader, but in most scenarios the system 
> collection is not up and the above Exception is thrown...
> So maybe it’s the case that for UpdateProcessors while initializing a core, 
> the routine is not implemented optimal for the lazy loading mechanism?
>  
> Here are the steps to reproduce the issue:
>  # Unpack solr 7.3.0
>  1.1 Add SOLR_OPTS="$SOLR_OPTS -Denable.runtime.lib=true" to solr.in.sh
>  1.2 Start Solr with -c option
>  # Setup .system collection:
>  2.1 Upload custom test jar --> curl -X POST -H 'Content-Type: 
> application/octet-stream' --data-binary @ jar>/update-processor-0.0.1-SNAPSHOT.jar http:// solr>/solr/.system/blob/test_blob
>  2.2 Alter maxShardsPerNode --> 
> .../admin/collections?action=MODIFYCOLLECTION=.system=2
>  2.2 Add Replica to .system collection --> 
> .../admin/collections?action=ADDREPLICA=.system=shard1
>  # Setup test collection:
>  3.1 Upload test conf to ZK --> ./zkcli.sh -zkhost  -cmd 
> upconfig -confdir  -confname test_conf
>  3.2 Create a test1 collection with commented UP-chain inside solrconfig.xml 
> via Admin UI
>  3.3 Add blob to test collection --> curl http:// Solr>/solr/test1/config -H 'Content-type:application/json' -d 
> '\{"add-runtimelib": { "name":"test_blob", "version":1 }}'
>  3.4 Uncomment the UP-chain and upload test-conf again --> ./zkcli.sh -zkhost 
>  -cmd upconfig -confdir  -confname test_conf
>  3.5 Reload test1 collection
>  3.6 Everything should work as expected now (no erros are shown)
>  # Restart SOLR
>  4.1 Now you can see: SolrException: Blob loading failed: No active replica 
> available for .system collection
> Expected: The lazy loading mechanism should work for UpdateProcessors inside 
> core init routine, but it isn't due to above Exception.
> Sometimes you are lucky and the test1 collection will be initialize after the 
> .system collection. But in ~90% of the time this isn't the case...
>  Let me know if you need further details here,
>  
> Johannes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12294) System collection - Lazy loading mechanism not working for custom UpdateProcessors

[Johannes Brucher (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Brucher updated SOLR-12294:

Attachment: update-processor-0.0.1-SNAPSHOT.jar

> System collection - Lazy loading mechanism not working for custom 
> UpdateProcessors
> --
>
> Key: SOLR-12294
> URL: https://issues.apache.org/jira/browse/SOLR-12294
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Affects Versions: 7.3
>Reporter: Johannes Brucher
>Priority: Critical
> Attachments: schema.xml, solrconfig.xml, 
> update-processor-0.0.1-SNAPSHOT.jar
>
>
> Hi all,
> I'm facing an issue regarding custom code inside a .system-collection and 
> starting up a Solr Cloud cluster.
> I thought, like its stated in the documentation, that in case using the 
> .system collection custom code is lazy loaded, because it can happen that a 
> collection that uses custom code is initialized before the .system collection 
> is up and running.
> I did all the necessary configuration and while debugging, I can see that the 
> custom code is wrapped via a PluginBag$LazyPluginHolder. So far its seems 
> good, but I still get Exceptions when starting the Solr Cloud with the 
> following errors:
> SolrException: Blob loading failed: .no active replica available for .system 
> collection...
> In my case I'm using custom code for a couple of UpdateProcessors. So it 
> seems, that this lazy mechanism is not working well for UpdateProcessors.
> Inside the calzz LazyPluginHolder the comment says:
> "A class that loads plugins Lazily. When the get() method is invoked the 
> Plugin is initialized and returned."
> When a core is initialized and you have a custom UpdateProcessor, the 
> get-method is invoked directly and the lazy loading mechanism tries to get 
> the custom class from the MemClassLoader, but in most scenarios the system 
> collection is not up and the above Exception is thrown...
> So maybe it’s the case that for UpdateProcessors while initializing a core, 
> the routine is not implemented optimal for the lazy loading mechanism?
>  
> Here are the steps to reproduce the issue:
>  # Unpack solr 7.3.0
>  1.1 Add SOLR_OPTS="$SOLR_OPTS -Denable.runtime.lib=true" to solr.in.sh
>  1.2 Start Solr with -c option
>  # Setup .system collection:
>  2.1 Upload custom test jar --> curl -X POST -H 'Content-Type: 
> application/octet-stream' --data-binary @ jar>/update-processor-0.0.1-SNAPSHOT.jar http:// solr>/solr/.system/blob/test_blob
>  2.2 Alter maxShardsPerNode --> 
> .../admin/collections?action=MODIFYCOLLECTION=.system=2
>  2.2 Add Replica to .system collection --> 
> .../admin/collections?action=ADDREPLICA=.system=shard1
>  # Setup test collection:
>  3.1 Upload test conf to ZK --> ./zkcli.sh -zkhost  -cmd 
> upconfig -confdir  -confname test_conf
>  3.2 Create a test1 collection with commented UP-chain inside solrconfig.xml 
> via Admin UI
>  3.3 Add blob to test collection --> curl http:// Solr>/solr/test1/config -H 'Content-type:application/json' -d 
> '\{"add-runtimelib": { "name":"test_blob", "version":1 }}'
>  3.4 Uncomment the UP-chain and upload test-conf again --> ./zkcli.sh -zkhost 
>  -cmd upconfig -confdir  -confname test_conf
>  3.5 Reload test1 collection
>  3.6 Everything should work as expected now (no erros are shown)
>  # Restart SOLR
>  4.1 Now you can see: SolrException: Blob loading failed: No active replica 
> available for .system collection
> Expected: The lazy loading mechanism should work for UpdateProcessors inside 
> core init routine, but it isn't due to above Exception.
> Sometimes you are lucky and the test1 collection will be initialize after the 
> .system collection. But in ~90% of the time this isn't the case...
>  Let me know if you need further details here,
>  
> Johannes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12278) Ignore very large document on indexing

[Walter Underwood (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458631#comment-16458631
 ] 

Walter Underwood commented on SOLR-12278:
-

Finding the size should be much quicker than the time already spent receiving 
the update request, parsing it, and creating the Solr input document. This is 
optimizing the wrong thing.

If you want to reject large requests, configure Jetty to reject large requests.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12294) System collection - Lazy loading mechanism not working for custom UpdateProcessors

[Johannes Brucher (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Brucher updated SOLR-12294:

Attachment: solrconfig.xml
schema.xml

> System collection - Lazy loading mechanism not working for custom 
> UpdateProcessors
> --
>
> Key: SOLR-12294
> URL: https://issues.apache.org/jira/browse/SOLR-12294
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Affects Versions: 7.3
>Reporter: Johannes Brucher
>Priority: Critical
> Attachments: schema.xml, solrconfig.xml
>
>
> Hi all,
> I'm facing an issue regarding custom code inside a .system-collection and 
> starting up a Solr Cloud cluster.
> I thought, like its stated in the documentation, that in case using the 
> .system collection custom code is lazy loaded, because it can happen that a 
> collection that uses custom code is initialized before the .system collection 
> is up and running.
> I did all the necessary configuration and while debugging, I can see that the 
> custom code is wrapped via a PluginBag$LazyPluginHolder. So far its seems 
> good, but I still get Exceptions when starting the Solr Cloud with the 
> following errors:
> SolrException: Blob loading failed: .no active replica available for .system 
> collection...
> In my case I'm using custom code for a couple of UpdateProcessors. So it 
> seems, that this lazy mechanism is not working well for UpdateProcessors.
> Inside the calzz LazyPluginHolder the comment says:
> "A class that loads plugins Lazily. When the get() method is invoked the 
> Plugin is initialized and returned."
> When a core is initialized and you have a custom UpdateProcessor, the 
> get-method is invoked directly and the lazy loading mechanism tries to get 
> the custom class from the MemClassLoader, but in most scenarios the system 
> collection is not up and the above Exception is thrown...
> So maybe it’s the case that for UpdateProcessors while initializing a core, 
> the routine is not implemented optimal for the lazy loading mechanism?
>  
> Here are the steps to reproduce the issue:
>  # Unpack solr 7.3.0
>  1.1 Add SOLR_OPTS="$SOLR_OPTS -Denable.runtime.lib=true" to solr.in.sh
>  1.2 Start Solr with -c option
>  # Setup .system collection:
>  2.1 Upload custom test jar --> curl -X POST -H 'Content-Type: 
> application/octet-stream' --data-binary @ jar>/update-processor-0.0.1-SNAPSHOT.jar http:// solr>/solr/.system/blob/test_blob
>  2.2 Alter maxShardsPerNode --> 
> .../admin/collections?action=MODIFYCOLLECTION=.system=2
>  2.2 Add Replica to .system collection --> 
> .../admin/collections?action=ADDREPLICA=.system=shard1
>  # Setup test collection:
>  3.1 Upload test conf to ZK --> ./zkcli.sh -zkhost  -cmd 
> upconfig -confdir  -confname test_conf
>  3.2 Create a test1 collection with commented UP-chain inside solrconfig.xml 
> via Admin UI
>  3.3 Add blob to test collection --> curl http:// Solr>/solr/test1/config -H 'Content-type:application/json' -d 
> '\{"add-runtimelib": { "name":"test_blob", "version":1 }}'
>  3.4 Uncomment the UP-chain and upload test-conf again --> ./zkcli.sh -zkhost 
>  -cmd upconfig -confdir  -confname test_conf
>  3.5 Reload test1 collection
>  3.6 Everything should work as expected now (no erros are shown)
>  # Restart SOLR
>  4.1 Now you can see: SolrException: Blob loading failed: No active replica 
> available for .system collection
> Expected: The lazy loading mechanism should work for UpdateProcessors inside 
> core init routine, but it isn't due to above Exception.
> Sometimes you are lucky and the test1 collection will be initialize after the 
> .system collection. But in ~90% of the time this isn't the case...
>  Let me know if you need further details here,
>  
> Johannes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12294) System collection - Lazy loading mechanism not working for custom UpdateProcessors

[Johannes Brucher (JIRA)]

Johannes Brucher created SOLR-12294:
---

 Summary: System collection - Lazy loading mechanism not working 
for custom UpdateProcessors
 Key: SOLR-12294
 URL: https://issues.apache.org/jira/browse/SOLR-12294
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: UpdateRequestProcessors
Affects Versions: 7.3
Reporter: Johannes Brucher


Hi all,

I'm facing an issue regarding custom code inside a .system-collection and 
starting up a Solr Cloud cluster.

I thought, like its stated in the documentation, that in case using the .system 
collection custom code is lazy loaded, because it can happen that a collection 
that uses custom code is initialized before the .system collection is up and 
running.

I did all the necessary configuration and while debugging, I can see that the 
custom code is wrapped via a PluginBag$LazyPluginHolder. So far its seems good, 
but I still get Exceptions when starting the Solr Cloud with the following 
errors:

SolrException: Blob loading failed: .no active replica available for .system 
collection...

In my case I'm using custom code for a couple of UpdateProcessors. So it seems, 
that this lazy mechanism is not working well for UpdateProcessors.

Inside the calzz LazyPluginHolder the comment says:

"A class that loads plugins Lazily. When the get() method is invoked the Plugin 
is initialized and returned."

When a core is initialized and you have a custom UpdateProcessor, the 
get-method is invoked directly and the lazy loading mechanism tries to get the 
custom class from the MemClassLoader, but in most scenarios the system 
collection is not up and the above Exception is thrown...

So maybe it’s the case that for UpdateProcessors while initializing a core, the 
routine is not implemented optimal for the lazy loading mechanism?

 

Here are the steps to reproduce the issue:
 # Unpack solr 7.3.0
 1.1 Add SOLR_OPTS="$SOLR_OPTS -Denable.runtime.lib=true" to solr.in.sh
 1.2 Start Solr with -c option
 # Setup .system collection:
 2.1 Upload custom test jar --> curl -X POST -H 'Content-Type: 
application/octet-stream' --data-binary @/update-processor-0.0.1-SNAPSHOT.jar http:///solr/.system/blob/test_blob
 2.2 Alter maxShardsPerNode --> 
.../admin/collections?action=MODIFYCOLLECTION=.system=2
 2.2 Add Replica to .system collection --> 
.../admin/collections?action=ADDREPLICA=.system=shard1
 # Setup test collection:
 3.1 Upload test conf to ZK --> ./zkcli.sh -zkhost  -cmd upconfig 
-confdir  -confname test_conf
 3.2 Create a test1 collection with commented UP-chain inside solrconfig.xml 
via Admin UI
 3.3 Add blob to test collection --> curl http:///solr/test1/config 
-H 'Content-type:application/json' -d '\{"add-runtimelib": { 
"name":"test_blob", "version":1 }}'
 3.4 Uncomment the UP-chain and upload test-conf again --> ./zkcli.sh -zkhost 
 -cmd upconfig -confdir  -confname test_conf
 3.5 Reload test1 collection
 3.6 Everything should work as expected now (no erros are shown)
 # Restart SOLR
 4.1 Now you can see: SolrException: Blob loading failed: No active replica 
available for .system collection

Expected: The lazy loading mechanism should work for UpdateProcessors inside 
core init routine, but it isn't due to above Exception.

Sometimes you are lucky and the test1 collection will be initialize after the 
.system collection. But in ~90% of the time this isn't the case...

 Let me know if you need further details here,

 

Johannes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12293) Updates need to use their own connection pool to maintain connection reuse and prevent spurious recoveries.

[Mark Miller (JIRA)]

Mark Miller created SOLR-12293:
--

 Summary: Updates need to use their own connection pool to maintain 
connection reuse and prevent spurious recoveries.
 Key: SOLR-12293
 URL: https://issues.apache.org/jira/browse/SOLR-12293
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Mark Miller
Assignee: Mark Miller


Currently the pool is shared too broadly - for example during replication we 
don't guarantee we read the full streams when downloading index files and we 
don't necessarily want to, emptying the stream for a huge file due to error or 
abort is too expensive. We can't have these connections pollute the update 
connection pool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12278) Ignore very large document on indexing

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458617#comment-16458617
 ] 

David Smiley commented on SOLR-12278:
-

+1 overall.  
 * probably won't pass precommit; I see {{ Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12290) Do not close any servlet streams and improve our servlet stream closing prevention code for users and devs.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458594#comment-16458594
 ] 

Mark Miller commented on SOLR-12290:


Yeah, it’s not super obvious, but the problem with closing ourselves is that 
the we can’t clear any data that might be left on the stream and we also can’t 
gaurantee some data does t show up from the other side right after an early 
close. So to ensure connection reuse you cannot close yourself.

> Do not close any servlet streams and improve our servlet stream closing 
> prevention code for users and devs.
> ---
>
> Key: SOLR-12290
> URL: https://issues.apache.org/jira/browse/SOLR-12290
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-12290.patch, SOLR-12290.patch, SOLR-12290.patch, 
> SOLR-12290.patch
>
>
> Original Summary:
> When you fetch a file for replication we close the request output stream 
> after writing the file which ruins the connection for reuse.
> We can't close response output streams, we need to reuse these connections. 
> If we do close them, clients are hit with connection problems when they try 
> and reuse the connection from their pool.
> New Summary:
> At some point the above was addressed during refactoring. We should remove 
> these neutered closes and review our close shield code.
> If you are here to track down why this is done:
> Connection reuse requires that we read all streams and do not close them - 
> instead the container itself must manage request and response streams. If we 
> allow them to be closed, not only do we lose some connection reuse, but we 
> can cause spurious client errors that can cause expensive recoveries for no 
> reason. The spec allows us to count on the container to manage streams. It's 
> our job simply to not close them and to always read them fully, from client 
> and server. 
> Java itself can help with always reading the streams fully up to some small 
> default amount of unread stream slack, but that is very dangerous to count 
> on, so we always manually eat up anything on the streams our normal logic 
> ends up not reading for whatever reason.
> We also cannot call abort without ruining the connection or sendError. These 
> should be options of very last resort (requiring a blood sacrifice) or when 
> shutting down.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8273) Add a ConditionalTokenFilter

[Robert Muir (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458569#comment-16458569
 ] 

Robert Muir commented on LUCENE-8273:
-

sounds good. yeah i know the resource stuff/keywork marking is tricky, i looked 
at what the existing factory is doing and its pretty crazy. 

it seems you need to make the ConditionalTokenFilterFactory implement the 
resourceloaderaware stuff always, because its separately a bug that the current 
patch will "hide" the resourceloader from anything inside the if? So I think it 
should implement the interface and pass the loader in its inform() method to 
stuff inside. Maybe this leads towards a solution to what you need for the 
conditional part, too.

> Add a ConditionalTokenFilter
> 
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch, LUCENE-8273.patch, 
> LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265.  It would be useful to be able to wrap a TokenFilter 
> in such a way that it could optionally be bypassed based on the current state 
> of the TokenStream.  This could be used to, for example, only apply 
> WordDelimiterFilter to terms that contain hyphens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12290) Do not close any servlet streams and improve our servlet stream closing prevention code for users and devs.

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458566#comment-16458566
 ] 

David Smiley commented on SOLR-12290:
-

Sounds good Mark!  Thanks for the clarifications.  I didn't know this problem 
could lead to client recovery; ouch!

> Do not close any servlet streams and improve our servlet stream closing 
> prevention code for users and devs.
> ---
>
> Key: SOLR-12290
> URL: https://issues.apache.org/jira/browse/SOLR-12290
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-12290.patch, SOLR-12290.patch, SOLR-12290.patch, 
> SOLR-12290.patch
>
>
> Original Summary:
> When you fetch a file for replication we close the request output stream 
> after writing the file which ruins the connection for reuse.
> We can't close response output streams, we need to reuse these connections. 
> If we do close them, clients are hit with connection problems when they try 
> and reuse the connection from their pool.
> New Summary:
> At some point the above was addressed during refactoring. We should remove 
> these neutered closes and review our close shield code.
> If you are here to track down why this is done:
> Connection reuse requires that we read all streams and do not close them - 
> instead the container itself must manage request and response streams. If we 
> allow them to be closed, not only do we lose some connection reuse, but we 
> can cause spurious client errors that can cause expensive recoveries for no 
> reason. The spec allows us to count on the container to manage streams. It's 
> our job simply to not close them and to always read them fully, from client 
> and server. 
> Java itself can help with always reading the streams fully up to some small 
> default amount of unread stream slack, but that is very dangerous to count 
> on, so we always manually eat up anything on the streams our normal logic 
> ends up not reading for whatever reason.
> We also cannot call abort without ruining the connection or sendError. These 
> should be options of very last resort (requiring a blood sacrifice) or when 
> shutting down.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8273) Add a ConditionalTokenFilter

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458550#comment-16458550
 ] 

Alan Woodward commented on LUCENE-8273:
---

{{if}} is a keyword, unfortunately, so that won't work.  Maybe {{ifCondition}} 
instead?

Access to resources will be a bit trickier, I think maybe the best way to do 
that would be to have a specialised method {{ifInList}} or something similar 
that takes a path to a word list.  I'll see what I can come up with.

> Add a ConditionalTokenFilter
> 
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch, LUCENE-8273.patch, 
> LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265.  It would be useful to be able to wrap a TokenFilter 
> in such a way that it could optionally be bypassed based on the current state 
> of the TokenStream.  This could be used to, for example, only apply 
> WordDelimiterFilter to terms that contain hyphens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.3-Linux (32bit/jdk1.8.0_162) - Build # 150 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.3-Linux/150/
Java: 32bit/jdk1.8.0_162 -client -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.AddReplicaTest.test

Error Message:
core_node6:{"core":"addreplicatest_coll_shard1_replica_n5","base_url":"https://127.0.0.1:46035/solr","node_name":"127.0.0.1:46035_solr","state":"active","type":"NRT"}

Stack Trace:
java.lang.AssertionError: 
core_node6:{"core":"addreplicatest_coll_shard1_replica_n5","base_url":"https://127.0.0.1:46035/solr","node_name":"127.0.0.1:46035_solr","state":"active","type":"NRT"}
at 
__randomizedtesting.SeedInfo.seed([3ED2E6ED3C7BDC92:B686D9379287B16A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.AddReplicaTest.test(AddReplicaTest.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13274 lines...]
   [junit4] Suite: 

[JENKINS] Lucene-Solr-repro - Build # 567 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/567/

[...truncated 33 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-Tests-7.x/589/consoleText

[repro] Revision: 56998e6de3d1c71ad19a7400a99cd7dd2925de05

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testTrigger -Dtests.seed=BFC1E72CC339B99C -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.locale=fi-FI -Dtests.timezone=Antarctica/Vostok 
-Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
871ffbe10edcad6574da4f4b93e9b56c9761c2be
[repro] git fetch
[repro] git checkout 56998e6de3d1c71ad19a7400a99cd7dd2925de05

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro] ant compile-test

[...truncated 3316 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.IndexSizeTriggerTest" -Dtests.showOutput=onerror  
-Dtests.seed=BFC1E72CC339B99C -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=fi-FI -Dtests.timezone=Antarctica/Vostok -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[...truncated 7863 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest

[repro] Re-testing 100% failures at the tip of branch_7x
[repro] ant clean

[...truncated 8 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro] ant compile-test

[...truncated 3316 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.IndexSizeTriggerTest" -Dtests.showOutput=onerror  
-Dtests.seed=BFC1E72CC339B99C -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=fi-FI -Dtests.timezone=Antarctica/Vostok -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[...truncated 6748 lines...]
[repro] Setting last failure code to 256

[repro] Failures at the tip of branch_7x:
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest

[repro] Re-testing 100% failures at the tip of branch_7x without a seed
[repro] ant clean

[...truncated 8 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro] ant compile-test

[...truncated 3316 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.IndexSizeTriggerTest" -Dtests.showOutput=onerror  
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=fi-FI 
-Dtests.timezone=Antarctica/Vostok -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[...truncated 7183 lines...]
[repro] Setting last failure code to 256

[repro] Failures at the tip of branch_7x without a seed:
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[repro] git checkout 871ffbe10edcad6574da4f4b93e9b56c9761c2be

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 5 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8273) Add a ConditionalTokenFilter

[Robert Muir (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458507#comment-16458507
 ] 

Robert Muir commented on LUCENE-8273:
-

I'm also curious about common use cases where the condition is just matching a 
list of words, basically what the KeywordMarker factory provides today. Does 
the user have access to stuff like resource loaders to read from files / is it 
intuitive so they won't be reading the list of words in on every token or other 
mistakes? If we can make this simple, I think we can deprecate KeywordMarker 
and many other exception-list-type mechanisms hardcoded in all the filters, 
which would be a really nice cleanup. 

> Add a ConditionalTokenFilter
> 
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch, LUCENE-8273.patch, 
> LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265.  It would be useful to be able to wrap a TokenFilter 
> in such a way that it could optionally be bypassed based on the current state 
> of the TokenStream.  This could be used to, for example, only apply 
> WordDelimiterFilter to terms that contain hyphens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8273) Add a ConditionalTokenFilter

[Robert Muir (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458501#comment-16458501
 ] 

Robert Muir commented on LUCENE-8273:
-

I like the custom analyzer integration. Can we rename {{ifMatches}} just to 
{{if}}? This makes it more natural for the user to pair with the necessary 
{{endif}}, doesn't imply any regex matching, etc. I think its useful to mention 
the necessary endif in the javadocs for both these methods, if users forget to 
call it they should get a compile error from build(), but it may not be 
obvious. I would also add to the {{ifTerm}} docs that it is just sugar, with 
snippet of how to implement it with if + CharTermAttribute. This gives an 
example in case the user needs to work on some other attribute.

> Add a ConditionalTokenFilter
> 
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch, LUCENE-8273.patch, 
> LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265.  It would be useful to be able to wrap a TokenFilter 
> in such a way that it could optionally be bypassed based on the current state 
> of the TokenStream.  This could be used to, for example, only apply 
> WordDelimiterFilter to terms that contain hyphens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-7.x - Build # 590 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/590/

3 tests failed.
FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.test

Error Message:
Error from server at http://127.0.0.1:38687/solr: KeeperErrorCode = Session 
expired for /configs/conf1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:38687/solr: KeeperErrorCode = Session expired 
for /configs/conf1
at 
__randomizedtesting.SeedInfo.seed([E3E98BA607A655A3:6BBDB47CA95A385B]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.MoveReplicaTest.test(MoveReplicaTest.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 

[jira] [Created] (SOLR-12292) Make it easier to configure Solr with CORS

[JIRA]

Jan Høydahl created SOLR-12292:
--

 Summary: Make it easier to configure Solr with CORS
 Key: SOLR-12292
 URL: https://issues.apache.org/jira/browse/SOLR-12292
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: Server
Reporter: Jan Høydahl


While working on SOLR-8207 I wanted to collect info from other SolrCloud nodes 
from the AdminUI. However this is blocked by 
[CORS|https://en.wikipedia.org/wiki/Cross-origin_resource_sharing] policy. In 
that Jira I instead did the fan-out on the Solr server side for the two handler 
I needed.

It would be nice if all nodes in a SolrCloud cluster could automatically accept 
any other node as a legal origin, and make it easy for users to add other 
origins by config.

If we use the [Jetty CORS 
filter|http://www.eclipse.org/jetty/documentation/9.4.9.v20180320/cross-origin-filter.html]
 in web.xml, perhaps we could parse a env.var from solr.in.xx and inject into 
the {{allowedOrigins}} property of that filter? There is also SOLR-6059 which 
tries to implement CORS inside of Solr handlers and not in Jetty. Don't know 
pros/cons of those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8273) Add a ConditionalTokenFilter

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458472#comment-16458472
 ] 

Alan Woodward commented on LUCENE-8273:
---

[~msoko...@gmail.com] in that case, the filter would still get applied, because 
we only check the token that is passed to the synonym filter from outside.  
Anything that's pulled by the synonym filter itself doesn't get checked.  
Although thinking about it, it would be possible to run the check in the 
OneTimeWrapper as well and return 'false' from things that don't pass the 
check.  I'm not sure how that would work with the graph itself though, it might 
end up corrupting things.

> Add a ConditionalTokenFilter
> 
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch, LUCENE-8273.patch, 
> LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265.  It would be useful to be able to wrap a TokenFilter 
> in such a way that it could optionally be bypassed based on the current state 
> of the TokenStream.  This could be used to, for example, only apply 
> WordDelimiterFilter to terms that contain hyphens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8273) Add a ConditionalTokenFilter

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458469#comment-16458469
 ] 

Alan Woodward commented on LUCENE-8273:
---

And here's a patch that integrates it into CustomAnalyzer

> Add a ConditionalTokenFilter
> 
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch, LUCENE-8273.patch, 
> LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265.  It would be useful to be able to wrap a TokenFilter 
> in such a way that it could optionally be bypassed based on the current state 
> of the TokenStream.  This could be used to, for example, only apply 
> WordDelimiterFilter to terms that contain hyphens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8273) Add a ConditionalTokenFilter

[Alan Woodward (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward updated LUCENE-8273:
--
Attachment: LUCENE-8273.patch

> Add a ConditionalTokenFilter
> 
>
> Key: LUCENE-8273
> URL: https://issues.apache.org/jira/browse/LUCENE-8273
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8273.patch, LUCENE-8273.patch, LUCENE-8273.patch, 
> LUCENE-8273.patch
>
>
> Spinoff of LUCENE-8265.  It would be useful to be able to wrap a TokenFilter 
> in such a way that it could optionally be bypassed based on the current state 
> of the TokenStream.  This could be used to, for example, only apply 
> WordDelimiterFilter to terms that contain hyphens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8280) Repeatable failure for GeoConvexPolygons

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458451#comment-16458451
 ] 

ASF subversion and git services commented on LUCENE-8280:
-

Commit ec50b224dfed558b18b5ed39e44ef6d504c66442 in lucene-solr's branch 
refs/heads/branch_6x from [~kwri...@metacarta.com]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ec50b22 ]

LUCENE-8280: Reorganize to allow us to try lots of strategies until we get one.


> Repeatable failure for GeoConvexPolygons
> 
>
> Key: LUCENE-8280
> URL: https://issues.apache.org/jira/browse/LUCENE-8280
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/spatial3d
>Reporter: Karl Wright
>Assignee: Karl Wright
>Priority: Major
> Fix For: 6.7, 7.4, master (8.0)
>
>
> Reproduce with:
> {code}
> ant test  -Dtestcase=TestGeo3DPoint -Dtests.method=testGeo3DRelations 
> -Dtests.seed=1F49469C1989BC0 -Dtests.slow=true -Dtests.badapples=true 
> -Dtests.locale=fr-BE -Dtests.timezone=Europe/Malta -Dtests.asserts=true 
> -Dtests.file.encoding=Cp1252
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8280) Repeatable failure for GeoConvexPolygons

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458450#comment-16458450
 ] 

ASF subversion and git services commented on LUCENE-8280:
-

Commit 419cdd0981ce5fa2b11752a026921931cc3c5f30 in lucene-solr's branch 
refs/heads/branch_7x from [~kwri...@metacarta.com]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=419cdd0 ]

LUCENE-8280: Reorganize to allow us to try lots of strategies until we get one.


> Repeatable failure for GeoConvexPolygons
> 
>
> Key: LUCENE-8280
> URL: https://issues.apache.org/jira/browse/LUCENE-8280
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/spatial3d
>Reporter: Karl Wright
>Assignee: Karl Wright
>Priority: Major
> Fix For: 6.7, 7.4, master (8.0)
>
>
> Reproduce with:
> {code}
> ant test  -Dtestcase=TestGeo3DPoint -Dtests.method=testGeo3DRelations 
> -Dtests.seed=1F49469C1989BC0 -Dtests.slow=true -Dtests.badapples=true 
> -Dtests.locale=fr-BE -Dtests.timezone=Europe/Malta -Dtests.asserts=true 
> -Dtests.file.encoding=Cp1252
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8280) Repeatable failure for GeoConvexPolygons

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458449#comment-16458449
 ] 

ASF subversion and git services commented on LUCENE-8280:
-

Commit ff68acf2449f0f705a949e7afb592c4139fd52ad in lucene-solr's branch 
refs/heads/master from [~kwri...@metacarta.com]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ff68acf ]

LUCENE-8280: Reorganize to allow us to try lots of strategies until we get one.


> Repeatable failure for GeoConvexPolygons
> 
>
> Key: LUCENE-8280
> URL: https://issues.apache.org/jira/browse/LUCENE-8280
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/spatial3d
>Reporter: Karl Wright
>Assignee: Karl Wright
>Priority: Major
> Fix For: 6.7, 7.4, master (8.0)
>
>
> Reproduce with:
> {code}
> ant test  -Dtestcase=TestGeo3DPoint -Dtests.method=testGeo3DRelations 
> -Dtests.seed=1F49469C1989BC0 -Dtests.slow=true -Dtests.badapples=true 
> -Dtests.locale=fr-BE -Dtests.timezone=Europe/Malta -Dtests.asserts=true 
> -Dtests.file.encoding=Cp1252
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12283) Unable To Load ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in .system collection

[Maxence SAUNIER (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maxence SAUNIER updated SOLR-12283:
---
Attachment: solr.log

> Unable To Load ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in 
> .system collection
> 
>
> Key: SOLR-12283
> URL: https://issues.apache.org/jira/browse/SOLR-12283
> Project: Solr
>  Issue Type: Test
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: contrib - DataImportHandler
>Affects Versions: 6.6.1, 7.3
> Environment: Debian
> SolrCloud
>Reporter: Maxence SAUNIER
>Priority: Blocker
> Attachments: indexation_events.xml, modified-DIH.zip, 
> mysql-connector-java-5.1.46-bin.jar, mysql-connector-java-5.1.46.jar, 
> request_handler_config.json, solr-core-7.3.0.jar, 
> solr-dataimporthandler-7.3.0.jar, solr-dataimporthandler-extras-7.3.0.jar, 
> solr-solrj-7.3.0.jar, solr.log, solr.log, solr.log
>
>
> Hello,
> It's been 2 weeks that I try to correct this problem with the community 
> user-solr but no success. I seriously wonder if this is not a problem in the 
> code. I do not have the impression that many people use DIH with Solr's cloud 
> version.
> On Internet, no similar problem.
> For information, the following configuration of DIH comes from DIHs that work 
> in production on a single Solr server. The connections to the databases are 
> therefore correct.
> *Errors messages:*
> {panel:title=DataImporter}
> {code:java}
> Full Import 
> failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable 
> to PropertyWriter implementation:ZKPropertiesWriter
>   at 
> org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:339)
>   at 
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:420)
>   at 
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
>   at 
> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at 
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at org.eclipse.jetty.server.Server.handle(Server.java:530)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
>   at 
> 

[jira] [Commented] (SOLR-12283) Unable To Load ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in .system collection

[Maxence SAUNIER (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458448#comment-16458448
 ] 

Maxence SAUNIER commented on SOLR-12283:


I have restart solr with your DIH jar. I join the file. [^solr.log] 

> Unable To Load ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in 
> .system collection
> 
>
> Key: SOLR-12283
> URL: https://issues.apache.org/jira/browse/SOLR-12283
> Project: Solr
>  Issue Type: Test
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: contrib - DataImportHandler
>Affects Versions: 6.6.1, 7.3
> Environment: Debian
> SolrCloud
>Reporter: Maxence SAUNIER
>Priority: Blocker
> Attachments: indexation_events.xml, modified-DIH.zip, 
> mysql-connector-java-5.1.46-bin.jar, mysql-connector-java-5.1.46.jar, 
> request_handler_config.json, solr-core-7.3.0.jar, 
> solr-dataimporthandler-7.3.0.jar, solr-dataimporthandler-extras-7.3.0.jar, 
> solr-solrj-7.3.0.jar, solr.log, solr.log, solr.log
>
>
> Hello,
> It's been 2 weeks that I try to correct this problem with the community 
> user-solr but no success. I seriously wonder if this is not a problem in the 
> code. I do not have the impression that many people use DIH with Solr's cloud 
> version.
> On Internet, no similar problem.
> For information, the following configuration of DIH comes from DIHs that work 
> in production on a single Solr server. The connections to the databases are 
> therefore correct.
> *Errors messages:*
> {panel:title=DataImporter}
> {code:java}
> Full Import 
> failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable 
> to PropertyWriter implementation:ZKPropertiesWriter
>   at 
> org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:339)
>   at 
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:420)
>   at 
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
>   at 
> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at 
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at org.eclipse.jetty.server.Server.handle(Server.java:530)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
>   at 
> 

[jira] [Commented] (LUCENE-8275) Push up #checkPendingDeletes to Directory

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458442#comment-16458442
 ] 

ASF subversion and git services commented on LUCENE-8275:
-

Commit fe018e37f3a8bc4af52ca98e0e49c1317b159c80 in lucene-solr's branch 
refs/heads/master from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fe018e3 ]

LUCENE-8275: Fix TestDirectoryTaxonomyWriter.testRecreateAndRefresh


> Push up #checkPendingDeletes to Directory
> -
>
> Key: LUCENE-8275
> URL: https://issues.apache.org/jira/browse/LUCENE-8275
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8275.patch, LUCENE-8275.patch
>
>
>  IndexWriter checks in it's ctor if the incoming directory is an
> FSDirectory. If that is the case it ensures that the directory retries
> deleting it's pending deletes and if there are pending deletes it will
> fail creating the writer. Yet, this check didn't unwrap filter directories
> or subclasses like FileSwitchDirectory such that in the case of MDW we
> never checked for pending deletes.
> 
> There are also two places in FSDirectory that first removed the file
> that was supposed to be created / renamed to from the pending deletes set
> and then tried to clean up pending deletes which excluded the file. These
> places now remove the file from the set after the pending deletes are 
> checked.
>  
> This caused some test failures lately unfortunately very timing dependent:
>  
> {noformat}
> FAILED:  
> junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager
> Error Message:
> Captured an uncaught exception in thread: Thread[id=1567, name=Thread-1363, 
> state=RUNNABLE, group=TGRP-TestSearcherManager]
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an 
> uncaught exception in thread: Thread[id=1567, name=Thread-1363, 
> state=RUNNABLE, group=TGRP-TestSearcherManager]
> Caused by: java.lang.RuntimeException: 
> java.nio.file.FileAlreadyExistsException: 
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J1/temp/lucene.search.TestSearcherManager_BA998C838D219DA9-001/tempDir-001/_0.fdt
> at __randomizedtesting.SeedInfo.seed([BA998C838D219DA9]:0)
> at 
> org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:590)
> Caused by: java.nio.file.FileAlreadyExistsException: 
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J1/temp/lucene.search.TestSearcherManager_BA998C838D219DA9-001/tempDir-001/_0.fdt
> at 
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
> at 
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at 
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at 
> java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215)
> at 
> java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
> at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
> at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
> at 
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
> at 
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
> at 
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
> at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
> at java.base/java.nio.file.Files.newOutputStream(Files.java:218)
> at 
> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:413)
> at 
> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:409)
> at 
> org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
> at 
> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:665)
> at 
> org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
> at 
> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> at 
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:116)
> at 
> 

Re: [JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_144) - Build # 7296 - Still Unstable!

[Simon Willnauer]

pushed a fix to master and 7x

On Mon, Apr 30, 2018 at 11:46 AM, Simon Willnauer
 wrote:
> I am looking into the TestDirectoryTaxonomyWriter#testRecreateAndRefresh 
> failure
>
> On Mon, Apr 30, 2018 at 9:59 AM, Policeman Jenkins Server
>  wrote:
>> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7296/
>> Java: 64bit/jdk1.8.0_144 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>
>> 29 tests failed.
>> FAILED:  
>> org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter.testRecreateAndRefresh
>>
>> Error Message:
>> Directory 
>> MockDirectoryWrapper(NIOFSDirectory@C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\facet\test\J1\temp\lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter_F392C2FFDA61922B-001\index-NIOFSDirectory-001
>>  lockFactory=org.apache.lucene.store.NativeFSLockFactory@5e6a7788) still has 
>> pending deleted files; cannot initialize IndexWriter
>>
>> Stack Trace:
>> java.lang.IllegalArgumentException: Directory 
>> MockDirectoryWrapper(NIOFSDirectory@C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\facet\test\J1\temp\lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter_F392C2FFDA61922B-001\index-NIOFSDirectory-001
>>  lockFactory=org.apache.lucene.store.NativeFSLockFactory@5e6a7788) still has 
>> pending deleted files; cannot initialize IndexWriter
>> at 
>> __randomizedtesting.SeedInfo.seed([F392C2FFDA61922B:BD08DA5FF63FC513]:0)
>> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:707)
>> at 
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.openIndexWriter(DirectoryTaxonomyWriter.java:240)
>> at 
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:167)
>> at 
>> org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter.testRecreateAndRefresh(TestDirectoryTaxonomyWriter.java:214)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>> at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>> at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>> at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>> at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>> at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>> at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>> at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>> at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>> at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>> at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>> at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>> at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>> at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at 
>> 

[jira] [Commented] (LUCENE-8275) Push up #checkPendingDeletes to Directory

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458441#comment-16458441
 ] 

ASF subversion and git services commented on LUCENE-8275:
-

Commit 1cd6f1b6e7f3e24906f552d9460b7878aef82268 in lucene-solr's branch 
refs/heads/branch_7x from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1cd6f1b ]

LUCENE-8275: Fix TestDirectoryTaxonomyWriter.testRecreateAndRefresh


> Push up #checkPendingDeletes to Directory
> -
>
> Key: LUCENE-8275
> URL: https://issues.apache.org/jira/browse/LUCENE-8275
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8275.patch, LUCENE-8275.patch
>
>
>  IndexWriter checks in it's ctor if the incoming directory is an
> FSDirectory. If that is the case it ensures that the directory retries
> deleting it's pending deletes and if there are pending deletes it will
> fail creating the writer. Yet, this check didn't unwrap filter directories
> or subclasses like FileSwitchDirectory such that in the case of MDW we
> never checked for pending deletes.
> 
> There are also two places in FSDirectory that first removed the file
> that was supposed to be created / renamed to from the pending deletes set
> and then tried to clean up pending deletes which excluded the file. These
> places now remove the file from the set after the pending deletes are 
> checked.
>  
> This caused some test failures lately unfortunately very timing dependent:
>  
> {noformat}
> FAILED:  
> junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager
> Error Message:
> Captured an uncaught exception in thread: Thread[id=1567, name=Thread-1363, 
> state=RUNNABLE, group=TGRP-TestSearcherManager]
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an 
> uncaught exception in thread: Thread[id=1567, name=Thread-1363, 
> state=RUNNABLE, group=TGRP-TestSearcherManager]
> Caused by: java.lang.RuntimeException: 
> java.nio.file.FileAlreadyExistsException: 
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J1/temp/lucene.search.TestSearcherManager_BA998C838D219DA9-001/tempDir-001/_0.fdt
> at __randomizedtesting.SeedInfo.seed([BA998C838D219DA9]:0)
> at 
> org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:590)
> Caused by: java.nio.file.FileAlreadyExistsException: 
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J1/temp/lucene.search.TestSearcherManager_BA998C838D219DA9-001/tempDir-001/_0.fdt
> at 
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
> at 
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at 
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at 
> java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215)
> at 
> java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
> at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
> at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
> at 
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
> at 
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
> at 
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
> at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
> at java.base/java.nio.file.Files.newOutputStream(Files.java:218)
> at 
> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:413)
> at 
> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:409)
> at 
> org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
> at 
> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:665)
> at 
> org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
> at 
> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> at 
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:116)
> at 
> 

[jira] [Resolved] (LUCENE-8282) Reduce boxing and unnecessary object creation in DV updates

[Simon Willnauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Willnauer resolved LUCENE-8282.
-
Resolution: Fixed

> Reduce boxing and unnecessary object creation in DV updates
> ---
>
> Key: LUCENE-8282
> URL: https://issues.apache.org/jira/browse/LUCENE-8282
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8282.patch
>
>
> DV updates used the boxed type Long to keep API generic. Yet, the missing
> type caused a lot of code duplication, boxing and unnecessary object creation.
> This change cuts over to type safe APIs using BytesRef and long (the 
> primitive)
> In this change most of the code that is almost identical between binary and 
> numeric
> is not shared reducing the maintenance overhead and likelihood of introducing 
> bugs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8282) Reduce boxing and unnecessary object creation in DV updates

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458437#comment-16458437
 ] 

ASF subversion and git services commented on LUCENE-8282:
-

Commit 8f4cd091c23634f3c4e9bf711832e0a50db92ea3 in lucene-solr's branch 
refs/heads/branch_7x from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8f4cd09 ]

LUCENE-8282: Reduce boxing and unnecessary object creation in DV updates

DV updates used the boxed type Long to keep API generic. Yet, the missing
type caused a lot of code duplication, boxing and unnecessary object creation.
This change cuts over to type safe APIs using BytesRef and long (the primitive)

In this change most of the code that is almost identical between binary and 
numeric
is not shared reducing the maintenance overhead and likelihood of introducing 
bugs.

> Reduce boxing and unnecessary object creation in DV updates
> ---
>
> Key: LUCENE-8282
> URL: https://issues.apache.org/jira/browse/LUCENE-8282
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8282.patch
>
>
> DV updates used the boxed type Long to keep API generic. Yet, the missing
> type caused a lot of code duplication, boxing and unnecessary object creation.
> This change cuts over to type safe APIs using BytesRef and long (the 
> primitive)
> In this change most of the code that is almost identical between binary and 
> numeric
> is not shared reducing the maintenance overhead and likelihood of introducing 
> bugs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_144) - Build # 7296 - Still Unstable!

[Simon Willnauer]

I am looking into the TestDirectoryTaxonomyWriter#testRecreateAndRefresh failure

On Mon, Apr 30, 2018 at 9:59 AM, Policeman Jenkins Server
 wrote:
> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7296/
> Java: 64bit/jdk1.8.0_144 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>
> 29 tests failed.
> FAILED:  
> org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter.testRecreateAndRefresh
>
> Error Message:
> Directory 
> MockDirectoryWrapper(NIOFSDirectory@C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\facet\test\J1\temp\lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter_F392C2FFDA61922B-001\index-NIOFSDirectory-001
>  lockFactory=org.apache.lucene.store.NativeFSLockFactory@5e6a7788) still has 
> pending deleted files; cannot initialize IndexWriter
>
> Stack Trace:
> java.lang.IllegalArgumentException: Directory 
> MockDirectoryWrapper(NIOFSDirectory@C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\facet\test\J1\temp\lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter_F392C2FFDA61922B-001\index-NIOFSDirectory-001
>  lockFactory=org.apache.lucene.store.NativeFSLockFactory@5e6a7788) still has 
> pending deleted files; cannot initialize IndexWriter
> at 
> __randomizedtesting.SeedInfo.seed([F392C2FFDA61922B:BD08DA5FF63FC513]:0)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:707)
> at 
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.openIndexWriter(DirectoryTaxonomyWriter.java:240)
> at 
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:167)
> at 
> org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter.testRecreateAndRefresh(TestDirectoryTaxonomyWriter.java:214)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
> at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> 

[jira] [Commented] (LUCENE-8282) Reduce boxing and unnecessary object creation in DV updates

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458430#comment-16458430
 ] 

ASF subversion and git services commented on LUCENE-8282:
-

Commit b43b09190d52a959b8d3b10fcadfabfa58691955 in lucene-solr's branch 
refs/heads/master from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b43b091 ]

LUCENE-8282: Reduce boxing and unnecessary object creation in DV updates

DV updates used the boxed type Long to keep API generic. Yet, the missing
type caused a lot of code duplication, boxing and unnecessary object creation.
This change cuts over to type safe APIs using BytesRef and long (the primitive)

In this change most of the code that is almost identical between binary and 
numeric
is not shared reducing the maintenance overhead and likelihood of introducing 
bugs.

> Reduce boxing and unnecessary object creation in DV updates
> ---
>
> Key: LUCENE-8282
> URL: https://issues.apache.org/jira/browse/LUCENE-8282
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.4, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8282.patch
>
>
> DV updates used the boxed type Long to keep API generic. Yet, the missing
> type caused a lot of code duplication, boxing and unnecessary object creation.
> This change cuts over to type safe APIs using BytesRef and long (the 
> primitive)
> In this change most of the code that is almost identical between binary and 
> numeric
> is not shared reducing the maintenance overhead and likelihood of introducing 
> bugs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12282) JSONResponseWriter should return Content-Type application/javascript for JSONP requests

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458427#comment-16458427
 ] 

Jan Høydahl commented on SOLR-12282:


Hmm, why would we want to keep this supposedly insecure JSONP capability around 
when solutions for handling pure JSON with CORS are available and mature?

> JSONResponseWriter should return Content-Type application/javascript for 
> JSONP requests
> ---
>
> Key: SOLR-12282
> URL: https://issues.apache.org/jira/browse/SOLR-12282
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Response Writers
>Reporter: Markus Schuch
>Priority: Minor
> Attachments: SOLR-12282.patch
>
>
> The JSONResponseWriter handles two purposes:
>  * writing JSON responses (wt=json)
>  * writing JSONP responses, namely JSON responses wrapped by a JavaScript 
> function (wt=json=myFunction)
> The response writer returns the same Content-Type in both cases. (default: 
> application/json)
> But for JSONP the correct Content-Type would be "application/javascript".
> The response type is configurable, but it is currently not possible to return 
> the correct Content-Type in both cases with one configuration at the same 
> time.
> The attached patch changes the default Content-Type to 
> {{application/javascript; charset=utf-8}} for responses with wrapper 
> functions (JSONP). If param {{content-type}} is configured, this Content-Type 
> will be returned for both JSON and JSONP to ensure backward compatibility.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12278) Ignore very large document on indexing

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458426#comment-16458426
 ] 

Cao Manh Dat commented on SOLR-12278:
-

After discuss privately with [~shalinmangar], this case seems quite rare and it 
should be an optional processor.

I uploaded a new patch that match the new solution.

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12278) Ignore very large document on indexing

[Cao Manh Dat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cao Manh Dat updated SOLR-12278:

Attachment: SOLR-12278.patch

> Ignore very large document on indexing
> --
>
> Key: SOLR-12278
> URL: https://issues.apache.org/jira/browse/SOLR-12278
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-12278.patch, SOLR-12278.patch
>
>
> Solr should be able to ignore very large document, so it won't affect the 
> index as well as the tlog. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_144) - Build # 7296 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7296/
Java: 64bit/jdk1.8.0_144 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

29 tests failed.
FAILED:  
org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter.testRecreateAndRefresh

Error Message:
Directory 
MockDirectoryWrapper(NIOFSDirectory@C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\facet\test\J1\temp\lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter_F392C2FFDA61922B-001\index-NIOFSDirectory-001
 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5e6a7788) still has 
pending deleted files; cannot initialize IndexWriter

Stack Trace:
java.lang.IllegalArgumentException: Directory 
MockDirectoryWrapper(NIOFSDirectory@C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\facet\test\J1\temp\lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter_F392C2FFDA61922B-001\index-NIOFSDirectory-001
 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5e6a7788) still has 
pending deleted files; cannot initialize IndexWriter
at 
__randomizedtesting.SeedInfo.seed([F392C2FFDA61922B:BD08DA5FF63FC513]:0)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:707)
at 
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.openIndexWriter(DirectoryTaxonomyWriter.java:240)
at 
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.(DirectoryTaxonomyWriter.java:167)
at 
org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyWriter.testRecreateAndRefresh(TestDirectoryTaxonomyWriter.java:214)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 

[JENKINS] Lucene-Solr-repro - Build # 566 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/566/

[...truncated 28 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-Tests-master/2510/consoleText

[repro] Revision: 70abbe7433fc205e4abd05ebfc0fcf9399bf0f46

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testTrigger -Dtests.seed=73D21EFCC54D8AEE -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.locale=nl-NL -Dtests.timezone=Europe/Vilnius 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testMixedBounds -Dtests.seed=73D21EFCC54D8AEE 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=nl-NL 
-Dtests.timezone=Europe/Vilnius -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testMergeIntegration -Dtests.seed=73D21EFCC54D8AEE 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=nl-NL 
-Dtests.timezone=Europe/Vilnius -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testSplitIntegration -Dtests.seed=73D21EFCC54D8AEE 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=nl-NL 
-Dtests.timezone=Europe/Vilnius -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=NodeAddedTriggerTest 
-Dtests.method=testRestoreState -Dtests.seed=73D21EFCC54D8AEE 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=en-MT 
-Dtests.timezone=Asia/Kashgar -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=ZkControllerTest 
-Dtests.seed=73D21EFCC54D8AEE -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=en-IN -Dtests.timezone=Asia/Aqtobe -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=GraphTest 
-Dtests.seed=961F3B63324F46A -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=ru -Dtests.timezone=Pacific/Guam -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
570fff86727742360703a7d17f33127f3341edda
[repro] git fetch
[repro] git checkout 70abbe7433fc205e4abd05ebfc0fcf9399bf0f46

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/solrj
[repro]   GraphTest
[repro]solr/core
[repro]   ZkControllerTest
[repro]   IndexSizeTriggerTest
[repro]   NodeAddedTriggerTest
[repro] ant compile-test

[...truncated 2450 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.GraphTest" -Dtests.showOutput=onerror  
-Dtests.seed=961F3B63324F46A -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=ru -Dtests.timezone=Pacific/Guam -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[...truncated 68 lines...]
[repro] ant compile-test

[...truncated 1329 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=15 
-Dtests.class="*.ZkControllerTest|*.IndexSizeTriggerTest|*.NodeAddedTriggerTest"
 -Dtests.showOutput=onerror  -Dtests.seed=73D21EFCC54D8AEE -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.locale=en-IN -Dtests.timezone=Asia/Aqtobe 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8

[...truncated 4863 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.client.solrj.io.graph.GraphTest
[repro]   0/5 failed: org.apache.solr.cloud.ZkControllerTest
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.NodeAddedTriggerTest
[repro]   1/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[repro] git checkout 570fff86727742360703a7d17f33127f3341edda

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 5 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-master - Build # 2511 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2511/

2 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.AssignBackwardCompatibilityTest

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.cloud.AssignBackwardCompatibilityTest: 1) Thread[id=8972, 
name=qtp935415609-8972, state=TIMED_WAITING, 
group=TGRP-AssignBackwardCompatibilityTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.AssignBackwardCompatibilityTest: 
   1) Thread[id=8972, name=qtp935415609-8972, state=TIMED_WAITING, 
group=TGRP-AssignBackwardCompatibilityTest]
at sun.misc.Unsafe.park(Native Method)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([5233DFD0BCCDB76F]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.AssignBackwardCompatibilityTest

Error Message:
There are still zombie threads that couldn't be terminated:1) 
Thread[id=8972, name=qtp935415609-8972, state=TIMED_WAITING, 
group=TGRP-AssignBackwardCompatibilityTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) 
at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=8972, name=qtp935415609-8972, state=TIMED_WAITING, 
group=TGRP-AssignBackwardCompatibilityTest]
at sun.misc.Unsafe.park(Native Method)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([5233DFD0BCCDB76F]:0)




Build Log:
[...truncated 13600 lines...]
   [junit4] Suite: org.apache.solr.cloud.AssignBackwardCompatibilityTest
   [junit4]   2> Creating dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.AssignBackwardCompatibilityTest_5233DFD0BCCDB76F-001/init-core-data-001
   [junit4]   2> 2913133 INFO  
(SUITE-AssignBackwardCompatibilityTest-seed#[5233DFD0BCCDB76F]-worker) [] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.AssignBackwardCompatibilityTest_5233DFD0BCCDB76F-001/tempDir-001
   [junit4]   2> 2913134 INFO  

[jira] [Commented] (SOLR-11881) Connection Reset Causing LIR

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458341#comment-16458341
 ] 

Varun Thacker commented on SOLR-11881:
--

> For the second exception here is something similiar: 
>[https://github.com/eclipse/jetty.project/issues/1047]

Yeah that issue seems to have been fixed but here's the mailing list thread 
that the jetty folks pointed me to a bug with ssl/async : 
[https://dev.eclipse.org/mhonarc/lists/jetty-dev/msg03165.html]
 

> Connection Reset Causing LIR
> 
>
> Key: SOLR-11881
> URL: https://issues.apache.org/jira/browse/SOLR-11881
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-11881.patch, SOLR-11881.patch
>
>
> We can see that a connection reset is causing LIR.
> If a leader -> replica update get's a connection like this the leader will 
> initiate LIR
> {code}
> 2018-01-08 17:39:16.980 ERROR (qtp1010030701-468988) [c:person s:shard7_1 
> r:core_node56 x:person_shard7_1_replica1] 
> o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
> replica https://host08.domain:8985/solr/person_shard7_1_replica2/
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at 
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:312)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:185)
> at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> From https://issues.apache.org/jira/browse/SOLR-6931 Mark says "On a heavy 
> working SolrCloud cluster, even a rare response like this from a replica can 
> cause a recovery and heavy cluster disruption" . 
> Looking at SOLR-6931 we added a http retry handler but we only retry on GET 
> requests. Updates are POST requests 
> {{ConcurrentUpdateSolrClient#sendUpdateStream}}
> Update requests between the leader and replica should be retry-able since 
> they have been versioned. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11881) Connection Reset Causing LIR

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458338#comment-16458338
 ] 

Mark Miller commented on SOLR-11881:


Another possibility (beyond some different bug) for the second one is that it's 
reusing connections and there is a case we don't fully clear the connection 
streams. Normally, Jetty will just not reuse that connection if that happens, 
perhaps SSL with this new async stuff can hit this if that happens though. Just 
a guess and reminder that I should review our code that ensures everyone is 
fully reading streams.

> Connection Reset Causing LIR
> 
>
> Key: SOLR-11881
> URL: https://issues.apache.org/jira/browse/SOLR-11881
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-11881.patch, SOLR-11881.patch
>
>
> We can see that a connection reset is causing LIR.
> If a leader -> replica update get's a connection like this the leader will 
> initiate LIR
> {code}
> 2018-01-08 17:39:16.980 ERROR (qtp1010030701-468988) [c:person s:shard7_1 
> r:core_node56 x:person_shard7_1_replica1] 
> o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
> replica https://host08.domain:8985/solr/person_shard7_1_replica2/
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at 
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:312)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:185)
> at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> From https://issues.apache.org/jira/browse/SOLR-6931 Mark says "On a heavy 
> working SolrCloud cluster, even a rare response like this from a replica can 
> cause a recovery and heavy cluster disruption" . 
> Looking at SOLR-6931 we added a http retry handler but we only retry on GET 
> requests. Updates are POST requests 
> {{ConcurrentUpdateSolrClient#sendUpdateStream}}
> Update requests between the leader and replica should be retry-able since 
> they have been versioned. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org