[jira] [Updated] (LUCENE-7896) Upgrade to RandomizedRunner 2.5.2

[Dawid Weiss (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Weiss updated LUCENE-7896:

Fix Version/s: (was: master (8.0))

> Upgrade to RandomizedRunner 2.5.2
> -
>
> Key: LUCENE-7896
> URL: https://issues.apache.org/jira/browse/LUCENE-7896
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Minor
> Fix For: 7.0
>
> Attachments: LUCENE-7896.patch
>
>
> RR 2.5.2 fixed a nasty error message that gets printed while running tests 
> that is pretty annoying if you have the environment hitting this. Lets 
> upgrade to 2.5.2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-7896) Upgrade to RandomizedRunner 2.5.2

[Dawid Weiss (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Weiss resolved LUCENE-7896.
-
Resolution: Fixed

Fixed a long time ago.

> Upgrade to RandomizedRunner 2.5.2
> -
>
> Key: LUCENE-7896
> URL: https://issues.apache.org/jira/browse/LUCENE-7896
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Minor
> Fix For: 7.0
>
> Attachments: LUCENE-7896.patch
>
>
> RR 2.5.2 fixed a nasty error message that gets printed while running tests 
> that is pretty annoying if you have the environment hitting this. Lets 
> upgrade to 2.5.2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 2512 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/2512/

[...truncated 40 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-Tests-master/3072/consoleText

[repro] Revision: f80e8e11672d31c6e12069d2bd12a28b92e5a336

[repro] Repro line:  ant test  -Dtestcase=TestSimTriggerIntegration 
-Dtests.method=testNodeAddedTriggerRestoreState -Dtests.seed=B0607B106BCE1EF3 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=lt 
-Dtests.timezone=America/Cuiaba -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
3be5b59907acc3049121e980bee71df01185d40e
[repro] git fetch
[repro] git checkout f80e8e11672d31c6e12069d2bd12a28b92e5a336

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   TestSimTriggerIntegration
[repro] ant compile-test

[...truncated 3593 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.TestSimTriggerIntegration" -Dtests.showOutput=onerror  
-Dtests.seed=B0607B106BCE1EF3 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=lt -Dtests.timezone=America/Cuiaba -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[...truncated 3128 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   2/5 failed: 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[repro] git checkout 3be5b59907acc3049121e980bee71df01185d40e

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-master - Build # 3073 - Still unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/3073/

2 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
ObjectTracker found 4 object(s) that were not released!!! [SolrCore, 
InternalHttpClient, MMapDirectory, MMapDirectory] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.core.SolrCore  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at org.apache.solr.core.SolrCore.(SolrCore.java:1054)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:874)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1191)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1101)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:394)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:340)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:164)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) 
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
  at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
 at org.eclipse.jetty.server.Server.handle(Server.java:502)  at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at 
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)  
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)  
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) 
 at java.lang.Thread.run(Thread.java:748)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.http.impl.client.InternalHttpClient  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
  at 
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
  at 
org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)  
at org.apache.solr.handler.IndexFetcher.(IndexFetcher.java:267)  at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) 
 at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237) 
 at 
org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)
  at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)  
at 

PreCommit-LUCENE-Build

[Murali Krishna]

Hi,
https://builds.apache.org/job/PreCommit-LUCENE-Build/ seems to be always
waiting for executor. Looks like there are only 2 hosts marked for Lucene,
and Solr builds usually holds them. Who maintains this and can we add more
hosts?

Thanks,
Murali

[jira] [Updated] (LUCENE-8601) Adding attributes to IndexFieldType

[Murali Krishna P (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Murali Krishna P updated LUCENE-8601:
-
Attachment: LUCENE-8601.04.patch

> Adding attributes to IndexFieldType
> ---
>
> Key: LUCENE-8601
> URL: https://issues.apache.org/jira/browse/LUCENE-8601
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 7.5
>Reporter: Murali Krishna P
>Priority: Major
> Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, 
> LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.patch
>
>
> Today, we can write a custom Field using custom IndexFieldType, but when the 
> DefaultIndexingChain converts [IndexFieldType to 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662],
>  only few key informations such as indexing options and doc value type are 
> retained. The [Codec gets the 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90],
>  but not the type details.
>   
>  FieldInfo has support for ['attributes'| 
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47]
>  and it would be great if we can add 'attributes' to IndexFieldType also and 
> copy it to FieldInfo's 'attribute'.
>   
>  This would allow someone to write a custom codec (extending docvalueformat 
> for example) for only the 'special field' that he wants and delegate the rest 
> of the fields to the default codec.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType

[Murali Krishna P (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723651#comment-16723651
 ] 

Murali Krishna P commented on LUCENE-8601:
--

Good catch [~jpountz], fixed it and modified the test: [^LUCENE-8601.04.patch]

> Adding attributes to IndexFieldType
> ---
>
> Key: LUCENE-8601
> URL: https://issues.apache.org/jira/browse/LUCENE-8601
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 7.5
>Reporter: Murali Krishna P
>Priority: Major
> Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, 
> LUCENE-8601.03.patch, LUCENE-8601.04.patch, LUCENE-8601.patch
>
>
> Today, we can write a custom Field using custom IndexFieldType, but when the 
> DefaultIndexingChain converts [IndexFieldType to 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662],
>  only few key informations such as indexing options and doc value type are 
> retained. The [Codec gets the 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90],
>  but not the type details.
>   
>  FieldInfo has support for ['attributes'| 
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47]
>  and it would be great if we can add 'attributes' to IndexFieldType also and 
> copy it to FieldInfo's 'attribute'.
>   
>  This would allow someone to write a custom codec (extending docvalueformat 
> for example) for only the 'special field' that he wants and delegate the rest 
> of the fields to the default codec.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13079) refactor and harden common 'suspend-trigger' patern in autoscaling test setup

[Hoss Man (JIRA)]

Hoss Man created SOLR-13079:
---

 Summary: refactor and harden common 'suspend-trigger' patern in 
autoscaling test setup
 Key: SOLR-13079
 URL: https://issues.apache.org/jira/browse/SOLR-13079
 Project: Solr
  Issue Type: Sub-task
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Hoss Man


A semi-common pattern in autoscaling tests is that immediately after 
configuring the cluster, they (ab)use 
{{AutoScalingHandlerTest.createAutoScalingRequest(...)}} to exec 
{{suspend-trigger}} on {{".scheduled_maintenance"}}.

The problem that can occasionally pops up in jenkins failures is that it's 
possible the OverseerThread hasn't gotten a chance to add this implicit trigger 
to the schedule by the time the test code tries to do this...

{noformat}
   [junit4] ERROR   0.03s J1 | TestSimTriggerIntegration.testCooldown <<<
   [junit4]> Throwable #1: java.io.IOException: 
java.util.concurrent.ExecutionException: java.io.IOException: 
org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, 
errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No 
trigger exists with name: .scheduled_maintenance]}], 
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([51029DC2D3857924:60BCF026AD2F0CD6]:0)
   [junit4]>at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:654)
   [junit4]>at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224)
   [junit4]>at 
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
   [junit4]>at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.setupTest(TestSimTriggerIntegration.java:136)
{noformat}

I think we should:

# refactor the {{AutoScalingHandlerTest.AutoScalingRequest}} static inner class 
into {{CloudTestUtils}}
# add a helper method to {{CloudTestUtils}} to make it easy to wait for a 
trigger to be scheduled
# add a helper method to {{CloudTestUtils}} to make it easy to susspend an 
existing trigger





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-13079) refactor and harden common 'suspend-trigger' patern in autoscaling test setup

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man reassigned SOLR-13079:
---

  Assignee: Hoss Man
Attachment: SOLR-13079.patch

Attaching a patch – it's mostly a lot of rote refactoring.

For #1, Tests that use to look like this...
{code:java}
import static 
org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.createAutoScalingRequest;
...
SolrRequest req = createAutoScalingRequest(SolrRequest.METHOD.POST, ...);
{code}
...now look like this...
{code:java}
import org.apache.solr.cloud.CloudTestUtils.AutoScalingRequest;
...
SolrRequest req = AutoScalingRequest.create(SolrRequest.METHOD.POST, ...);
{code}
For #2 & #3, tests that use to look like something like this...
{code:java}
// disable .scheduled_maintenance
String suspendTriggerCommand = "{" +
"'suspend-trigger' : {'name' : '.scheduled_maintenance'}" +
"}";
SolrRequest req = createAutoScalingRequest(SolrRequest.METHOD.POST, 
suspendTriggerCommand);
SolrClient solrClient = cluster.getSolrClient();
NamedList response = solrClient.request(req);
assertEquals(response.get("result").toString(), "success");
{code}
...now look like this if they are a sim test...
{code:java}
// disable .scheduled_maintenance (once it exists)
CloudTestUtils.waitForTriggerToBeScheduled(cluster, ".scheduled_maintenance");
CloudTestUtils.suspendTrigger(cluster, ".scheduled_maintenance");
{code}
...or this if they use a full MiniSolrCloudCluster...
{code:java}
// disable .scheduled_maintenance (once it exists)
CloudTestUtils.waitForTriggerToBeScheduled(cluster.getOpenOverseer().getSolrCloudManager(),
 ".scheduled_maintenance");
CloudTestUtils.suspendTrigger(cluster.getOpenOverseer().getSolrCloudManager(), 
".scheduled_maintenance");
{code}
...any concerns?

> refactor and harden common 'suspend-trigger' patern in autoscaling test setup
> -
>
> Key: SOLR-13079
> URL: https://issues.apache.org/jira/browse/SOLR-13079
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Attachments: SOLR-13079.patch
>
>
> A semi-common pattern in autoscaling tests is that immediately after 
> configuring the cluster, they (ab)use 
> {{AutoScalingHandlerTest.createAutoScalingRequest(...)}} to exec 
> {{suspend-trigger}} on {{".scheduled_maintenance"}}.
> The problem that can occasionally pops up in jenkins failures is that it's 
> possible the OverseerThread hasn't gotten a chance to add this implicit 
> trigger to the schedule by the time the test code tries to do this...
> {noformat}
>[junit4] ERROR   0.03s J1 | TestSimTriggerIntegration.testCooldown <<<
>[junit4]> Throwable #1: java.io.IOException: 
> java.util.concurrent.ExecutionException: java.io.IOException: 
> org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, 
> errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No 
> trigger exists with name: .scheduled_maintenance]}], 
>[junit4]>at 
> __randomizedtesting.SeedInfo.seed([51029DC2D3857924:60BCF026AD2F0CD6]:0)
>[junit4]>at 
> org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:654)
>[junit4]>at 
> org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224)
>[junit4]>at 
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
>[junit4]>at 
> org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.setupTest(TestSimTriggerIntegration.java:136)
> {noformat}
> I think we should:
> # refactor the {{AutoScalingHandlerTest.AutoScalingRequest}} static inner 
> class into {{CloudTestUtils}}
> # add a helper method to {{CloudTestUtils}} to make it easy to wait for a 
> trigger to be scheduled
> # add a helper method to {{CloudTestUtils}} to make it easy to susspend an 
> existing trigger



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene Branding: the TLP, and "Lucene Java"

[مجید قاراخانی]

On 2007/04/11 02:41:17, Chris Hostetter  wrote:
>
> I was motivated to start this thread by LUCENE-860, but it's been in the>
> back of my mind for a while.>
>
> As the Lucene Top Level Project grows and get's more Sub-Projects, I>
> (personally) have been finding it hard in email/documentation/discussion>
> to clarify when people are refering to the "Lucene" Top Level Project>
> versus the "Lucene" java project.  I can't help but wonder if the TLP>
> should have a different name, or if "Lucene Java" should taken on a more>
> specific name that doesn't just sound like a name followed a langauge -->
> ie: JLucene, LuceneJ ... anything that makes it more clear that when the>
> word "Lucene" is used it's talking about the broder Top Level project>
> address all aspects of OSS Search Software.>
>
> Am I the only one that wonders about this as time goes on?>
>
> -Hoss>
>
>
> ->
> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org>
> For additional commands, e-mail: java-dev-h...@lucene.apache.org>
>
>

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 407 - Still unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/407/

5 tests failed.
FAILED:  org.apache.solr.cloud.LIROnShardRestartTest.testAllReplicasInLIR

Error Message:
Path must not end with / character

Stack Trace:
java.lang.IllegalArgumentException: Path must not end with / character
at 
__randomizedtesting.SeedInfo.seed([3672748A295765FD:6CEA4E4C57D7021A]:0)
at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:58)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1523)
at 
org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346)
at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
at 
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346)
at 
org.apache.solr.cloud.LIROnShardRestartTest.testAllReplicasInLIR(LIROnShardRestartTest.java:168)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 

[jira] [Updated] (SOLR-13061) Solr replica remaining down status when hitting the maxQueueSize as 20000 after Solr servers restarted

[Zhaohui Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhaohui Ma updated SOLR-13061:
--
Environment: 
Cluster info: 6 nodes, 30 Solr servers (5 Solr server per node)

1000 collections, 10 shards per collection, 3 replica per shard

Exception happened when restarting Solr cluster.

  was:
Cluster info: 6 nodes, 30 Solr servers

1000 collections, 10 shards per collection, 3 replica per shard

Exception happened when restarting Solr cluster.


> Solr replica remaining down status when hitting the maxQueueSize as 2 
> after Solr servers restarted
> --
>
> Key: SOLR-13061
> URL: https://issues.apache.org/jira/browse/SOLR-13061
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 7.2, 7.3, 7.3.1, 7.4, 7.5
> Environment: Cluster info: 6 nodes, 30 Solr servers (5 Solr server 
> per node)
> 1000 collections, 10 shards per collection, 3 replica per shard
> Exception happened when restarting Solr cluster.
>Reporter: Zhaohui Ma
>Priority: Blocker
>  Labels: performance
>
> 1. Cluster info: 6 nodes, 30 Solr servers
> 1000 collections, 10 shards per collection, 3 replica per shard.
> Exception happened when restarting Solr cluster.
>  
> 2. Exception happened when restarting Solr cluster. The question is NO 
> exception hander is defined when this exception 
> "java.lang.IllegalStateException: queue is full" is thrown when arriving at 
> the threshold
> STATE_UPDATE_MAX_QUEUE 2 defined in Overseer. And the core fails to 
> preRegister and never come up again.
>  
> 3. Suggestions:
> a. Is this configuration STATE_UPDATE_MAX_QUEUE reasonable? Any plan or risk 
> to enlarge this queue size as 2 is too much small.
> b. Should this configuration STATE_UPDATE_MAX_QUEUE configurable by user? 
> Currently it is hard code in Overseer.java: 
>     public static final int STATE_UPDATE_MAX_QUEUE = 2;
> c. IllegalStateException should be handled and retry logic should be added.
>  
> 4. Detailed error is given as below.
> 2018-12-12 11:20:24,737 | ERROR | 
> coreContainerWorkExecutor-2-thread-1-processing-n:8.5.165.7:21101_solr | 
> Error waiting for SolrCore to be created | 
> org.apache.solr.core.CoreContainer.lambda$load$1(CoreContainer.java:578)
>  java.util.concurrent.ExecutionException: 
> org.apache.solr.common.SolrException: Unable to create core 
> [collection9_shard1_replica3]
>  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>  at org.apache.solr.core.CoreContainer.lambda$load$1(CoreContainer.java:574)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>  at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
>  Caused by: org.apache.solr.common.SolrException: Unable to create core 
> [collection9_shard1_replica3]
>  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1087)
>  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:546)
>  ... 5 more
>  Caused by: java.lang.IllegalStateException: queue is full
>  at 
> org.apache.solr.cloud.ZkDistributedQueue.offer(ZkDistributedQueue.java:311)
>  at org.apache.solr.cloud.ZkController.publish(ZkController.java:1346)
>  at org.apache.solr.cloud.ZkController.publish(ZkController.java:1245)
>  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1634)
>  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1061)
>  ... 6 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12727) Upgrade ZooKeeper dependency to 3.4.13

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723529#comment-16723529
 ] 

ASF subversion and git services commented on SOLR-12727:


Commit 3be5b59907acc3049121e980bee71df01185d40e in lucene-solr's branch 
refs/heads/master from [~cp.erick...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3be5b59 ]

SOLR-12727: Added Dat to the credits, partly as a test of whether infra has 
fixed a commit attribution error


> Upgrade ZooKeeper dependency to 3.4.13
> --
>
> Key: SOLR-12727
> URL: https://issues.apache.org/jira/browse/SOLR-12727
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 7.4
>Reporter: Shawn Heisey
>Assignee: Erick Erickson
>Priority: Major
> Fix For: master (8.0), 7.7
>
> Attachments: SOLR-12727.patch, SOLR-12727.patch, SOLR-12727.patch, 
> SOLR-12727.patch, SOLR-12727.patch, SOLR-12727.patch
>
>
> Upgrade ZK dependency to 3.4.13.  This fixes ZOOKEEPER-2184 which will make 
> the ZK client re-resolve the server hostnames when a connection fails.  This 
> will fix issues where a failed ZK container is replaced with a new one that 
> has a different IP address and DNS gets updated with the new address.
> Typically these upgrades do not require code changes, but that should be 
> verified.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 2212 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/2212/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

11 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at 
__randomizedtesting.SeedInfo.seed([59C58C4D53889DD:2E638D9F4F409C0D]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState(TestSimTriggerIntegration.java:340)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeAddedTriggerRestoreState

Error Message:
java.util.concurrent.ExecutionException: java.io.IOException: 

[jira] [Comment Edited] (SOLR-13076) TimeOut breaks the simulation framework

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723289#comment-16723289
 ] 

Hoss Man edited comment on SOLR-13076 at 12/17/18 11:06 PM:


Once thing we need to watch out for here is that this "fix" may break some of 
the "waitFor.." type code used in tests, where the point of the "wait" logic is 
to give competing threads a chance to finish executing some background logic, 
but if the timesource being used in the TimeOut is based on the _simulated_ 
TimeSource then they might not wait long enoug.

-Example: {{CloudTestUtils.waitForState}} sets up a 90 second timeout to said 
for the specified cluster state to exist, using 
{{cloudManager.getTimeSource()}}  -- prior to [~ab]'s fix, that timeout was 
going to use thread.sleep and was willing to ultimately wait for a "real" 90 
seconds (from the perspective of the test host) but now it will only wait a 
"simulated" 90 seconds, which may not be enough time for a jenkins box with low 
CPU counts.-

Perhaps we need a thorough audit any uses of TimeOut in the _test_ code to 
sanity check if they should be using the "clusters" concept of time (ie: 
waiting for triggers to fire) vs using the "real" concept of time (ie: waiting 
for threads to run)



*UPDATE:* I just realized i missread this comment, and it doesn't affect 
{{CloudTestUtils.waitForState}} the way i thought at all ... i still have some 
concerns that this "fix" might expose other bugs in tests that are using 
{{waitFor}} on the cluster's TimeSource, when they should be using NANO_TIME, 
to give background threads time to process things ... but we'll just have to 
see what shakes out.


was (Author: hossman):
Once thing we need to watch out for here is that this "fix" may break some of 
the "waitFor.." type code used in tests, where the point of the "wait" logic is 
to give competing threads a chance to finish executing some background logic, 
but if the timesource being used in the TimeOut is based on the _simulated_ 
TimeSource then they might not wait long enoug.

Example: {{CloudTestUtils.waitForState}} sets up a 90 second timeout to said 
for the specified cluster state to exist, using 
{{cloudManager.getTimeSource()}}  -- prior to [~ab]'s fix, that timeout was 
going to use thread.sleep and was willing to ultimately wait for a "real" 90 
seconds (from the perspective of the test host) but now it will only wait a 
"simulated" 90 seconds, which may not be enough time for a jenkins box with low 
CPU counts.

Perhaps we need a thorough audit any uses of TimeOut in the _test_ code to 
sanity check if they should be using the "clusters" concept of time (ie: 
waiting for triggers to fire) vs using the "real" concept of time (ie: waiting 
for threads to run)

> TimeOut breaks the simulation framework
> ---
>
> Key: SOLR-13076
> URL: https://issues.apache.org/jira/browse/SOLR-13076
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Affects Versions: 7.6, master (8.0)
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: master (8.0), 7.7
>
>
> {{TimeOut}} uses actual {{Thread.sleep}} in its {{waitFor}} method instead of 
> using {{TimeSource.sleep}}. This breaks the simulation framework, which often 
> uses a non-real time {{TimeSource}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #528: SOLR-12955 2

[barrotsteindev]

Github user barrotsteindev commented on the issue:

https://github.com/apache/lucene-solr/pull/528
  
Oh yes,
I still have a few Todo's I added that I'd like to tackle.
I won't have time for more today,
Hopefully I'll be able to get these done by the end of the week.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #528: SOLR-12955 2

[dsmiley]

Github user dsmiley commented on the issue:

https://github.com/apache/lucene-solr/pull/528
  
Looks good.  More to do?


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr 8.0

[David Smiley]

I see 10 JIRA issues matching this filter:   project in (SOLR, LUCENE) AND
priority = Blocker and status = open and fixVersion = "master (8.0)"
   click here:
https://issues.apache.org/jira/issues/?jql=project%20in%20(SOLR%2C%20LUCENE)%20AND%20priority%20%3D%20Blocker%20and%20status%20%3D%20open%20and%20fixVersion%20%3D%20%22master%20(8.0)%22%20

Thru the end of the month, I intend to work on those issues not yet
assigned.

On Mon, Dec 17, 2018 at 4:51 AM Adrien Grand  wrote:

> +1
>
> On Mon, Dec 17, 2018 at 10:38 AM Alan Woodward 
> wrote:
> >
> > Hi all,
> >
> > Now that 7.6 is out of the door (thanks Nick!) we should think about
> cutting the 8.0 branch and moving master to 9.0.  I’ll volunteer to create
> the branch this week - say Wednesday?  Then we should have some time to
> clean up the master branch and uncover anything that still needs to be done
> on 8.0 before we start the release process next year.
> >
> > On 22 Oct 2018, at 18:12, Cassandra Targett 
> wrote:
> >
> > I'm a bit delayed, but +1 on the 7.6 and 8.0 plan from me too.
> >
> > On Fri, Oct 19, 2018 at 7:18 AM Erick Erickson 
> wrote:
> >>
> >> +1, this gives us all a chance to prioritize getting the blockers out
> >> of the way in a careful manner.
> >> On Fri, Oct 19, 2018 at 7:56 AM jim ferenczi 
> wrote:
> >> >
> >> > +1 too. With this new perspective we could create the branch just
> after the 7.6 release and target the 8.0 release for January 2019 which
> gives almost 3 month to finish the blockers ?
> >> >
> >> > Le jeu. 18 oct. 2018 à 23:56, David Smiley 
> a écrit :
> >> >>
> >> >> +1 to a 7.6 —lots of stuff in there
> >> >> On Thu, Oct 18, 2018 at 4:47 PM Nicholas Knize 
> wrote:
> >> >>>
> >> >>> If we're planning to postpone cutting an 8.0 branch until a few
> weeks from now then I'd like to propose (and volunteer to RM) a 7.6 release
> targeted for late November or early December (following the typical 2 month
> release pattern). It feels like this might give a little breathing room for
> finishing up 8.0 blockers? And looking at the change log there appear to be
> a healthy list of features, bug fixes, and improvements to both Solr and
> Lucene that warrant a 7.6 release? Personally I wouldn't mind releasing the
> LatLonShape encoding changes in LUCENE-8521 and selective indexing work
> done in LUCENE-8496. Any objections or thoughts?
> >> >>>
> >> >>> - Nick
> >> >>>
> >> >>>
> >> >>> On Thu, Oct 18, 2018 at 5:32 AM Đạt Cao Mạnh <
> caomanhdat...@gmail.com> wrote:
> >> 
> >>  Thanks Cassandra and Jim,
> >> 
> >>  I created a blocker issue for Solr 8.0 SOLR-12883, currently in
> jira/http2 branch there are a draft-unmature implementation of SPNEGO
> authentication which enough to makes the test pass, this implementation
> will be removed when SOLR-12883 gets resolved . Therefore I don't see any
> problem on merging jira/http2 to master branch in the next week.
> >> 
> >>  On Thu, Oct 18, 2018 at 2:33 AM jim ferenczi <
> jim.feren...@gmail.com> wrote:
> >> >
> >> > > But if you're working with a different assumption - that just
> the existence of the branch does not stop Dat from still merging his work
> and the work being included in 8.0 - then I agree, waiting for him to merge
> doesn't need to stop the creation of the branch.
> >> >
> >> > Yes that's my reasoning. This issue is a blocker so we won't
> release without it but we can work on the branch in the meantime and let
> other people work on new features that are not targeted to 8.
> >> >
> >> > Le mer. 17 oct. 2018 à 20:51, Cassandra Targett <
> casstarg...@gmail.com> a écrit :
> >> >>
> >> >> OK - I was making an assumption that the timeline for the first
> 8.0 RC would be ASAP after the branch is created.
> >> >>
> >> >> It's a common perception that making a branch freezes adding new
> features to the release, perhaps in an unofficial way (more of a courtesy
> rather than a rule). But if you're working with a different assumption -
> that just the existence of the branch does not stop Dat from still merging
> his work and the work being included in 8.0 - then I agree, waiting for him
> to merge doesn't need to stop the creation of the branch.
> >> >>
> >> >> If, however, once the branch is there people object to Dat
> merging his work because it's "too late", then the branch shouldn't be
> created yet because we want to really try to clear that blocker for 8.0.
> >> >>
> >> >> Cassandra
> >> >>
> >> >> On Wed, Oct 17, 2018 at 12:13 PM jim ferenczi <
> jim.feren...@gmail.com> wrote:
> >> >>>
> >> >>> Ok thanks for answering.
> >> >>>
> >> >>> > - I think Solr needs a couple more weeks since the work Dat
> is doing isn't quite done yet.
> >> >>>
> >> >>> We can wait a few more weeks to create the branch but I don't
> think that one action (creating the branch) prevents the other (the work
> Dat is doing).
> >> >>> HTTP/2 is one of the blocker for 

[GitHub] lucene-solr issue #528: SOLR-12955 2

[barrotsteindev]

Github user barrotsteindev commented on the issue:

https://github.com/apache/lucene-solr/pull/528
  
addressed comments about previous PR and re-based on latest master


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8581) Change LatLonShape encoding to use 4 BYTES Per Dimension

[Nicholas Knize (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723352#comment-16723352
 ] 

Nicholas Knize commented on LUCENE-8581:


Everything's looking good. +1 to merge once these last two changes are in place 
/cc [~ivera]

> Change LatLonShape encoding to use 4 BYTES Per Dimension
> 
>
> Key: LUCENE-8581
> URL: https://issues.apache.org/jira/browse/LUCENE-8581
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, 
> LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch
>
>
> {{LatLonShape}} tessellated triangles currently use a relatively naive 
> encoding with the first four dimensions as the bounding box of the triangle 
> and the last three dimensions as the vertices of the triangle. To encode the 
> {{x,y}} vertices in the last three dimensions requires {{bytesPerDim}} to be 
> set to 8, with 4 bytes for the x & y axis, respectively. We can reduce 
> {{bytesPerDim}} to 4 by encoding the index(es) of the vertices shared by the 
> bounding box along with the orientation of the triangle. This also opens the 
> door for supporting {{CONTAINS}} queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-13078) Harden TestSimNodeAddedTrigger

[Jason Gerlowski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gerlowski reassigned SOLR-13078:
--

Assignee: Jason Gerlowski

> Harden TestSimNodeAddedTrigger
> --
>
> Key: SOLR-13078
> URL: https://issues.apache.org/jira/browse/SOLR-13078
> Project: Solr
>  Issue Type: Test
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
>
> Jenkins has been failing occasionally with issues in TestSimNodeAddedTrigger. 
>  We should look into these and make it pass more reliably.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13078) Harden TestSimNodeAddedTrigger

[Jason Gerlowski (JIRA)]

Jason Gerlowski created SOLR-13078:
--

 Summary: Harden TestSimNodeAddedTrigger
 Key: SOLR-13078
 URL: https://issues.apache.org/jira/browse/SOLR-13078
 Project: Solr
  Issue Type: Test
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Jason Gerlowski


Jenkins has been failing occasionally with issues in TestSimNodeAddedTrigger.  
We should look into these and make it pass more reliably.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-8610) NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated token Attributes

[Michael Gibney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Gibney resolved LUCENE-8610.

   Resolution: Not A Bug
Lucene Fields:   (was: New,Patch Available)

Will look at addressing this in {{solr.PreAnalyzedAnalyzer}}. There may be 
other {{TokenStreams}} that don't strictly follow the requirement to initialize 
all token Attributes on instantiation, but as of now nothing relating to this 
issue is known to be broken.

> NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated 
> token Attributes
> -
>
> Key: LUCENE-8610
> URL: https://issues.apache.org/jira/browse/LUCENE-8610
> Project: Lucene - Core
>  Issue Type: Wish
>  Components: core/index
>Affects Versions: 7.4, master (8.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: LUCENE-8610.patch
>
>
> {{DefaultIndexingChain.invert(...)}} calls 
> {{invertState.setAttributeSource(stream)}} before {{stream.incrementToken()}} 
> is called.
> For instances of {{stream}} that lazily instantiate token attributes (e.g., 
> as {{solr.PreAnalyzedField$PreAnalyzedTokenizer}} does upon the first call to 
> {{incrementToken()}} that returns {{true}}), this can result in caching a 
> {{null}} value in {{invertState.termAttribute}} for a given {{stream}} 
> instance.
> Subsequent calls that reuse the same {{stream}} instance (reusing 
> {{TokenStreamComponents}}) for field values with at least 1 token will call 
> {{termHashPerField.start(...)}} which sets {{termsHashPerField.termAtt}} from 
> the {{null}} value cached in the {{FieldInvertState.termAttribute}}. An NPE 
> would be thrown when {{termsHashPerField.add()}} reasonably but incorrectly 
> assumes a non-null value for {{termAtt}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.6-Windows (64bit/jdk-9.0.4) - Build # 24 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Windows/24/
Java: 64bit/jdk-9.0.4 -XX:+UseCompressedOops -XX:+UseParallelGC

5 tests failed.
FAILED:  org.apache.solr.cloud.MoveReplicaTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
at 
__randomizedtesting.SeedInfo.seed([A9FF3BAEB8A6EB34:21AB0474165A86CC]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertFalse(Assert.java:68)
at org.junit.Assert.assertFalse(Assert.java:79)
at org.apache.solr.cloud.MoveReplicaTest.test(MoveReplicaTest.java:146)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  
org.apache.solr.cloud.autoscaling.TriggerSetPropertiesIntegrationTest.testSetProperties

Error Message:


Stack Trace:
java.lang.AssertionError
at 
__randomizedtesting.SeedInfo.seed([A9FF3BAEB8A6EB34:C29BECE70B8B7EB0]:0)
at org.junit.Assert.fail(Assert.java:92)
at 

[jira] [Commented] (SOLR-13076) TimeOut breaks the simulation framework

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723289#comment-16723289
 ] 

Hoss Man commented on SOLR-13076:
-

Once thing we need to watch out for here is that this "fix" may break some of 
the "waitFor.." type code used in tests, where the point of the "wait" logic is 
to give competing threads a chance to finish executing some background logic, 
but if the timesource being used in the TimeOut is based on the _simulated_ 
TimeSource then they might not wait long enoug.

Example: {{CloudTestUtils.waitForState}} sets up a 90 second timeout to said 
for the specified cluster state to exist, using 
{{cloudManager.getTimeSource()}}  -- prior to [~ab]'s fix, that timeout was 
going to use thread.sleep and was willing to ultimately wait for a "real" 90 
seconds (from the perspective of the test host) but now it will only wait a 
"simulated" 90 seconds, which may not be enough time for a jenkins box with low 
CPU counts.

Perhaps we need a thorough audit any uses of TimeOut in the _test_ code to 
sanity check if they should be using the "clusters" concept of time (ie: 
waiting for triggers to fire) vs using the "real" concept of time (ie: waiting 
for threads to run)

> TimeOut breaks the simulation framework
> ---
>
> Key: SOLR-13076
> URL: https://issues.apache.org/jira/browse/SOLR-13076
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Affects Versions: 7.6, master (8.0)
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: master (8.0), 7.7
>
>
> {{TimeOut}} uses actual {{Thread.sleep}} in its {{waitFor}} method instead of 
> using {{TimeSource.sleep}}. This breaks the simulation framework, which often 
> uses a non-real time {{TimeSource}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8610) NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated token Attributes

[Michael Gibney (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723263#comment-16723263
 ] 

Michael Gibney commented on LUCENE-8610:


"'Must' implies a requirement" – yes, that's definitely how I initially read 
it. I guess any potential confusion would depend on how one reads the preceding 
"To make sure that filters and consumers know which attributes are available":
 1. if read as an _explanation_, then "must" implies a requirement
 2. if read as a _condition_, then "must" is a qualified requirement (or 
suggestion/warning)

Thanks for the clarification, and I'll see if I can take a stab at bringing 
{{PreAnalyzedTokenizer}} into line with the requirement.

> NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated 
> token Attributes
> -
>
> Key: LUCENE-8610
> URL: https://issues.apache.org/jira/browse/LUCENE-8610
> Project: Lucene - Core
>  Issue Type: Wish
>  Components: core/index
>Affects Versions: 7.4, master (8.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: LUCENE-8610.patch
>
>
> {{DefaultIndexingChain.invert(...)}} calls 
> {{invertState.setAttributeSource(stream)}} before {{stream.incrementToken()}} 
> is called.
> For instances of {{stream}} that lazily instantiate token attributes (e.g., 
> as {{solr.PreAnalyzedField$PreAnalyzedTokenizer}} does upon the first call to 
> {{incrementToken()}} that returns {{true}}), this can result in caching a 
> {{null}} value in {{invertState.termAttribute}} for a given {{stream}} 
> instance.
> Subsequent calls that reuse the same {{stream}} instance (reusing 
> {{TokenStreamComponents}}) for field values with at least 1 token will call 
> {{termHashPerField.start(...)}} which sets {{termsHashPerField.termAtt}} from 
> the {{null}} value cached in the {{FieldInvertState.termAttribute}}. An NPE 
> would be thrown when {{termsHashPerField.add()}} reasonably but incorrectly 
> assumes a non-null value for {{termAtt}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13077) PreAnalyzedField TokenStreamComponents should be reusable

[Michael Gibney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Gibney updated SOLR-13077:
--
Description: 
{{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated from 
scratch for every field value.

This is necessary at the moment because the current implementation has no a 
priori knowledge about the schema/TokenStream that it's deserializing – 
Attributes are implicit in the serialized token stream, and token Attributes 
are lazily instantiated in {{incrementToken()}}.

Reuse of {{TokenStreamComponents}} with the current implementation would at a 
minimum cause problems at index time, when Attributes are cached in indexing 
components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For 
instance, if the first field encountered has no value specified for 
{{PayloadAttribute}}, a {{null}} value would be cached for that 
{{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that 
{{AttributeSource}} were to be reused for a field that _does_ specify a 
{{PayloadAttribute}}, indexing components would "consult" the cached {{null}} 
value, and the payload (and all subsequent payloads) would be silently ignored 
(not indexed).

This is not exactly _broken_ currently, but I gather it's an unorthodox 
implementation of {{TokenStream}}, and the current workaround of disabling 
{{TokenStreamComponents}} reuse necessarily adds to object creation and GC 
overhead.

For reference (and see LUCENE-8610), the [TokenStream 
API|https://lucene.apache.org/core/7_5_0/core/org/apache/lucene/analysis/TokenStream.html]
 says:
{quote}To make sure that filters and consumers know which attributes are 
available, the attributes must be added during instantiation.
{quote}

  was:
{{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated from 
scratch for every field value.

This is necessary at the moment because the current implementation has no a 
priori knowledge about the schema/TokenStream that it's deserializing -- 
Attributes are implicit in the serialized token stream, and token Attributes 
are lazily instantiated in {{incrementToken()}}.

Reuse of {{TokenStreamComponents}} with the current implementation would at a 
minimum cause problems at index time, when Attributes are cached in indexing 
components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For 
instance, if the first field encountered has no value specified for 
{{PayloadAttribute}}, a {{null}} value would be cached for that 
{{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that 
{{AttributeSource}} were to be reused for a field that _does_ specify a 
{{PayloadAttribute}}, indexing components would "consult" the cached {{null}} 
value, and the payload (and all subsequent payloads) would be silently ignored 
(not indexed).

This is not exactly _broken_ currently, but I gather it's an unorthodox 
implementation of {{TokenStream}}, and the current workaround of disabling 
{{TokenStreamComponents}} reuse necessarily adds to object creation and GC 
overhead.

For reference, the [TokenStream 
API|https://lucene.apache.org/core/7_5_0/core/org/apache/lucene/analysis/TokenStream.html]
 says:
bq.To make sure that filters and consumers know which attributes are available, 
the attributes must be added during instantiation.


> PreAnalyzedField TokenStreamComponents should be reusable
> -
>
> Key: SOLR-13077
> URL: https://issues.apache.org/jira/browse/SOLR-13077
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Reporter: Michael Gibney
>Priority: Minor
>
> {{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated 
> from scratch for every field value.
> This is necessary at the moment because the current implementation has no a 
> priori knowledge about the schema/TokenStream that it's deserializing – 
> Attributes are implicit in the serialized token stream, and token Attributes 
> are lazily instantiated in {{incrementToken()}}.
> Reuse of {{TokenStreamComponents}} with the current implementation would at a 
> minimum cause problems at index time, when Attributes are cached in indexing 
> components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For 
> instance, if the first field encountered has no value specified for 
> {{PayloadAttribute}}, a {{null}} value would be cached for that 
> {{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that 
> {{AttributeSource}} were to be reused for a field that _does_ specify a 
> {{PayloadAttribute}}, indexing components would "consult" the cached {{null}} 
> value, and the payload (and all subsequent payloads) would be silently 
> ignored (not 

[jira] [Resolved] (SOLR-13043) clean up suspicious ExecutorService lifecycles in MiniSolrCloudCluster

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man resolved SOLR-13043.
-
   Resolution: Fixed
Fix Version/s: 7.7
   master (8.0)

> clean up suspicious ExecutorService lifecycles in MiniSolrCloudCluster
> --
>
> Key: SOLR-13043
> URL: https://issues.apache.org/jira/browse/SOLR-13043
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Fix For: master (8.0), 7.7
>
> Attachments: SOLR-13043.patch
>
>
> The two ExecutorService's in MiniSolrCloudCluster have lifecycles that looks 
> fishy...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8581) Change LatLonShape encoding to use 4 BYTES Per Dimension

[Nicholas Knize (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723224#comment-16723224
 ] 

Nicholas Knize commented on LUCENE-8581:


{quote}Then let's also rename constants so that they enumerate coordinates in 
CCW order
{quote}
+1

 

> Change LatLonShape encoding to use 4 BYTES Per Dimension
> 
>
> Key: LUCENE-8581
> URL: https://issues.apache.org/jira/browse/LUCENE-8581
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, 
> LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch
>
>
> {{LatLonShape}} tessellated triangles currently use a relatively naive 
> encoding with the first four dimensions as the bounding box of the triangle 
> and the last three dimensions as the vertices of the triangle. To encode the 
> {{x,y}} vertices in the last three dimensions requires {{bytesPerDim}} to be 
> set to 8, with 4 bytes for the x & y axis, respectively. We can reduce 
> {{bytesPerDim}} to 4 by encoding the index(es) of the vertices shared by the 
> bounding box along with the orientation of the triangle. This also opens the 
> door for supporting {{CONTAINS}} queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8581) Change LatLonShape encoding to use 4 BYTES Per Dimension

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723189#comment-16723189
 ] 

Adrien Grand commented on LUCENE-8581:
--

Then let's also rename constants so that they enumerate coordinates in CCW 
order, ie. Y_MINX_MINY_MAXX_MAXY_X rather than Y_MINX_MAXY_X_MINY_MAXX?

> Change LatLonShape encoding to use 4 BYTES Per Dimension
> 
>
> Key: LUCENE-8581
> URL: https://issues.apache.org/jira/browse/LUCENE-8581
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, 
> LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch
>
>
> {{LatLonShape}} tessellated triangles currently use a relatively naive 
> encoding with the first four dimensions as the bounding box of the triangle 
> and the last three dimensions as the vertices of the triangle. To encode the 
> {{x,y}} vertices in the last three dimensions requires {{bytesPerDim}} to be 
> set to 8, with 4 bytes for the x & y axis, respectively. We can reduce 
> {{bytesPerDim}} to 4 by encoding the index(es) of the vertices shared by the 
> bounding box along with the orientation of the triangle. This also opens the 
> door for supporting {{CONTAINS}} queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8581) Change LatLonShape encoding to use 4 BYTES Per Dimension

[Nicholas Knize (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723186#comment-16723186
 ] 

Nicholas Knize commented on LUCENE-8581:


{quote}I got you, new patch rotates edges and indeed simplifies the logic.
{quote}
++1 this is great
{quote}whether to go with CW or CCW order?
{quote}
Right hand rule is the typical convention and part of the standard. So let's go 
w/ CCW.

> Change LatLonShape encoding to use 4 BYTES Per Dimension
> 
>
> Key: LUCENE-8581
> URL: https://issues.apache.org/jira/browse/LUCENE-8581
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, 
> LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch
>
>
> {{LatLonShape}} tessellated triangles currently use a relatively naive 
> encoding with the first four dimensions as the bounding box of the triangle 
> and the last three dimensions as the vertices of the triangle. To encode the 
> {{x,y}} vertices in the last three dimensions requires {{bytesPerDim}} to be 
> set to 8, with 4 bytes for the x & y axis, respectively. We can reduce 
> {{bytesPerDim}} to 4 by encoding the index(es) of the vertices shared by the 
> bounding box along with the orientation of the triangle. This also opens the 
> door for supporting {{CONTAINS}} queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8610) NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated token Attributes

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723172#comment-16723172
 ] 

Alan Woodward commented on LUCENE-8610:
---

'Must' implies a requirement, but I don't think we actually enforce that 
anywhere - that may be something to look into as well.

> NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated 
> token Attributes
> -
>
> Key: LUCENE-8610
> URL: https://issues.apache.org/jira/browse/LUCENE-8610
> Project: Lucene - Core
>  Issue Type: Wish
>  Components: core/index
>Affects Versions: 7.4, master (8.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: LUCENE-8610.patch
>
>
> {{DefaultIndexingChain.invert(...)}} calls 
> {{invertState.setAttributeSource(stream)}} before {{stream.incrementToken()}} 
> is called.
> For instances of {{stream}} that lazily instantiate token attributes (e.g., 
> as {{solr.PreAnalyzedField$PreAnalyzedTokenizer}} does upon the first call to 
> {{incrementToken()}} that returns {{true}}), this can result in caching a 
> {{null}} value in {{invertState.termAttribute}} for a given {{stream}} 
> instance.
> Subsequent calls that reuse the same {{stream}} instance (reusing 
> {{TokenStreamComponents}}) for field values with at least 1 token will call 
> {{termHashPerField.start(...)}} which sets {{termsHashPerField.termAtt}} from 
> the {{null}} value cached in the {{FieldInvertState.termAttribute}}. An NPE 
> would be thrown when {{termsHashPerField.add()}} reasonably but incorrectly 
> assumes a non-null value for {{termAtt}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8610) NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated token Attributes

[Michael Gibney (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723164#comment-16723164
 ] 

Michael Gibney commented on LUCENE-8610:


Thanks, [~romseygeek]! Created SOLR-13077.

I do see that the [TokenStream 
API|https://lucene.apache.org/core/7_6_0/core/org/apache/lucene/analysis/TokenStream.html]
 says:
{quote}To make sure that filters and consumers know which attributes are 
available, the attributes must be added during instantiation.
{quote}

Is this a requirement, or a recommendation? It reads a bit like a requirement; 
but you could also read it as being a recommendation and a warning ("if you 
*do* add attributes after instantiation, filters and consumers might not know 
which attributes are available, so proceed at your own risk")

> NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated 
> token Attributes
> -
>
> Key: LUCENE-8610
> URL: https://issues.apache.org/jira/browse/LUCENE-8610
> Project: Lucene - Core
>  Issue Type: Wish
>  Components: core/index
>Affects Versions: 7.4, master (8.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: LUCENE-8610.patch
>
>
> {{DefaultIndexingChain.invert(...)}} calls 
> {{invertState.setAttributeSource(stream)}} before {{stream.incrementToken()}} 
> is called.
> For instances of {{stream}} that lazily instantiate token attributes (e.g., 
> as {{solr.PreAnalyzedField$PreAnalyzedTokenizer}} does upon the first call to 
> {{incrementToken()}} that returns {{true}}), this can result in caching a 
> {{null}} value in {{invertState.termAttribute}} for a given {{stream}} 
> instance.
> Subsequent calls that reuse the same {{stream}} instance (reusing 
> {{TokenStreamComponents}}) for field values with at least 1 token will call 
> {{termHashPerField.start(...)}} which sets {{termsHashPerField.termAtt}} from 
> the {{null}} value cached in the {{FieldInvertState.termAttribute}}. An NPE 
> would be thrown when {{termsHashPerField.add()}} reasonably but incorrectly 
> assumes a non-null value for {{termAtt}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13077) PreAnalyzedField TokenStreamComponents should be reusable

[Michael Gibney (JIRA)]

Michael Gibney created SOLR-13077:
-

 Summary: PreAnalyzedField TokenStreamComponents should be reusable
 Key: SOLR-13077
 URL: https://issues.apache.org/jira/browse/SOLR-13077
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: Schema and Analysis
Reporter: Michael Gibney


{{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated from 
scratch for every field value.

This is necessary at the moment because the current implementation has no a 
priori knowledge about the schema/TokenStream that it's deserializing -- 
Attributes are implicit in the serialized token stream, and token Attributes 
are lazily instantiated in {{incrementToken()}}.

Reuse of {{TokenStreamComponents}} with the current implementation would at a 
minimum cause problems at index time, when Attributes are cached in indexing 
components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For 
instance, if the first field encountered has no value specified for 
{{PayloadAttribute}}, a {{null}} value would be cached for that 
{{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that 
{{AttributeSource}} were to be reused for a field that _does_ specify a 
{{PayloadAttribute}}, indexing components would "consult" the cached {{null}} 
value, and the payload (and all subsequent payloads) would be silently ignored 
(not indexed).

This is not exactly _broken_ currently, but I gather it's an unorthodox 
implementation of {{TokenStream}}, and the current workaround of disabling 
{{TokenStreamComponents}} reuse necessarily adds to object creation and GC 
overhead.

For reference, the [TokenStream 
API|https://lucene.apache.org/core/7_5_0/core/org/apache/lucene/analysis/TokenStream.html]
 says:
bq.To make sure that filters and consumers know which attributes are available, 
the attributes must be added during instantiation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-13076) TimeOut breaks the simulation framework

[Andrzej Bialecki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrzej Bialecki  resolved SOLR-13076.
--
   Resolution: Fixed
Fix Version/s: 7.7
   master (8.0)

> TimeOut breaks the simulation framework
> ---
>
> Key: SOLR-13076
> URL: https://issues.apache.org/jira/browse/SOLR-13076
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Affects Versions: 7.6, master (8.0)
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: master (8.0), 7.7
>
>
> {{TimeOut}} uses actual {{Thread.sleep}} in its {{waitFor}} method instead of 
> using {{TimeSource.sleep}}. This breaks the simulation framework, which often 
> uses a non-real time {{TimeSource}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13076) TimeOut breaks the simulation framework

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723130#comment-16723130
 ] 

ASF subversion and git services commented on SOLR-13076:


Commit 462adbc7f639839f2628202c6281ecdab223c9c8 in lucene-solr's branch 
refs/heads/branch_7x from Andrzej Bialecki
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=462adbc ]

SOLR-13076: TimeOut breaks the simulation framework.


> TimeOut breaks the simulation framework
> ---
>
> Key: SOLR-13076
> URL: https://issues.apache.org/jira/browse/SOLR-13076
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Affects Versions: 7.6, master (8.0)
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: master (8.0), 7.7
>
>
> {{TimeOut}} uses actual {{Thread.sleep}} in its {{waitFor}} method instead of 
> using {{TimeSource.sleep}}. This breaks the simulation framework, which often 
> uses a non-real time {{TimeSource}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13076) TimeOut breaks the simulation framework

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723100#comment-16723100
 ] 

ASF subversion and git services commented on SOLR-13076:


Commit f5479383b1613966ca5de59876603228b5de6b09 in lucene-solr's branch 
refs/heads/master from Andrzej Bialecki
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f547938 ]

SOLR-13076: TimeOut breaks the simulation framework.


> TimeOut breaks the simulation framework
> ---
>
> Key: SOLR-13076
> URL: https://issues.apache.org/jira/browse/SOLR-13076
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Affects Versions: 7.6, master (8.0)
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
>
> {{TimeOut}} uses actual {{Thread.sleep}} in its {{waitFor}} method instead of 
> using {{TimeSource.sleep}}. This breaks the simulation framework, which often 
> uses a non-real time {{TimeSource}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8612) Add the ability to enforce gaps between intervals

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723085#comment-16723085
 ] 

Alan Woodward commented on LUCENE-8612:
---

One thing I haven't added here but that we may want to think about is combining 
the 'before' count with a positional filter, so that a 
{{Intervals.extend(Intervals.term("b", 10, 0))}} won't match any instances of 
{{b}} that appear before position 10.

> Add the ability to enforce gaps between intervals
> -
>
> Key: LUCENE-8612
> URL: https://issues.apache.org/jira/browse/LUCENE-8612
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8612.patch
>
>
> At the moment you can search for intervals with a maximum number of positions 
> between them, but you cannot enforce gaps.  It would be useful to be able to 
> search for `a b [2 spaces] c`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13076) TimeOut breaks the simulation framework

[Andrzej Bialecki (JIRA)]

Andrzej Bialecki  created SOLR-13076:


 Summary: TimeOut breaks the simulation framework
 Key: SOLR-13076
 URL: https://issues.apache.org/jira/browse/SOLR-13076
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: AutoScaling
Affects Versions: 7.6, master (8.0)
Reporter: Andrzej Bialecki 
Assignee: Andrzej Bialecki 


{{TimeOut}} uses actual {{Thread.sleep}} in its {{waitFor}} method instead of 
using {{TimeSource.sleep}}. This breaks the simulation framework, which often 
uses a non-real time {{TimeSource}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8612) Add the ability to enforce gaps between intervals

[Alan Woodward (JIRA)]

Alan Woodward created LUCENE-8612:
-

 Summary: Add the ability to enforce gaps between intervals
 Key: LUCENE-8612
 URL: https://issues.apache.org/jira/browse/LUCENE-8612
 Project: Lucene - Core
  Issue Type: Task
Reporter: Alan Woodward
Assignee: Alan Woodward
 Attachments: LUCENE-8612.patch

At the moment you can search for intervals with a maximum number of positions 
between them, but you cannot enforce gaps.  It would be useful to be able to 
search for `a b [2 spaces] c`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8612) Add the ability to enforce gaps between intervals

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723044#comment-16723044
 ] 

Alan Woodward commented on LUCENE-8612:
---

Here is a patch adding {{Intervals.extend(source, before, after)}}, which 
allows you to make an interval cover extra positions before and after the 
actual terms.  To search for {{a [space] b [space] c}}, for example, you could 
call {{Intervals.phrase(Intervals.term("a"), 
Intervals.extend(Intervals.term("b", 1, 1)), Intervals.term("c")}}

The patch also clarifies the javadocs for how IntervalIterator should report 
start() and end() after the interval has been exhausted for the current 
document (important for the unordered algorithm), and fixes IntervalMatches to 
report start and end positions from its underlying IntervalIterator, rather 
than any wrapped Matches - for most iterators they are the same, but for 
ExtendedIntervalIterator we need to report the extended bounds so that phrase 
queries will still correctly report matches.

> Add the ability to enforce gaps between intervals
> -
>
> Key: LUCENE-8612
> URL: https://issues.apache.org/jira/browse/LUCENE-8612
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8612.patch
>
>
> At the moment you can search for intervals with a maximum number of positions 
> between them, but you cannot enforce gaps.  It would be useful to be able to 
> search for `a b [2 spaces] c`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8612) Add the ability to enforce gaps between intervals

[Alan Woodward (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward updated LUCENE-8612:
--
Attachment: LUCENE-8612.patch

> Add the ability to enforce gaps between intervals
> -
>
> Key: LUCENE-8612
> URL: https://issues.apache.org/jira/browse/LUCENE-8612
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8612.patch
>
>
> At the moment you can search for intervals with a maximum number of positions 
> between them, but you cannot enforce gaps.  It would be useful to be able to 
> search for `a b [2 spaces] c`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723036#comment-16723036
 ] 

Adrien Grand commented on LUCENE-8601:
--

Should the copy constructor of FieldType clone the attributes map?

> Adding attributes to IndexFieldType
> ---
>
> Key: LUCENE-8601
> URL: https://issues.apache.org/jira/browse/LUCENE-8601
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 7.5
>Reporter: Murali Krishna P
>Priority: Major
> Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, 
> LUCENE-8601.03.patch, LUCENE-8601.patch
>
>
> Today, we can write a custom Field using custom IndexFieldType, but when the 
> DefaultIndexingChain converts [IndexFieldType to 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662],
>  only few key informations such as indexing options and doc value type are 
> retained. The [Codec gets the 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90],
>  but not the type details.
>   
>  FieldInfo has support for ['attributes'| 
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47]
>  and it would be great if we can add 'attributes' to IndexFieldType also and 
> copy it to FieldInfo's 'attribute'.
>   
>  This would allow someone to write a custom codec (extending docvalueformat 
> for example) for only the 'special field' that he wants and delegate the rest 
> of the fields to the default codec.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-7.6 - Build # 112 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.6/112/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testBasics

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
at 
__randomizedtesting.SeedInfo.seed([4AAFD49A7BC1FFF9:7AB6432FA189]:0)
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:252)
at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:49)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:525)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$200(AbstractPollingIoAcceptor.java:67)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:409)
at 
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13398 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberosAlt
   [junit4]   2> Creating dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.6/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberosAlt_4AAFD49A7BC1FFF9-001/init-core-data-001
   [junit4]   2> 1412416 WARN  
(SUITE-TestSolrCloudWithKerberosAlt-seed#[4AAFD49A7BC1FFF9]-worker) [] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=233 numCloses=233
   [junit4]   2> 1412416 INFO  
(SUITE-TestSolrCloudWithKerberosAlt-seed#[4AAFD49A7BC1FFF9]-worker) [] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1412417 INFO  
(SUITE-TestSolrCloudWithKerberosAlt-seed#[4AAFD49A7BC1FFF9]-worker) [] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1412418 INFO  
(SUITE-TestSolrCloudWithKerberosAlt-seed#[4AAFD49A7BC1FFF9]-worker) [] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1412452 INFO  
(TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[4AAFD49A7BC1FFF9]) [] 
o.a.s.SolrTestCaseJ4 ###Starting testBasics
   [junit4]   2> 1441200 WARN  
(TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[4AAFD49A7BC1FFF9]) [] 
o.a.d.s.c.DefaultDirectoryService You didn't change the admin password of 
directory service instance 'DefaultKrbServer'.  Please update the admin 
password as soon as possible to prevent a possible security breach.
   [junit4]   2> 1447448 INFO  
(TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[4AAFD49A7BC1FFF9]) [] 
o.a.s.SolrTestCaseJ4 ###Ending testBasics
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestSolrCloudWithKerberosAlt -Dtests.method=testBasics 
-Dtests.seed=4AAFD49A7BC1FFF9 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=en-SG -Dtests.timezone=Europe/Vatican -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   35.0s J2 | TestSolrCloudWithKerberosAlt.testBasics <<<
   [junit4]> Throwable #1: java.net.BindException: Address already in use
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([4AAFD49A7BC1FFF9:7AB6432FA189]:0)
   [junit4]>at sun.nio.ch.Net.bind0(Native Method)
   [junit4]>at sun.nio.ch.Net.bind(Net.java:433)
   [junit4]>at sun.nio.ch.Net.bind(Net.java:425)
   [junit4]>at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
   [junit4]>at 
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
   [junit4]>at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:252)
   [junit4]>at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:49)
   [junit4]>at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:525)
   [junit4]>at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$200(AbstractPollingIoAcceptor.java:67)
   [junit4]>at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:409)
   [junit4]>at 

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[jpountz]

Github user jpountz commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242175311
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242172918
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

Re: VOTE: Solr Reference Guide for Solr 7.6 RC1

[Cassandra Targett]

Thanks everyone, the vote passed. I’ll start the release procedures.

Cassandra
On Dec 14, 2018, 2:29 PM -0600, Joel Bernstein , wrote:
> +1
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> > On Fri, Dec 14, 2018 at 2:42 PM Anshum Gupta  wrote:
> > > +1
> > >
> > > Looked at the SolrCloud section and a few other things. Looks good to me!
> > >
> > > > On Mon, Dec 10, 2018 at 7:36 AM Cassandra Targett  
> > > > wrote:
> > > > > Please vote for the release of the Solr Reference Guide for 7.6:
> > > > >
> > > > > The PDF artifacts can be downloaded from:
> > > > > https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-7.6-RC1/
> > > > >
> > > > > $ cat apache-solr-ref-guide-7.6.pdf.sha512
> > > > > 6fb7e3bc61f1dfae9d9d8a7af64e15dd68f8bbe903157a96bf3842ee476dc6238776a3245a93808fcc8d9c015b982f4afc42c9e2c67e74c28f1bca2974778c03
> > > > >   apache-solr-ref-guide-7.6.pdf
> > > > >
> > > > > For this release, the PDF is up to 1415 pages (16.2Mb).
> > > > >
> > > > > The HTML version is also available at 
> > > > > https://lucene.apache.org/solr/guide/7_6.
> > > > >
> > > > > Here's my +1.
> > > > >
> > > > > Thanks,
> > > > > Cassandra

[jira] [Resolved] (LUCENE-8315) Make FeatureField easier to use

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand resolved LUCENE-8315.
--
   Resolution: Fixed
Fix Version/s: 7.4
   master (8.0)

> Make FeatureField easier to use
> ---
>
> Key: LUCENE-8315
> URL: https://issues.apache.org/jira/browse/LUCENE-8315
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Fix For: master (8.0), 7.4
>
> Attachments: LUCENE-8315.patch
>
>
> I'd like to do some changes to FeatureField so that when the saturation 
> function is used and the pivot should be computed automatically from index 
> statistics, then things also work as expected with distributed term 
> frequencies. It works by returning the feature term in extractTerms.
> As a side-effect, no need to provide the IndexSearcher when creating the 
> query anymore, it is autamatically picked up at rewrite time, which makes it 
> nicer to use too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-8207) Add case insensitive support to RegExp

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand resolved LUCENE-8207.
--
Resolution: Won't Fix

> Add case insensitive support to RegExp
> --
>
> Key: LUCENE-8207
> URL: https://issues.apache.org/jira/browse/LUCENE-8207
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/search
>Reporter: Vincent Arnaud
>Priority: Major
> Attachments: LUCENE-8207.patch
>
>
> This ticket proposes to add case insensitive flag _(?i)_ as in Java to RegExp.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-8072) Improve accuracy of similarity scores

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand resolved LUCENE-8072.
--
Resolution: Won't Fix

> Improve accuracy of similarity scores
> -
>
> Key: LUCENE-8072
> URL: https://issues.apache.org/jira/browse/LUCENE-8072
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-8072.patch
>
>
> I noticed two things we could do to improve the accuracy of our scores:
>  - use {{Math.log1p(x)}} instead of {{Math.log(1+x)}}, especially when x is 
> expected to be small
>  - use doubles for intermediate values that are used to compute norms in 
> BM25Similarity



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[jpountz]

Github user jpountz commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242159267
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723009#comment-16723009
 ] 

Michael McCandless commented on LUCENE-8601:


Thanks [~muralikpbhat]; new patch looks great.  I'll push soon.

> Adding attributes to IndexFieldType
> ---
>
> Key: LUCENE-8601
> URL: https://issues.apache.org/jira/browse/LUCENE-8601
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 7.5
>Reporter: Murali Krishna P
>Priority: Major
> Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, 
> LUCENE-8601.03.patch, LUCENE-8601.patch
>
>
> Today, we can write a custom Field using custom IndexFieldType, but when the 
> DefaultIndexingChain converts [IndexFieldType to 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662],
>  only few key informations such as indexing options and doc value type are 
> retained. The [Codec gets the 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90],
>  but not the type details.
>   
>  FieldInfo has support for ['attributes'| 
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47]
>  and it would be great if we can add 'attributes' to IndexFieldType also and 
> copy it to FieldInfo's 'attribute'.
>   
>  This would allow someone to write a custom codec (extending docvalueformat 
> for example) for only the 'special field' that he wants and delegate the rest 
> of the fields to the default codec.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[jpountz]

Github user jpountz commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242158001
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242148673
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 2211 - Still unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/2211/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testSelectedCollections

Error Message:
Failed while waiting for active collection Timeout waiting to see state for 
collection=testSelected2 :null Live Nodes: [127.0.0.1:34668_solr, 
127.0.0.1:37352_solr, 127.0.0.1:56842_solr, 127.0.0.1:57189_solr] Last 
available state: null

Stack Trace:
java.lang.RuntimeException: Failed while waiting for active collection
Timeout waiting to see state for collection=testSelected2 :null
Live Nodes: [127.0.0.1:34668_solr, 127.0.0.1:37352_solr, 127.0.0.1:56842_solr, 
127.0.0.1:57189_solr]
Last available state: null
at 
__randomizedtesting.SeedInfo.seed([F8E22C521D84F556:C24CC98B23E02C38]:0)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:725)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:731)
at 
org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testSelectedCollections(ComputePlanActionTest.java:469)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 

[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType

[Murali Krishna P (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722961#comment-16722961
 ] 

Murali Krishna P commented on LUCENE-8601:
--

Done that change [^LUCENE-8601.03.patch], thank you!

> Adding attributes to IndexFieldType
> ---
>
> Key: LUCENE-8601
> URL: https://issues.apache.org/jira/browse/LUCENE-8601
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 7.5
>Reporter: Murali Krishna P
>Priority: Major
> Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, 
> LUCENE-8601.03.patch, LUCENE-8601.patch
>
>
> Today, we can write a custom Field using custom IndexFieldType, but when the 
> DefaultIndexingChain converts [IndexFieldType to 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662],
>  only few key informations such as indexing options and doc value type are 
> retained. The [Codec gets the 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90],
>  but not the type details.
>   
>  FieldInfo has support for ['attributes'| 
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47]
>  and it would be great if we can add 'attributes' to IndexFieldType also and 
> copy it to FieldInfo's 'attribute'.
>   
>  This would allow someone to write a custom codec (extending docvalueformat 
> for example) for only the 'special field' that he wants and delegate the rest 
> of the fields to the default codec.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8601) Adding attributes to IndexFieldType

[Murali Krishna P (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Murali Krishna P updated LUCENE-8601:
-
Attachment: LUCENE-8601.03.patch

> Adding attributes to IndexFieldType
> ---
>
> Key: LUCENE-8601
> URL: https://issues.apache.org/jira/browse/LUCENE-8601
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 7.5
>Reporter: Murali Krishna P
>Priority: Major
> Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, 
> LUCENE-8601.03.patch, LUCENE-8601.patch
>
>
> Today, we can write a custom Field using custom IndexFieldType, but when the 
> DefaultIndexingChain converts [IndexFieldType to 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662],
>  only few key informations such as indexing options and doc value type are 
> retained. The [Codec gets the 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90],
>  but not the type details.
>   
>  FieldInfo has support for ['attributes'| 
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47]
>  and it would be great if we can add 'attributes' to IndexFieldType also and 
> copy it to FieldInfo's 'attribute'.
>   
>  This would allow someone to write a custom codec (extending docvalueformat 
> for example) for only the 'special field' that he wants and delegate the rest 
> of the fields to the default codec.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-MacOSX (64bit/jdk1.8.0) - Build # 976 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-MacOSX/976/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

No tests ran.

Build Log:
[...truncated 1133 lines...]
ERROR: command execution failed.
ERROR: Step ‘Archive the artifacts’ failed: no workspace for 
Lucene-Solr-7.x-MacOSX #976
ERROR: Step ‘Scan for compiler warnings’ failed: no workspace for 
Lucene-Solr-7.x-MacOSX #976
ERROR: Step ‘Publish JUnit test result report’ failed: no workspace for 
Lucene-Solr-7.x-MacOSX #976
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
ERROR: MacOSX VBOX is offline; cannot locate ANT 1.8.2
Setting ANT_1_8_2_HOME=
ERROR: MacOSX VBOX is offline; cannot locate ANT 1.8.2
Setting ANT_1_8_2_HOME=
ERROR: MacOSX VBOX is offline; cannot locate ANT 1.8.2
Setting ANT_1_8_2_HOME=
ERROR: MacOSX VBOX is offline; cannot locate ANT 1.8.2
Setting ANT_1_8_2_HOME=

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8601) Adding attributes to IndexFieldType

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722893#comment-16722893
 ] 

Michael McCandless commented on LUCENE-8601:


Thanks [~muralikpbhat]; can we make the {{FieldType}} attributes be {{null}} to 
start, and lazily create the {{HashMap}} the first time someone calls 
{{setAttribute}}?  This way people who never use attributes won't pay any added 
cost (except the {{null}} check).

> Adding attributes to IndexFieldType
> ---
>
> Key: LUCENE-8601
> URL: https://issues.apache.org/jira/browse/LUCENE-8601
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 7.5
>Reporter: Murali Krishna P
>Priority: Major
> Attachments: LUCENE-8601.01.patch, LUCENE-8601.02.patch, 
> LUCENE-8601.patch
>
>
> Today, we can write a custom Field using custom IndexFieldType, but when the 
> DefaultIndexingChain converts [IndexFieldType to 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L662],
>  only few key informations such as indexing options and doc value type are 
> retained. The [Codec gets the 
> FieldInfo|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L90],
>  but not the type details.
>   
>  FieldInfo has support for ['attributes'| 
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L47]
>  and it would be great if we can add 'attributes' to IndexFieldType also and 
> copy it to FieldInfo's 'attribute'.
>   
>  This would allow someone to write a custom codec (extending docvalueformat 
> for example) for only the 'special field' that he wants and delegate the rest 
> of the fields to the default codec.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-7.x - Build # 1140 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/1140/

1 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest

Error Message:
expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
at 
__randomizedtesting.SeedInfo.seed([28D7386F842C52F5:85B78C649913FA80]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13564 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: 

[jira] [Commented] (SOLR-12304) Interesting Terms parameter is ignored by MLT Component

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722863#comment-16722863
 ] 

Alessandro Benedetti commented on SOLR-12304:
-

I will just repeat myself, but I don't have anything to add apart the fact I am 
happy to contribute a different patch or help.
So I would love some update on this from the community.

> Interesting Terms parameter is ignored by MLT Component
> ---
>
> Key: SOLR-12304
> URL: https://issues.apache.org/jira/browse/SOLR-12304
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: MoreLikeThis
>Affects Versions: 7.2
>Reporter: Alessandro Benedetti
>Priority: Major
> Attachments: SOLR-12304.patch, SOLR-12304.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently the More Like This component just ignores the mlt.InterestingTerms 
> parameter ( which is usable by the MoreLikeThisHandler).
> Scope of this issue is to fix the bug and add related tests ( which will 
> succeed after the fix )
> *N.B.* MoreLikeThisComponent and MoreLikeThisHandler are very coupled and the 
> tests for the MoreLikeThisHandler are intersecting the MoreLikeThisComponent 
> ones .
>  It is out of scope for this issue any consideration or refactor of that.
>  Other issues will follow.
> *N.B.* out of scope for this issue is the distributed case, which is much 
> more complicated and requires much deeper investigations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12799) Allow Authentication Plugins to easily intercept internode requests

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722865#comment-16722865
 ] 

Cao Manh Dat commented on SOLR-12799:
-

Or we can try to get rid of HttpSolrClient in 2-3 weeks timeline?

> Allow Authentication Plugins to easily intercept internode requests
> ---
>
> Key: SOLR-12799
> URL: https://issues.apache.org/jira/browse/SOLR-12799
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: master (8.0)
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Solr security framework currently allows a plugin to declare statically by 
> implementing the {{HttpClientBuilderPlugin}} interface whether it will handle 
> internode requests. If it implements the interface, the plugin MUST handle 
> ALL internode requests, even requests originating from Solr itself. Likewise, 
> if a plugin does not implement the interface, ALL requests will be 
> authenticated by the built-in {{PKIAuthenticationPlugin}}.
> In some cases (such as SOLR-12121) there is a need to forward end-user 
> credentials on internode requests, but let PKI handle it for solr-originated 
> requests. This is currently not possible without a dirty hack where each 
> plugin duplicates some PKI logic and calls PKI plugin from its own 
> interceptor even if it is disabled.
> This Jira makes this use case officially supported by the framework by:
>  * Letting {{PKIAuthenticationPlugin}} be always enabled. PKI will now in its 
> interceptor on a per-request basis first give the authc plugin a chance to 
> handle the request
>  * Adding a protected method to abstract class {{AuthenticationPlugin}}
>{code:java}
> protected boolean interceptInternodeRequest(HttpRequest httpRequest, 
> HttpContext httpContext)
> {code}
> that can be overridden by plugins in order to easily intercept requests 
> without registering its own interceptor. Returning 'false' delegates to PKI.
> Existing Authc plugins do *not* need to change as a result of this, and they 
> will work exactly as before, i.e. either handle ALL or NONE internode auth.
> New plugins choosing to *override* the new {{interceptInternodeRequest}} 
> method will obtain per-request control over who will secure each request. The 
> first user of this feature will be JWT token based auth in SOLR-12121.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[jpountz]

Github user jpountz commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242092902
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

Re: Lucene/Solr 8.0

[Adrien Grand]

+1

On Mon, Dec 17, 2018 at 10:38 AM Alan Woodward  wrote:
>
> Hi all,
>
> Now that 7.6 is out of the door (thanks Nick!) we should think about cutting 
> the 8.0 branch and moving master to 9.0.  I’ll volunteer to create the branch 
> this week - say Wednesday?  Then we should have some time to clean up the 
> master branch and uncover anything that still needs to be done on 8.0 before 
> we start the release process next year.
>
> On 22 Oct 2018, at 18:12, Cassandra Targett  wrote:
>
> I'm a bit delayed, but +1 on the 7.6 and 8.0 plan from me too.
>
> On Fri, Oct 19, 2018 at 7:18 AM Erick Erickson  
> wrote:
>>
>> +1, this gives us all a chance to prioritize getting the blockers out
>> of the way in a careful manner.
>> On Fri, Oct 19, 2018 at 7:56 AM jim ferenczi  wrote:
>> >
>> > +1 too. With this new perspective we could create the branch just after 
>> > the 7.6 release and target the 8.0 release for January 2019 which gives 
>> > almost 3 month to finish the blockers ?
>> >
>> > Le jeu. 18 oct. 2018 à 23:56, David Smiley  a 
>> > écrit :
>> >>
>> >> +1 to a 7.6 —lots of stuff in there
>> >> On Thu, Oct 18, 2018 at 4:47 PM Nicholas Knize  wrote:
>> >>>
>> >>> If we're planning to postpone cutting an 8.0 branch until a few weeks 
>> >>> from now then I'd like to propose (and volunteer to RM) a 7.6 release 
>> >>> targeted for late November or early December (following the typical 2 
>> >>> month release pattern). It feels like this might give a little breathing 
>> >>> room for finishing up 8.0 blockers? And looking at the change log there 
>> >>> appear to be a healthy list of features, bug fixes, and improvements to 
>> >>> both Solr and Lucene that warrant a 7.6 release? Personally I wouldn't 
>> >>> mind releasing the LatLonShape encoding changes in LUCENE-8521 and 
>> >>> selective indexing work done in LUCENE-8496. Any objections or thoughts?
>> >>>
>> >>> - Nick
>> >>>
>> >>>
>> >>> On Thu, Oct 18, 2018 at 5:32 AM Đạt Cao Mạnh  
>> >>> wrote:
>> 
>>  Thanks Cassandra and Jim,
>> 
>>  I created a blocker issue for Solr 8.0 SOLR-12883, currently in 
>>  jira/http2 branch there are a draft-unmature implementation of SPNEGO 
>>  authentication which enough to makes the test pass, this implementation 
>>  will be removed when SOLR-12883 gets resolved . Therefore I don't see 
>>  any problem on merging jira/http2 to master branch in the next week.
>> 
>>  On Thu, Oct 18, 2018 at 2:33 AM jim ferenczi  
>>  wrote:
>> >
>> > > But if you're working with a different assumption - that just the 
>> > > existence of the branch does not stop Dat from still merging his 
>> > > work and the work being included in 8.0 - then I agree, waiting for 
>> > > him to merge doesn't need to stop the creation of the branch.
>> >
>> > Yes that's my reasoning. This issue is a blocker so we won't release 
>> > without it but we can work on the branch in the meantime and let other 
>> > people work on new features that are not targeted to 8.
>> >
>> > Le mer. 17 oct. 2018 à 20:51, Cassandra Targett 
>> >  a écrit :
>> >>
>> >> OK - I was making an assumption that the timeline for the first 8.0 
>> >> RC would be ASAP after the branch is created.
>> >>
>> >> It's a common perception that making a branch freezes adding new 
>> >> features to the release, perhaps in an unofficial way (more of a 
>> >> courtesy rather than a rule). But if you're working with a different 
>> >> assumption - that just the existence of the branch does not stop Dat 
>> >> from still merging his work and the work being included in 8.0 - then 
>> >> I agree, waiting for him to merge doesn't need to stop the creation 
>> >> of the branch.
>> >>
>> >> If, however, once the branch is there people object to Dat merging 
>> >> his work because it's "too late", then the branch shouldn't be 
>> >> created yet because we want to really try to clear that blocker for 
>> >> 8.0.
>> >>
>> >> Cassandra
>> >>
>> >> On Wed, Oct 17, 2018 at 12:13 PM jim ferenczi 
>> >>  wrote:
>> >>>
>> >>> Ok thanks for answering.
>> >>>
>> >>> > - I think Solr needs a couple more weeks since the work Dat is 
>> >>> > doing isn't quite done yet.
>> >>>
>> >>> We can wait a few more weeks to create the branch but I don't think 
>> >>> that one action (creating the branch) prevents the other (the work 
>> >>> Dat is doing).
>> >>> HTTP/2 is one of the blocker for the release but it can be done in 
>> >>> master and backported to the appropriate branch as any other feature 
>> >>> ? We just need an issue with the blocker label to ensure that
>> >>> we don't miss it ;). Creating the branch early would also help in 
>> >>> case you don't want to release all the work at once in 8.0.0.
>> >>> Next week was just a proposal, what I meant was soon because we 

[jira] [Commented] (SOLR-12799) Allow Authentication Plugins to easily intercept internode requests

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722832#comment-16722832
 ] 

Jan Høydahl commented on SOLR-12799:


I see the new methods added for the new client.

But looks like e.g. {{BasicAuthIntegrationTest}} explicitly creates an old 
client in {{HttpClientUtil.createClient()}}? How can we make sure that this 
test actually runs with HTTP2 and tests the new code paths?

 

So I guess that plugins wanting to override this needs to override both those 
methods (which is a pity).

I feel the overrides for {{BasicAuthPlugin}} has too much redundant code, we 
could factor out the common part here in a new method and keep the overridden 
methods 2-3 lines only:
{code:java}
@Override
protected boolean interceptInternodeRequest(HttpRequest httpRequest, 
HttpContext httpContext) {
  if (forwardCredentials) {
if (httpContext instanceof HttpClientContext) {
  HttpClientContext httpClientContext = (HttpClientContext) httpContext;
  if (httpClientContext.getUserToken() instanceof BasicAuthUserPrincipal) {
BasicAuthUserPrincipal principal = (BasicAuthUserPrincipal) 
httpClientContext.getUserToken();
String userPassBase64 = Base64.encodeBase64String((principal.getName() 
+ ":" + principal.getPassword()).getBytes(StandardCharsets.UTF_8));
httpRequest.setHeader(HttpHeaders.AUTHORIZATION, "Basic " + 
userPassBase64);
return true;
  }
}
  }
  return false;
}

@Override
protected boolean interceptInternodeRequest(Request request) {
  if (forwardCredentials) {
Object userToken = 
request.getAttributes().get(Http2SolrClient.REQ_PRINCIPAL_KEY);
if (userToken instanceof BasicAuthUserPrincipal) {
  BasicAuthUserPrincipal principal = (BasicAuthUserPrincipal) userToken;
  String userPassBase64 = Base64.encodeBase64String((principal.getName() + 
":" + principal.getPassword()).getBytes(StandardCharsets.UTF_8));
  request.header(HttpHeaders.AUTHORIZATION, "Basic " + userPassBase64);
  return true;
}
  }
  return false;
}
{code}

> Allow Authentication Plugins to easily intercept internode requests
> ---
>
> Key: SOLR-12799
> URL: https://issues.apache.org/jira/browse/SOLR-12799
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: master (8.0)
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Solr security framework currently allows a plugin to declare statically by 
> implementing the {{HttpClientBuilderPlugin}} interface whether it will handle 
> internode requests. If it implements the interface, the plugin MUST handle 
> ALL internode requests, even requests originating from Solr itself. Likewise, 
> if a plugin does not implement the interface, ALL requests will be 
> authenticated by the built-in {{PKIAuthenticationPlugin}}.
> In some cases (such as SOLR-12121) there is a need to forward end-user 
> credentials on internode requests, but let PKI handle it for solr-originated 
> requests. This is currently not possible without a dirty hack where each 
> plugin duplicates some PKI logic and calls PKI plugin from its own 
> interceptor even if it is disabled.
> This Jira makes this use case officially supported by the framework by:
>  * Letting {{PKIAuthenticationPlugin}} be always enabled. PKI will now in its 
> interceptor on a per-request basis first give the authc plugin a chance to 
> handle the request
>  * Adding a protected method to abstract class {{AuthenticationPlugin}}
>{code:java}
> protected boolean interceptInternodeRequest(HttpRequest httpRequest, 
> HttpContext httpContext)
> {code}
> that can be overridden by plugins in order to easily intercept requests 
> without registering its own interceptor. Returning 'false' delegates to PKI.
> Existing Authc plugins do *not* need to change as a result of this, and they 
> will work exactly as before, i.e. either handle ALL or NONE internode auth.
> New plugins choosing to *override* the new {{interceptInternodeRequest}} 
> method will obtain per-request control over who will secure each request. The 
> first user of this feature will be JWT token based auth in SOLR-12121.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Windows (64bit/jdk1.8.0_172) - Build # 914 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/914/
Java: 64bit/jdk1.8.0_172 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.core.ExitableDirectoryReaderTest.testCacheAssumptions

Error Message:
Should have fewer docs than 100

Stack Trace:
java.lang.AssertionError: Should have fewer docs than 100
at 
__randomizedtesting.SeedInfo.seed([F8DEF6C69730BE88:8FA349E1F9C5F3AA]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.solr.core.ExitableDirectoryReaderTest.testCacheAssumptions(ExitableDirectoryReaderTest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13695 lines...]
   [junit4] Suite: org.apache.solr.core.ExitableDirectoryReaderTest
   [junit4]   2> 1167099 INFO  
(SUITE-ExitableDirectoryReaderTest-seed#[F8DEF6C69730BE88]-worker) [] 
o.a.s.SolrTestCaseJ4 SecureRandom 

Re: Lucene/Solr 8.0

[Alan Woodward]

Hi all,

Now that 7.6 is out of the door (thanks Nick!) we should think about cutting 
the 8.0 branch and moving master to 9.0.  I’ll volunteer to create the branch 
this week - say Wednesday?  Then we should have some time to clean up the 
master branch and uncover anything that still needs to be done on 8.0 before we 
start the release process next year.

> On 22 Oct 2018, at 18:12, Cassandra Targett  > wrote:
> 
> I'm a bit delayed, but +1 on the 7.6 and 8.0 plan from me too.
> 
> On Fri, Oct 19, 2018 at 7:18 AM Erick Erickson  > wrote:
> +1, this gives us all a chance to prioritize getting the blockers out
> of the way in a careful manner.
> On Fri, Oct 19, 2018 at 7:56 AM jim ferenczi  > wrote:
> >
> > +1 too. With this new perspective we could create the branch just after the 
> > 7.6 release and target the 8.0 release for January 2019 which gives almost 
> > 3 month to finish the blockers ?
> >
> > Le jeu. 18 oct. 2018 à 23:56, David Smiley  > > a écrit :
> >>
> >> +1 to a 7.6 —lots of stuff in there
> >> On Thu, Oct 18, 2018 at 4:47 PM Nicholas Knize  >> > wrote:
> >>>
> >>> If we're planning to postpone cutting an 8.0 branch until a few weeks 
> >>> from now then I'd like to propose (and volunteer to RM) a 7.6 release 
> >>> targeted for late November or early December (following the typical 2 
> >>> month release pattern). It feels like this might give a little breathing 
> >>> room for finishing up 8.0 blockers? And looking at the change log there 
> >>> appear to be a healthy list of features, bug fixes, and improvements to 
> >>> both Solr and Lucene that warrant a 7.6 release? Personally I wouldn't 
> >>> mind releasing the LatLonShape encoding changes in LUCENE-8521 and 
> >>> selective indexing work done in LUCENE-8496. Any objections or thoughts?
> >>>
> >>> - Nick
> >>>
> >>>
> >>> On Thu, Oct 18, 2018 at 5:32 AM Đạt Cao Mạnh  >>> > wrote:
> 
>  Thanks Cassandra and Jim,
> 
>  I created a blocker issue for Solr 8.0 SOLR-12883, currently in 
>  jira/http2 branch there are a draft-unmature implementation of SPNEGO 
>  authentication which enough to makes the test pass, this implementation 
>  will be removed when SOLR-12883 gets resolved . Therefore I don't see 
>  any problem on merging jira/http2 to master branch in the next week.
> 
>  On Thu, Oct 18, 2018 at 2:33 AM jim ferenczi   > wrote:
> >
> > > But if you're working with a different assumption - that just the 
> > > existence of the branch does not stop Dat from still merging his work 
> > > and the work being included in 8.0 - then I agree, waiting for him to 
> > > merge doesn't need to stop the creation of the branch.
> >
> > Yes that's my reasoning. This issue is a blocker so we won't release 
> > without it but we can work on the branch in the meantime and let other 
> > people work on new features that are not targeted to 8.
> >
> > Le mer. 17 oct. 2018 à 20:51, Cassandra Targett  > > a écrit :
> >>
> >> OK - I was making an assumption that the timeline for the first 8.0 RC 
> >> would be ASAP after the branch is created.
> >>
> >> It's a common perception that making a branch freezes adding new 
> >> features to the release, perhaps in an unofficial way (more of a 
> >> courtesy rather than a rule). But if you're working with a different 
> >> assumption - that just the existence of the branch does not stop Dat 
> >> from still merging his work and the work being included in 8.0 - then 
> >> I agree, waiting for him to merge doesn't need to stop the creation of 
> >> the branch.
> >>
> >> If, however, once the branch is there people object to Dat merging his 
> >> work because it's "too late", then the branch shouldn't be created yet 
> >> because we want to really try to clear that blocker for 8.0.
> >>
> >> Cassandra
> >>
> >> On Wed, Oct 17, 2018 at 12:13 PM jim ferenczi  >> > wrote:
> >>>
> >>> Ok thanks for answering.
> >>>
> >>> > - I think Solr needs a couple more weeks since the work Dat is 
> >>> > doing isn't quite done yet.
> >>>
> >>> We can wait a few more weeks to create the branch but I don't think 
> >>> that one action (creating the branch) prevents the other (the work 
> >>> Dat is doing).
> >>> HTTP/2 is one of the blocker for the release but it can be done in 
> >>> master and backported to the appropriate branch as any other feature 
> >>> ? We just need an issue with the blocker label to ensure that
> >>> we don't miss it ;). Creating the branch early would also help in 
> >>> case you don't want to release 

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242073523
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java
 ---
@@ -0,0 +1,1413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.lucene.codecs.CodecUtil;
+import org.apache.lucene.codecs.DocValuesProducer;
+import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.index.FieldInfos;
+import org.apache.lucene.index.ImpactsEnum;
+import org.apache.lucene.index.IndexFileNames;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PostingsEnum;
+import org.apache.lucene.index.SegmentReadState;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.index.SortedSetDocValues;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.index.TermsEnum.SeekStatus;
+import org.apache.lucene.store.ChecksumIndexInput;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.RandomAccessInput;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.IOUtils;
+import org.apache.lucene.util.LongValues;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.lucene.util.packed.DirectMonotonicReader;
+import org.apache.lucene.util.packed.DirectReader;
+
+/** reader for {@link Lucene80DocValuesFormat} */
+final class Lucene80DocValuesProducer extends DocValuesProducer implements 
Closeable {
+  private final Map numerics = new HashMap<>();
+  private final Map binaries = new HashMap<>();
+  private final Map sorted = new HashMap<>();
+  private final Map sortedSets = new HashMap<>();
+  private final Map sortedNumerics = new 
HashMap<>();
+  private long ramBytesUsed;
+  private final IndexInput data;
+  private final int maxDoc;
+
+  /** expert: instantiates a new reader */
+  Lucene80DocValuesProducer(SegmentReadState state, String dataCodec, 
String dataExtension, String metaCodec, String metaExtension) throws 
IOException {
+String metaName = 
IndexFileNames.segmentFileName(state.segmentInfo.name, state.segmentSuffix, 
metaExtension);
+this.maxDoc = state.segmentInfo.maxDoc();
+ramBytesUsed = RamUsageEstimator.shallowSizeOfInstance(getClass());
+
+int version = -1;
+
+// read in the entries from the metadata file.
+try (ChecksumIndexInput in = 
state.directory.openChecksumInput(metaName, state.context)) {
+  Throwable priorE = null;
+  try {
+version = CodecUtil.checkIndexHeader(in, metaCodec,
+
Lucene80DocValuesFormat.VERSION_START,
+
Lucene80DocValuesFormat.VERSION_CURRENT,
+state.segmentInfo.getId(),
+state.segmentSuffix);
+readFields(in, state.fieldInfos);
+  } catch (Throwable exception) {
+priorE = exception;
+  } finally {
+CodecUtil.checkFooter(in, priorE);
+  }
+}
+
+String dataName = 
IndexFileNames.segmentFileName(state.segmentInfo.name, state.segmentSuffix, 
dataExtension);
+this.data = state.directory.openInput(dataName, state.context);
+boolean success = false;
+try {
+  final int version2 = CodecUtil.checkIndexHeader(data, dataCodec,
+ 
Lucene80DocValuesFormat.VERSION_START,
+

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242073366
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java
 ---
@@ -0,0 +1,1413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.lucene.codecs.CodecUtil;
+import org.apache.lucene.codecs.DocValuesProducer;
+import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.index.FieldInfos;
+import org.apache.lucene.index.ImpactsEnum;
+import org.apache.lucene.index.IndexFileNames;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PostingsEnum;
+import org.apache.lucene.index.SegmentReadState;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.index.SortedSetDocValues;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.index.TermsEnum.SeekStatus;
+import org.apache.lucene.store.ChecksumIndexInput;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.RandomAccessInput;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.IOUtils;
+import org.apache.lucene.util.LongValues;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.lucene.util.packed.DirectMonotonicReader;
+import org.apache.lucene.util.packed.DirectReader;
+
+/** reader for {@link Lucene80DocValuesFormat} */
+final class Lucene80DocValuesProducer extends DocValuesProducer implements 
Closeable {
+  private final Map numerics = new HashMap<>();
+  private final Map binaries = new HashMap<>();
+  private final Map sorted = new HashMap<>();
+  private final Map sortedSets = new HashMap<>();
+  private final Map sortedNumerics = new 
HashMap<>();
+  private long ramBytesUsed;
+  private final IndexInput data;
+  private final int maxDoc;
+
+  /** expert: instantiates a new reader */
+  Lucene80DocValuesProducer(SegmentReadState state, String dataCodec, 
String dataExtension, String metaCodec, String metaExtension) throws 
IOException {
+String metaName = 
IndexFileNames.segmentFileName(state.segmentInfo.name, state.segmentSuffix, 
metaExtension);
+this.maxDoc = state.segmentInfo.maxDoc();
+ramBytesUsed = RamUsageEstimator.shallowSizeOfInstance(getClass());
+
+int version = -1;
+
+// read in the entries from the metadata file.
+try (ChecksumIndexInput in = 
state.directory.openChecksumInput(metaName, state.context)) {
+  Throwable priorE = null;
+  try {
+version = CodecUtil.checkIndexHeader(in, metaCodec,
+
Lucene80DocValuesFormat.VERSION_START,
+
Lucene80DocValuesFormat.VERSION_CURRENT,
+state.segmentInfo.getId(),
+state.segmentSuffix);
+readFields(in, state.fieldInfos);
+  } catch (Throwable exception) {
+priorE = exception;
+  } finally {
+CodecUtil.checkFooter(in, priorE);
+  }
+}
+
+String dataName = 
IndexFileNames.segmentFileName(state.segmentInfo.name, state.segmentSuffix, 
dataExtension);
+this.data = state.directory.openInput(dataName, state.context);
+boolean success = false;
+try {
+  final int version2 = CodecUtil.checkIndexHeader(data, dataCodec,
+ 
Lucene80DocValuesFormat.VERSION_START,
+

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242072816
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[jira] [Commented] (SOLR-12799) Allow Authentication Plugins to easily intercept internode requests

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722818#comment-16722818
 ] 

Cao Manh Dat commented on SOLR-12799:
-

Hi [~janhoy], I did merge http2 into master. But it will be nice if you can 
review my changes on BasicAuthPlugin#interceptInternodeRequest.

> Allow Authentication Plugins to easily intercept internode requests
> ---
>
> Key: SOLR-12799
> URL: https://issues.apache.org/jira/browse/SOLR-12799
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: master (8.0)
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Solr security framework currently allows a plugin to declare statically by 
> implementing the {{HttpClientBuilderPlugin}} interface whether it will handle 
> internode requests. If it implements the interface, the plugin MUST handle 
> ALL internode requests, even requests originating from Solr itself. Likewise, 
> if a plugin does not implement the interface, ALL requests will be 
> authenticated by the built-in {{PKIAuthenticationPlugin}}.
> In some cases (such as SOLR-12121) there is a need to forward end-user 
> credentials on internode requests, but let PKI handle it for solr-originated 
> requests. This is currently not possible without a dirty hack where each 
> plugin duplicates some PKI logic and calls PKI plugin from its own 
> interceptor even if it is disabled.
> This Jira makes this use case officially supported by the framework by:
>  * Letting {{PKIAuthenticationPlugin}} be always enabled. PKI will now in its 
> interceptor on a per-request basis first give the authc plugin a chance to 
> handle the request
>  * Adding a protected method to abstract class {{AuthenticationPlugin}}
>{code:java}
> protected boolean interceptInternodeRequest(HttpRequest httpRequest, 
> HttpContext httpContext)
> {code}
> that can be overridden by plugins in order to easily intercept requests 
> without registering its own interceptor. Returning 'false' delegates to PKI.
> Existing Authc plugins do *not* need to change as a result of this, and they 
> will work exactly as before, i.e. either handle ALL or NONE internode auth.
> New plugins choosing to *override* the new {{interceptInternodeRequest}} 
> method will obtain per-request control over who will secure each request. The 
> first user of this feature will be JWT token based auth in SOLR-12121.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242070202
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242069261
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242068739
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

Re: [JENKINS] Lucene-Solr-master-Linux (64bit/jdk-11) - Build # 23344 - Still Failing!

[Adrien Grand]

I pushed a fix.

On Mon, Dec 17, 2018 at 9:30 AM Policeman Jenkins Server
 wrote:
>
> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23344/
> Java: 64bit/jdk-11 -XX:+UseCompressedOops -XX:+UseSerialGC
>
> All tests passed
>
> Build Log:
> [...truncated 16118 lines...]
> [javac] Compiling 191 source files to 
> /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-solrj/classes/test
> [javac] 
> /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClientBadInputTest.java:31:
>  error: cannot find symbol
> [javac] import static 
> org.junit.internal.matchers.StringContains.containsString;
> [javac]  ^
> [javac]   symbol:   class StringContains
> [javac]   location: package org.junit.internal.matchers
> [javac] 
> /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClientBadInputTest.java:31:
>  error: static import only from classes and interfaces
> [javac] import static 
> org.junit.internal.matchers.StringContains.containsString;
> [javac] ^
> [javac] 
> /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClientBadInputTest.java:96:
>  error: cannot find symbol
> [javac] assertThat(thrown.getMessage(), 
> containsString(expectedString));
> [javac] ^
> [javac]   symbol:   method containsString(String)
> [javac]   location: class ConcurrentUpdateHttp2SolrClientBadInputTest
> [javac] Note: Some input files use or override a deprecated API.
> [javac] Note: Recompile with -Xlint:deprecation for details.
> [javac] Note: Some input files use unchecked or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked for details.
> [javac] 3 errors
>
> BUILD FAILED
> /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:633: The following 
> error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:577: The following 
> error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:59: The following 
> error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:280: The 
> following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/common-build.xml:558: 
> The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:903: 
> The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:915: 
> The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:2056:
>  Compile failed; see the compiler error output for details.
>
> Total time: 58 minutes 34 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Setting 
> ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
> [WARNINGS] Skipping publisher since build result is FAILURE
> Recording test results
> Setting 
> ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
> Setting 
> ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
> Setting 
> ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
> Setting 
> ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
> Setting 
> ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org



-- 
Adrien

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: MODERATE for annou...@apache.org

[Adrien Grand]

Hi Craig,

Thanks for catching that, it should be fixed now.

On Fri, Dec 14, 2018 at 7:17 PM Private LIst Moderation
 wrote:
>
> Your download page http://lucene.apache.org/solr/downloads.html has a link to 
> sha1 which is no longer there. Please update the page to refer to the sha512 
> checksums.
>
> I didn't notice it earlier but your core download page 
> http://lucene.apache.org/core/downloads.html has the same problem.
>
> If you can update the page timely, we can approve this announcement.
>
> Regards,
>
> Craig
>
>
> Begin forwarded message:
>
> From: announce-reject-1544811025.45878.geanoooanbnhjlbgn...@apache.org
> Subject: MODERATE for annou...@apache.org
> Date: December 14, 2018 at 10:10:25 AM PST
> To: Recipient list not shown: ;
> Cc: 
> announce-allow-tc.1544811025.pnplidaoalgkefgfblkl-nknize=apache@apache.org
> Reply-To: announce-accept-1544811025.45878.geanoooanbnhjlbgn...@apache.org
>
>
> To approve:
>   announce-accept-1544811025.45878.geanoooanbnhjlbgn...@apache.org
> To reject:
>   announce-reject-1544811025.45878.geanoooanbnhjlbgn...@apache.org
> To give a reason to reject:
> %%% Start comment
> %%% End comment
>
>
> From: Nicholas Knize 
> Subject: [ANNOUNCE] Apache Solr 7.6.0 released
> Date: December 14, 2018 at 10:10:08 AM PST
> To: "annou...@apache.org" , "gene...@lucene.apache.org" 
> , "Lucene/Solr dev" , 
> "solr-u...@lucene.apache.org" 
>
>
> 14 December 2018, Apache Solr™ 7.6.0 available
>
> The Lucene PMC is pleased to announce the release of Apache Solr 7.6.0
>
> Solr is the popular, blazing fast, open source NoSQL search platform from the 
> Apache Lucene project. Its major features include powerful full-text search, 
> hit highlighting, faceted search, dynamic clustering, database integration, 
> rich document (e.g., Word, PDF) handling, and geospatial search. Solr is 
> highly scalable, providing fault tolerant distributed search and indexing, 
> and powers the search and navigation features of many of the world's largest 
> internet sites.
>
> Solr 7.6.0 is available for immediate download at: 
> http://lucene.apache.org/solr/downloads.html
>
> See http://lucene.apache.org/solr/7_6_0/changes/Changes.html for a full list 
> of details.
>
> Solr 7.6.0 Release Highlights:
>
>* Field and FieldType now support a new uninvertible option to control 
> using costly field cache or more efficient docValues.
>* Collections API has been improved to support adding multiple replicas to 
> a collection shard at a time as well as splitting into multiple sub-shards 
> directly..
>* Autoscaling's suggestions API now include rebalance options as well as 
> suggestions to add new replicas for lost replicas.
>* Several new Stream Evaluators have been added to include: oscillate, 
> convexHull, enclosingDisk, pairSort, log10, percentiles, and pivot for 
> geometric and scientific analysis.
>* UnifiedHighlighter has been improved to support best/perfect 
> highlighting accuracy and full phrase highlighting.
>
> You are encouraged to thoroughly read the "Upgrade Notes" at 
> http://lucene.apache.org/solr/7_6_0/changes/Changes.html or in the 
> CHANGES.txt file accompanying the release.
>
> Solr 7.6 also includes many other new features as well as numerous 
> optimizations and bugfixes of the corresponding Apache Lucene release.
>
> Please report any feedback to the mailing lists 
> (http://lucene.apache.org/solr/community.html#mailing-lists-irc)
>
> Note: The Apache Software Foundation uses an extensive mirroring network for 
> distributing releases. It is possible that the mirror you are using may not 
> have replicated the release yet. If that is the case, please try another 
> mirror. This also goes for Maven access.
> --
>
> Nicholas Knize, Ph.D., GISP
> Geospatial Software Guy  |  Elasticsearch
> Apache Lucene Committer
> nkn...@apache.org
>
>
>
>
> Craig L Russell
> Secretary, Apache Software Foundation
> c...@apache.org http://db.apache.org/jdo
>


-- 
Adrien

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242062202
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242060979
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242060186
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene80/IndexedDISI.java ---
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+import java.io.DataInput;
+import java.io.IOException;
+
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BitSetIterator;
+import org.apache.lucene.util.FixedBitSet;
+import org.apache.lucene.util.RoaringDocIdSet;
+
+/**
+ * Disk-based implementation of a {@link DocIdSetIterator} which can return
+ * the index of the current document, i.e. the ordinal of the current 
document
+ * among the list of documents that this iterator can return. This is 
useful
+ * to implement sparse doc values by only having to encode values for 
documents
+ * that actually have a value.
+ * Implementation-wise, this {@link DocIdSetIterator} is inspired of
+ * {@link RoaringDocIdSet roaring bitmaps} and encodes ranges of {@code 
65536}
+ * documents independently and picks between 3 encodings depending on the
+ * density of the range:
+ *   {@code ALL} if the range contains 65536 documents exactly,
+ *   {@code DENSE} if the range contains 4096 documents or more; in 
that
+ *   case documents are stored in a bit set,
+ *   {@code SPARSE} otherwise, and the lower 16 bits of the doc IDs are
+ *   stored in a {@link DataInput#readShort() short}.
+ * 
+ * Only ranges that contain at least one value are encoded.
+ * This implementation uses 6 bytes per document in the worst-case, 
which happens
+ * in the case that all ranges contain exactly one document.
+ *
+ * 
+ * To avoid O(n) lookup time complexity, with n being the number of 
documents, two lookup
+ * tables are used:  * A lookup table for block blockCache and index, and 
a rank structure
+ * for DENSE block lookups.
+ *
+ * The lookup table is an array of {@code long}s with an entry for each 
block. It allows for
+ * direct jumping to the block, as opposed to iteration from the current 
position and forward
+ * one block at a time.
+ *
+ * Each long entry consists of 2 logical parts:
+ *
+ * The first 31 bits holds the index (number of set bits in the blocks) up 
to just before the
+ * wanted block. The next 33 bits holds the offset into the underlying 
slice.
+ * As there is a maximum of 2^16 blocks, it follows that the maximum size 
of any block must
+ * not exceed 2^17 bits to avoid overflow. This is currently the case, 
with the largest
+ * block being DENSE and using 2^16 + 32 bits, and is likely to continue 
to hold as using
+ * more than double the amount of bits is unlikely to be an efficient 
representation.
+ * The cache overhead is numDocs/1024 bytes.
+ *
+ * Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing 
(0 set bits).
+ * In the case of non-existing blocks, the entry in the lookup table has 
index equal to the
+ * previous entry and offset equal to the next non-empty block.
+ *
+ * The block lookup table is stored at the end of the total block 
structure.
+ *
+ *
+ * The rank structure for DENSE blocks is an array of unsigned {@code 
short}s with an entry
+ * for each sub-block of 512 bits out of the 65536 bits in the outer DENSE 
block.
+ *
+ * Each rank-entry states the number of set bits within the block up to 
the bit before the
+ * bit positioned at the start of the sub-block.
+ * Note that that the rank entry of the first sub-block is always 0 and 
that the last entry can
+ * at most be 65536-512 = 65024 and thus will always fit into an unsigned 
short.
+ *
+ * The rank structure for a given DENSE block is stored at the beginning 
of the 

[jira] [Commented] (SOLR-12799) Allow Authentication Plugins to easily intercept internode requests

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722784#comment-16722784
 ] 

Jan Høydahl commented on SOLR-12799:


Did some quick tests and beasting {{BasicAuthIntegrationTest}} that proves this 
feature. Looks promising, but I have not validated any http2 specific things.

Looks like you merged http2 into master already, right? Is there anything in 
particular you'd like another pair of eyes on regarding the merge of this 
feature?

> Allow Authentication Plugins to easily intercept internode requests
> ---
>
> Key: SOLR-12799
> URL: https://issues.apache.org/jira/browse/SOLR-12799
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: master (8.0)
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Solr security framework currently allows a plugin to declare statically by 
> implementing the {{HttpClientBuilderPlugin}} interface whether it will handle 
> internode requests. If it implements the interface, the plugin MUST handle 
> ALL internode requests, even requests originating from Solr itself. Likewise, 
> if a plugin does not implement the interface, ALL requests will be 
> authenticated by the built-in {{PKIAuthenticationPlugin}}.
> In some cases (such as SOLR-12121) there is a need to forward end-user 
> credentials on internode requests, but let PKI handle it for solr-originated 
> requests. This is currently not possible without a dirty hack where each 
> plugin duplicates some PKI logic and calls PKI plugin from its own 
> interceptor even if it is disabled.
> This Jira makes this use case officially supported by the framework by:
>  * Letting {{PKIAuthenticationPlugin}} be always enabled. PKI will now in its 
> interceptor on a per-request basis first give the authc plugin a chance to 
> handle the request
>  * Adding a protected method to abstract class {{AuthenticationPlugin}}
>{code:java}
> protected boolean interceptInternodeRequest(HttpRequest httpRequest, 
> HttpContext httpContext)
> {code}
> that can be overridden by plugins in order to easily intercept requests 
> without registering its own interceptor. Returning 'false' delegates to PKI.
> Existing Authc plugins do *not* need to change as a result of this, and they 
> will work exactly as before, i.e. either handle ALL or NONE internode auth.
> New plugins choosing to *override* the new {{interceptInternodeRequest}} 
> method will obtain per-request control over who will secure each request. The 
> first user of this feature will be JWT token based auth in SOLR-12121.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk-11) - Build # 23344 - Still Failing!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23344/
Java: 64bit/jdk-11 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 16118 lines...]
[javac] Compiling 191 source files to 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-solrj/classes/test
[javac] 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClientBadInputTest.java:31:
 error: cannot find symbol
[javac] import static 
org.junit.internal.matchers.StringContains.containsString;
[javac]  ^
[javac]   symbol:   class StringContains
[javac]   location: package org.junit.internal.matchers
[javac] 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClientBadInputTest.java:31:
 error: static import only from classes and interfaces
[javac] import static 
org.junit.internal.matchers.StringContains.containsString;
[javac] ^
[javac] 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/solrj/src/test/org/apache/solr/client/solrj/impl/ConcurrentUpdateHttp2SolrClientBadInputTest.java:96:
 error: cannot find symbol
[javac] assertThat(thrown.getMessage(), 
containsString(expectedString));
[javac] ^
[javac]   symbol:   method containsString(String)
[javac]   location: class ConcurrentUpdateHttp2SolrClientBadInputTest
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 3 errors

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:633: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:577: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:59: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:280: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/common-build.xml:558: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:903: 
The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:915: 
The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:2056: 
Compile failed; see the compiler error output for details.

Total time: 58 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #525: LUCENE-8585: Index-time jump-tables for DocVa...

[tokee]

Github user tokee commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/525#discussion_r242055494
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/lucene70/package-info.java ---
@@ -284,12 +284,12 @@
  * Stores additional per-position metadata information such as 
character offsets and user payloads
  * 
  * 
- * {@link org.apache.lucene.codecs.lucene70.Lucene70NormsFormat 
Norms}
+ * org.apache.lucene.codecs.lucene70.Lucene70NormsFormat Norms
  * .nvd, .nvm
  * Encodes length and boost factors for docs and fields
  * 
  * 
- * {@link org.apache.lucene.codecs.lucene70.Lucene70DocValuesFormat 
Per-Document Values}
+ * org.apache.lucene.codecs.lucene70.Lucene70DocValuesFormat 
Per-Document Values
  * .dvd, .dvm
  * Encodes additional scoring factors or other per-document 
information.
  * 
--- End diff --

Seems reasonable. I have mimicked the phrasing from the lucene60 
`package-info.java` in the upcoming commit and also adjusted the references 
from lucene50, lucene60 to point to lucene80.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org