Re: Plan for Velocity

[Ishan Chattopadhyaya]

https://issues.apache.org/jira/browse/SOLR-14065
Description contains the Erik's repository.

On Mon, Aug 31, 2020 at 2:38 AM Varun Thacker  wrote:

> SOLR-14065 deprecated Velocity in 8.4 . Is the plan to remove it for 9.0 ?
>
> If this has already been discussed please feel free to point me to it.
>

Re: Plan for Velocity

[Ishan Chattopadhyaya]

Yes, plan to remove it in 9.0.

On Mon, Aug 31, 2020 at 4:13 AM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> https://issues.apache.org/jira/browse/SOLR-14065
> Description contains the Erik's repository.
>
> On Mon, Aug 31, 2020 at 2:38 AM Varun Thacker  wrote:
>
>> SOLR-14065 deprecated Velocity in 8.4 . Is the plan to remove it for 9.0
>> ?
>>
>> If this has already been discussed please feel free to point me to it.
>>
>

Re: Remove section for 9.0 solr/CHANGES

[Varun Thacker]

Maybe even distinguish between features that do have a replacement ( via a
plugin ) and those that don't

On Sun, Aug 30, 2020 at 1:49 PM Varun Thacker  wrote:

> Currently when looking at the "Other Changes" section in the CHANGES entry
> for 9.0 most of the items are removal of features
>
> For 9.0 should we have a special "Feature Removal" section within the
> CHANGES file ?
>
> We could add Autoscaling / CDCR / DIH / ANT etc to that secttion
>

Plan for Velocity

[Varun Thacker]

SOLR-14065 deprecated Velocity in 8.4 . Is the plan to remove it for 9.0 ?

If this has already been discussed please feel free to point me to it.

Remove section for 9.0 solr/CHANGES

[Varun Thacker]

Currently when looking at the "Other Changes" section in the CHANGES entry
for 9.0 most of the items are removal of features

For 9.0 should we have a special "Feature Removal" section within the
CHANGES file ?

We could add Autoscaling / CDCR / DIH / ANT etc to that secttion

RE: Performance in Solr 9 / Java 11

[Uwe Schindler]

Hi,

 

Yes, mr-jar build were always done in Lucene 8x (on Lucene’s part of the 
build). You might think: “How can that be as I am – as release manager – used 
JDK8 to build, and I am forced to do so?”

 

This is a trick that works for some types of JDK9+ improvements, which rely on 
some simple “replacement” operations for static method that are highly optiized 
in later JDK versions using an compiler intrinsic. The trick is the following: 
We have (possibly slower) replacement implementations in Lucene’s codebase for 
the new method (see classes oal.utils.FutureArrays and FutureObjects) with 
identical static signature (this is why it only works for that case). After 
compilation of the code an additional Ant task is running that uses ASM to 
patch the produced JAR files of the Java 8 compiler to replace the usage of the 
FutureXxxx classes in the code:  

 
https://github.com/apache/lucene-solr/blob/branch_8x/lucene/tools/src/groovy/patch-mrjar-classes.groovy
 The patched classes are written to build/java9 and jarred into jar file below 
META-INF/versions/9

 

In master this was removed. If we have a similar thing in later JDK versions 
(static methods which got highly optimized versions with intrinsics in later 
Java versions), we can do the same, the above script just needs to be included 
into the Gradle build. But I have not figured out anything like this.

 

The classical “MR-builds” are much more complex to setup, as it requires every 
developer to have the later version already installed and we need to do 
parallel compilation of different source trees which also has the risk of 
producing broken code that won’t work at runtime. So the “just replace method 
calls in Java 8 classes” is very convenient & safe, but limited. Very often 
it’s better for more complex cases, to add a factory pattern to your source 
code that uses a completely different implementation at runtime (like 
FSDirectory.open()). One example that might come later is a replacement for 
MMapDirectory. I am working on this, but it’s not yet useable (see my short 
talk at barcamp @ berlinbuzzwords this year). Here, we would have a separate 
impl of MMapDirectory like MemorySegmentDirectory that’s choosen at runtime 
depending on Java version. We would have a separate gradle module to implement 
that, which requires a later compiler. The reason for that approach is: The 
code is so different, that a classical MR JAR would complicate a lot. It’s 
better to declare this as a separate “implementation class”, so users get 
better stack traces on error as they actually see what implementation is used.

 

In short: The release manager in 8.x does not need to care, it’s fully 
automatic and requires no later version of Java during build.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Adrien Grand  
Sent: Sunday, August 30, 2020 9:00 AM
To: Lucene Dev 
Subject: Re: Performance in Solr 9 / Java 11

 

Tomoko is correct, an MR JAR is created not only upon release but also every 
time you create a lucene-core JAR on branch_8x.

 

On Sun, Aug 30, 2020 at 5:49 AM Tomoko Uchida mailto:tomoko.uchida.1...@gmail.com> > wrote:

I believe mr-jar build is enabled in the 8x branch (LUCENE-7966), and the 
workaround was dropped on the master branch when the minimum java version was 
bumped up to java 11 (LUCENE-8738); if my understanding is correct.


 

$ jar tf core/lucene-core-8.6.1.jar | grep META-INF/versions
META-INF/versions/
META-INF/versions/9/
META-INF/versions/9/org/
META-INF/versions/9/org/apache/

...

 

$ jar tf core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar | grep META-INF/versions

// no outputs

 

 

 

 

2020年8月30日(日) 6:48 Mike Drob mailto:md...@apache.org> >:

Do you know if these mr-jars are built by default as part of the release 
process? I definitely had no idea about them when doing 8.5.2 and did not even 
think to verify anything about it. 

 

On Sat, Aug 29, 2020 at 4:05 PM Adrien Grand mailto:jpou...@gmail.com> > wrote:

It may only be indirectly related to your question, but there is support for 
vectorized operations of byte[] arrays that was added in JDK 13 (this blog 
https://richardstartin.github.io/posts/vectorised-byte-operations explains well 
what it is about) that we started leveraging for compressing terms dictionaries 
in Lucene 8.5: https://issues.apache.org/jira/browse/LUCENE-4702.

 

I don't know how well this is known but our build also has logic to create 
multi-release JARs. We don't use it in master today but it's used on branch_8x, 
which requires Java 8, in order to use APIs that were introduced in Java 9 such 
as Arrays#mismatch. See the "patch-mr-jar" target in the branch_8x build: 
https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602.
 So if APIs that could help performance were 

Re: Performance in Solr 9 / Java 11

[Adrien Grand]

Tomoko is correct, an MR JAR is created not only upon release but also
every time you create a lucene-core JAR on branch_8x.

On Sun, Aug 30, 2020 at 5:49 AM Tomoko Uchida 
wrote:

> I believe mr-jar build is enabled in the 8x branch (LUCENE-7966), and the
> workaround was dropped on the master branch when the minimum java version
> was bumped up to java 11 (LUCENE-8738); if my understanding is correct.
>
> $ jar tf core/lucene-core-8.6.1.jar | grep META-INF/versions
> META-INF/versions/
> META-INF/versions/9/
> META-INF/versions/9/org/
> META-INF/versions/9/org/apache/
> ...
>
> $ jar tf core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar | grep
> META-INF/versions
> // no outputs
>
>
>
>
> 2020年8月30日(日) 6:48 Mike Drob :
>
>> Do you know if these mr-jars are built by default as part of the release
>> process? I definitely had no idea about them when doing 8.5.2 and did not
>> even think to verify anything about it.
>>
>> On Sat, Aug 29, 2020 at 4:05 PM Adrien Grand  wrote:
>>
>>> It may only be indirectly related to your question, but there is support
>>> for vectorized operations of byte[] arrays that was added in JDK 13 (this
>>> blog https://richardstartin.github.io/posts/vectorised-byte-operations 
>>> explains
>>> well what it is about) that we started leveraging for compressing terms
>>> dictionaries in Lucene 8.5:
>>> https://issues.apache.org/jira/browse/LUCENE-4702.
>>>
>>> I don't know how well this is known but our build also has logic to
>>> create multi-release JARs. We don't use it in master today but it's used on
>>> branch_8x, which requires Java 8, in order to use APIs that were introduced
>>> in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the
>>> branch_8x build:
>>> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602.
>>> So if APIs that could help performance were introduced in say JDK 15, we
>>> might still be able to leverage them in Lucene/Solr 9 using the same
>>> mechanism.
>>>
>>>
>>>
>>> On Tue, Aug 11, 2020 at 1:12 AM Marcus Eagan 
>>> wrote:
>>>
 In my IDE, I have a few profiling tools that I bounce between that I
 started using in my work at Lucidworks but I continue to use in my current
 work today. I have suspicions that there may be some performance
 improvements in Java 11 that we can exploit further.  I'm curious as to if
 there has been any investigation, possibly Mark Miller or
 @u...@thetaphi.de ,  into performance improvements
 specific to the newer version of Java in Master? There are some obvious
 ones that we get for free, like a better GC, but curious as to prior work
 in this area before publishing anything that might be redundant or
 irrelevant.

 Best,

 --
 Marcus Eagan




>>>
>>> --
>>> Adrien
>>>
>>>
>>>

-- 
Adrien

[RESULT] [VOTE] Release Lucene/Solr 8.6.2 RC1

[Ignacio Vera]

It's been >72h since the vote was initiated and the result is:


+1  12  (10 binding)

 0  0

-1  0


This vote has PASSED