Lucene Indexing Speed Trips, Deadlocks

[Mark Miller]

So I've never really been in the position of having the time or tools to
easily and efficiently pound on Lucene updates - always lacked one of the
ingredients, and having a beer while a bit stocked up on both, I've been
doing some light hammering.

Not likely some high priority item, when I say efficient, I mean lots
coming in fast with minimal context switching or outside blocking and
locking - not that I'm in some reactive / back pressure bulldozer. But I
thought it might be interesting to someone, because I think I've seen hints
of it at a lesser scale when pointing any fingers towards Lucene was beyond
the effort or value.

So just a drop, don't mind a reply of "seems to hurt when I hit you" "Well
don't hit me so hard".

Somewhat older code, looks like maybe the issue could still hang out.

Seems to be the applylock in FrozenUpdates. I don't recall all the paths
coming in on it (I can easily reproduce), but it takes a few things
occurring around the same time, and the result ranges from massive slow
down that works through to essentially deadlock.

Seen a lot of this kind of behavior before, hard breaks to deadlock but
often hints of progress unwinding, so I gave a fair option on the lock a
try and seemed to address it for me without a noticeable penalty. I didn't
do rigorous testing and came at it from one angle. But took me to smooth
pounding from the off and on misery runs.


Also, unrelated, but since I saw someone struggling with it recently, I
might as well mention, I think there may be a few SPI related poor static
initializers in CharFilterFactory, TokenFilterFactory, TokenizerFactory -
don't quote me on the classes, but I believe a nice static holder pattern
there solves some fairly easy to trigger deadlock issues. Worked well for
me and I later found a similar class that already had this pattern with a
note about the deadlock doom impetus.

Sorry to pound, back pressure aint easy and "reactive" gets me about as
excited as Perl.

- Mark

Re: StoreField with long value using python3

[Andi Vajda]

On Tue, 20 Apr 2021, antony wrote:



Hello,

Anyone help me how to use StoredField(String name, long value). Its not
working because there is no 'long integer' in Python 3.


Not sure that's the problem.
It looks like the StoredField(int) overload is called before the 
StoredField(long). This is a bug.


A possible workaround would be to edit the StoredField class to add a 
constructor accepting a java.lang.Long() and then passing it a Long(cdate) ?

especially since all it does is set Object fieldsData on Field anyway...

I didn't try this out...

Andi..



I am using Python 3.8.0 and pylucene 8.6.1.

Source code:

cdate = int(datetime.datetime.now().strftime("%Y%m%d%H%M"))

print(1, cdate)

doc = Document()
doc.add(LongPoint('cdatetime', cdate))
doc.add(StoredField('cdatetime', cdate))

print(2, doc)

writer.addDocument(doc)
writer.commit()
writer.close()

Output:

1, cdate = 202104202312

2, Document
*stored*>


Thanks,
Antony



--
Sent from: http://pylucene-users-developers-list.2474766.n2.nabble.com/

StoreField with long value using python3

[antony]

Hello,

Anyone help me how to use StoredField(String name, long value). Its not
working because there is no 'long integer' in Python 3.

I am using Python 3.8.0 and pylucene 8.6.1. 

Source code:

cdate = int(datetime.datetime.now().strftime("%Y%m%d%H%M"))

print(1, cdate)

doc = Document()
doc.add(LongPoint('cdatetime', cdate))
doc.add(StoredField('cdatetime', cdate))

print(2, doc)

writer.addDocument(doc)
writer.commit()
writer.close()

Output:

1, cdate = 202104202312

2, Document
*stored*>


Thanks,
Antony



--
Sent from: http://pylucene-users-developers-list.2474766.n2.nabble.com/

Re: Helping to update Lucene FAQ

[Michael Wechner]

great!

I have registered myself, whereas my username is "michael.wechner". Can 
you give me edit/write access for the FAQ page?


I think some changes/updates are very clear, but what about changes I am 
not 100% sure about? Should I first ask/discuss them on this mailing 
list or what would be the best approach?


Thanks

Michael


Am 20.04.21 um 17:39 schrieb Michael McCandless:
That would be great and helpful, Michael!  Progress not perfection :)  
Those links are indeed ancient.


I think (not certain) that we have a way (symlink somewhere) for our 
javadocs links to automatically go to the latest release so we would 
not have to update these links on every release.


I thought it was something like https://lucene.apache.org/core/latest/ 
 but that is clearly wrong 
(HTTP 404).  Hrmph.


Thank you for digging!

Mike McCandless

http://blog.mikemccandless.com 


On Mon, Apr 19, 2021 at 12:20 PM Michael Wechner 
mailto:michael.wech...@wyona.com>> wrote:


Hi

I am currently doing some work re FAQ in general and noticed that the
following Lucene FAQ

https://cwiki.apache.org/confluence/display/lucene/LuceneFAQ


are not quite up to date. I read that you are looking for help

https://cwiki.apache.org/confluence/display/lucene


and I would be happy to help updating them, for example updating
the links

replace
https://lucene.apache.org/core/3_4_0/gettingstarted.html

by
https://lucene.apache.org/core/8_8_2/index.html


or

replace

https://lucene.apache.org/core/3_4_0/api/core/overview-summary.html#overview_description


by
https://lucene.apache.org/core/8_8_2/core/index.html


Would that be of help or do you have other plans re the FAQs?

All the best

Michael

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org

For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Helping to update Lucene FAQ

[Michael McCandless]

That would be great and helpful, Michael!  Progress not perfection :)
Those links are indeed ancient.

I think (not certain) that we have a way (symlink somewhere) for our
javadocs links to automatically go to the latest release so we would not
have to update these links on every release.

I thought it was something like https://lucene.apache.org/core/latest/ but
that is clearly wrong (HTTP 404).  Hrmph.

Thank you for digging!

Mike McCandless

http://blog.mikemccandless.com


On Mon, Apr 19, 2021 at 12:20 PM Michael Wechner 
wrote:

> Hi
>
> I am currently doing some work re FAQ in general and noticed that the
> following Lucene FAQ
>
> https://cwiki.apache.org/confluence/display/lucene/LuceneFAQ
>
> are not quite up to date. I read that you are looking for help
>
> https://cwiki.apache.org/confluence/display/lucene
>
> and I would be happy to help updating them, for example updating the links
>
> replace
> https://lucene.apache.org/core/3_4_0/gettingstarted.html
> by
> https://lucene.apache.org/core/8_8_2/index.html
>
> or
>
> replace
>
> https://lucene.apache.org/core/3_4_0/api/core/overview-summary.html#overview_description
> by
> https://lucene.apache.org/core/8_8_2/core/index.html
>
> Would that be of help or do you have other plans re the FAQs?
>
> All the best
>
> Michael
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Welcome Zach Chen as Lucene committer

[Bruno Roustant]

Welcome Zach!

Le mar. 20 avr. 2021 à 10:59, Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> a écrit :

> Congrats, Zach! Thanks for your contributions, looking forward to more!
>
> On Tue, 20 Apr, 2021, 2:26 pm Alan Woodward,  wrote:
>
>> Congratulations and welcome!
>>
>> > On 19 Apr 2021, at 15:13, Adrien Grand  wrote:
>> >
>> > I'm pleased to announce that Zach Chen has accepted the PMC's
>> invitation to become a committer.
>> >
>> > Zach, the tradition is that new committers introduce themselves with a
>> brief bio.
>> >
>> > Congratulations and welcome!
>> >
>> > --
>> > Adrien
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

JDK 17 Early Access build 18 is available

[Rory O'Donnell]

*Hi Uwe & Dawid, *

*OpenJDK 17 Early Access build 18is now available at 
**https://jdk.java.net/17 *


 * These early-access , open-source builds are provided under the
 o GNU General Public License, version 2, with the Classpath
   Exception 
 * Release Notes are available at http://jdk.java.net/17/release-notes
   


**G1 pauses may be extremely long with EA build JDK-17+18*

*During performance testing we noticed that due to a recent change 
(JDK-8262068) GC pauses after a G1 full GC may be extremely slow. The 
problem has been fixed with JDK-8264987 and that has already been 
integrated. This change will be available with the following EA build  
JDK-17+19.  For more technical info please see [1]



*JEP 382 [2]**  - Starting with build 19, **JDK 17 for macOS is 
*temporarily* switched from using OpenGL**to using Apple's Metal 
API**for Java 2D rendering.*


Heads up to anyone who is testing JDK 17 for running apps on macOS. 
Starting with build 19, JDK 17 for macOS is *temporarily* switched from 
using OpenGL to using Apple's Metal API for Java 2D rendering.


If you are running any kind of 2D / Swing/ AWT UI application on macOS, 
and see any rendering related problems
starting with JDK 17 b19, please do report them to us along with a test 
case and screen shots.


You may also set "-Dsun.java2d.opengl=true" to re-enable OpenGL - which  
implicitly disables Metal - to confirm that it is a Metal related 
rendering glltch.



Rgds,Rory

[1] 
https://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2021-April/034745.html

[2] https://openjdk.java.net/jeps/382

--
Rgds, Rory O'Donnell
Quality Engineering Manager
Oracle EMEA, Dublin, Ireland

Re: Welcome Zach Chen as Lucene committer

[Ishan Chattopadhyaya]

Congrats, Zach! Thanks for your contributions, looking forward to more!

On Tue, 20 Apr, 2021, 2:26 pm Alan Woodward,  wrote:

> Congratulations and welcome!
>
> > On 19 Apr 2021, at 15:13, Adrien Grand  wrote:
> >
> > I'm pleased to announce that Zach Chen has accepted the PMC's invitation
> to become a committer.
> >
> > Zach, the tradition is that new committers introduce themselves with a
> brief bio.
> >
> > Congratulations and welcome!
> >
> > --
> > Adrien
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Welcome Zach Chen as Lucene committer

[Alan Woodward]

Congratulations and welcome!

> On 19 Apr 2021, at 15:13, Adrien Grand  wrote:
> 
> I'm pleased to announce that Zach Chen has accepted the PMC's invitation to 
> become a committer.
> 
> Zach, the tradition is that new committers introduce themselves with a brief 
> bio.
> 
> Congratulations and welcome!
> 
> -- 
> Adrien


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Zach Chen as Lucene committer

[Ignacio Vera]

Congratulations and welcome!

On Tue, Apr 20, 2021 at 4:36 AM Tomás Fernández Löbbe 
wrote:

> Welcome Zach!
>
> On Mon, Apr 19, 2021 at 3:38 PM Greg Miller  wrote:
>
>> Congrats Zach!
>>
>> On Mon, Apr 19, 2021 at 3:09 PM Robert Muir  wrote:
>> >
>> > Congratulations!
>> >
>> >
>> > On Mon, Apr 19, 2021 at 10:14 AM Adrien Grand 
>> wrote:
>> >>
>> >> I'm pleased to announce that Zach Chen has accepted the PMC's
>> invitation to become a committer.
>> >>
>> >> Zach, the tradition is that new committers introduce themselves with a
>> brief bio.
>> >>
>> >> Congratulations and welcome!
>> >>
>> >> --
>> >> Adrien
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>