RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-19 Thread Uwe Schindler
Hi Robert,

 

+1

 

One small thing: It should use java.io.tmpdir, but append its own process id or 
something like that to it, so multiple runs (like on Jeekins) in parallel do 
not override each other. Maybe just use tmpname (the corresponding 
java.nio.file.Files function) or similar to generate a new temporary directory 
that’s unique. I think ANT can do this for us (I think there is an ANT task to 
get a temporary name, which is assigned to a property).

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Robert Muir  
Sent: Sunday, January 19, 2020 11:47 PM
To: dev@lucene.apache.org
Subject: Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

agreed. for all situations it may not help. but for my test 0 bytes were left 
at the end. i watched the io and usage as tests ran and the usage only even 
peaks at megabytes. 

 

just death by a thousand stabs... literally. gazillions of tests writing only a 
few MB each.

 

i think we can all agree these test indexes are "temp data"? so if the build 
respects java.io.tmpdir here, then imo it behaves nicely and gives user the 
chance to prevent hardware destruction.

 

On Sun, Jan 19, 2020, 3:36 PM Dawid Weiss mailto:dawid.we...@gmail.com> > wrote:

 

Maybe we should open an issue. For example it would be good to fix the gradle 
build here too.

 

It's definitely possible to relocate those folders (gradle-wise). I don't think 
it buys anything for Windows users or for those who have /tmp mounted on a 
physical drive but if you know what you're doing then it should also make tests 
faster.

 

Dawid



Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-19 Thread Robert Muir
agreed. for all situations it may not help. but for my test 0 bytes were
left at the end. i watched the io and usage as tests ran and the usage only
even peaks at megabytes.

just death by a thousand stabs... literally. gazillions of tests writing
only a few MB each.

i think we can all agree these test indexes are "temp data"? so if the
build respects java.io.tmpdir here, then imo it behaves nicely and gives
user the chance to prevent hardware destruction.

On Sun, Jan 19, 2020, 3:36 PM Dawid Weiss  wrote:

>
>
>> Maybe we should open an issue. For example it would be good to fix the
>> gradle build here too.
>>
>
> It's definitely possible to relocate those folders (gradle-wise). I don't
> think it buys anything for Windows users or for those who have /tmp mounted
> on a physical drive but if you know what you're doing then it should also
> make tests faster.
>
> Dawid
>


Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-19 Thread Uwe Schindler
...and safe your SSD from death.

Some statistics: in 12 hours the macOS slave wrote 114 Gigabytes to it's 
virtual harddisk - according to Darwin's top implementation, which displays 
disk IO since bootup of kernel.

Uwe

Am January 19, 2020 8:35:55 PM UTC schrieb Dawid Weiss :
>> Maybe we should open an issue. For example it would be good to fix
>the
>> gradle build here too.
>>
>
>It's definitely possible to relocate those folders (gradle-wise). I
>don't
>think it buys anything for Windows users or for those who have /tmp
>mounted
>on a physical drive but if you know what you're doing then it should
>also
>make tests faster.
>
>Dawid

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-19 Thread Dawid Weiss
> Maybe we should open an issue. For example it would be good to fix the
> gradle build here too.
>

It's definitely possible to relocate those folders (gradle-wise). I don't
think it buys anything for Windows users or for those who have /tmp mounted
on a physical drive but if you know what you're doing then it should also
make tests faster.

Dawid


Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-19 Thread Robert Muir
On Thu, Jan 16, 2020 at 2:38 AM Uwe Schindler  wrote:

> So Lucene and Solr tests are eating your SSD, be aware of that!!! 
>
>
By default all the tests are writing indexes to build/ directory where your
source checkout is, I think this is a bad default?

As a workaround I can manually pass -Dtests.workDir=/tmp/folder to 'ant' so
that test temporary data only hits a tmpfs mount. All the other stuff
(build classes, test xmls, etc) still goes to the usual places, just not
the gigabytes of indexes being created and destroyed.

Maybe we should open an issue. For example it would be good to fix the
gradle build here too.


RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-17 Thread Uwe Schindler
Sorry false positive, Solaris hung again.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

 <https://www.thetaphi.de> https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Uwe Schindler  
Sent: Friday, January 17, 2020 7:21 PM
To: dev@lucene.apache.org
Subject: RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

Hi,

 

I updated the virtualization on the server. It looks like Solaris is working 
again. But I have seen only one build yet – which passed without hangs. Builds 
are back enabled.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

 <https://www.thetaphi.de> https://www.thetaphi.de

eMail: u...@thetaphi.de <mailto:u...@thetaphi.de> 

 

From: Uwe Schindler mailto:u...@thetaphi.de> > 
Sent: Thursday, January 16, 2020 9:30 AM
To: dev@lucene.apache.org <mailto:dev@lucene.apache.org> 
Subject: RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

Actually the Solaris builds are sometimes hanging forever (more often with new 
hardware). So they don’t pollute mailinglist they just block executor. I have 
to manually kill them.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

 <https://www.thetaphi.de> https://www.thetaphi.de

eMail: u...@thetaphi.de <mailto:u...@thetaphi.de> 

 

From: Dawid Weiss mailto:dawid.we...@gmail.com> > 
Sent: Thursday, January 16, 2020 9:24 AM
To: Lucene Dev mailto:dev@lucene.apache.org> >
Subject: Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

 

With new CPU, builds seem up to 2 times faster. FYI, the Linux builds are 
running with tests.multiplicator=3 to better trigger JVM failures. You have to 
remind about that when you reproduce failures - it looks like this is not 
printed in reproduce lines.

 

It is printed with gradle builds (any non-defaults are). The option is 
"tests.multiplier" actually; check if you don't have a typo there?

 

But we can also drop Solaris builds, as they only work with 8.x - no Java 11 
support on Solaris anymore by Oracle. Any comments?

 

As somebody who tries to look at each failure sent to the mailing list (at 
least recently...) I still think the EA builds that are KNOWN to be buggy 
should be disabled until next upgrade. They pollute the mailing list with so 
many identical errors that this discourages one from looking through the logs.

 

Dawid 



RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-17 Thread Uwe Schindler
Hi,

 

I updated the virtualization on the server. It looks like Solaris is working 
again. But I have seen only one build yet – which passed without hangs. Builds 
are back enabled.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

 <https://www.thetaphi.de> https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Uwe Schindler  
Sent: Thursday, January 16, 2020 9:30 AM
To: dev@lucene.apache.org
Subject: RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

Actually the Solaris builds are sometimes hanging forever (more often with new 
hardware). So they don’t pollute mailinglist they just block executor. I have 
to manually kill them.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

 <https://www.thetaphi.de> https://www.thetaphi.de

eMail: u...@thetaphi.de <mailto:u...@thetaphi.de> 

 

From: Dawid Weiss mailto:dawid.we...@gmail.com> > 
Sent: Thursday, January 16, 2020 9:24 AM
To: Lucene Dev mailto:dev@lucene.apache.org> >
Subject: Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

 

With new CPU, builds seem up to 2 times faster. FYI, the Linux builds are 
running with tests.multiplicator=3 to better trigger JVM failures. You have to 
remind about that when you reproduce failures - it looks like this is not 
printed in reproduce lines.

 

It is printed with gradle builds (any non-defaults are). The option is 
"tests.multiplier" actually; check if you don't have a typo there?

 

But we can also drop Solaris builds, as they only work with 8.x - no Java 11 
support on Solaris anymore by Oracle. Any comments?

 

As somebody who tries to look at each failure sent to the mailing list (at 
least recently...) I still think the EA builds that are KNOWN to be buggy 
should be disabled until next upgrade. They pollute the mailing list with so 
many identical errors that this discourages one from looking through the logs.

 

Dawid 



Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-17 Thread Dawid Weiss
>> It is printed with gradle builds (any non-defaults are). The option is 
>> "tests.multiplier" actually; check if you don't have a typo there?

I think something like this could be added with relative ease. I don't
know how much useful it is though. :)

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-17 Thread Michael McCandless
On Thu, Jan 16, 2020 at 3:25 AM Dawid Weiss  wrote:

>
>
>> With new CPU, builds seem up to 2 times faster. FYI, the Linux builds are
>> running with tests.multiplicator=3 to better trigger JVM failures. You have
>> to remind about that when you reproduce failures - it looks like this is
>> not printed in reproduce lines.
>>
>
> It is printed with gradle builds (any non-defaults are). The option is
> "tests.multiplier" actually; check if you don't have a typo there?
>

Hmm if we do accidentally have a typo, will the new gradle build catch it?
Or is it still lenient, like the ant build?


> But we can also drop Solaris builds, as they only work with 8.x - no Java
>> 11 support on Solaris anymore by Oracle. Any comments?
>
>
> As somebody who tries to look at each failure sent to the mailing list (at
> least recently...) I still think the EA builds that are KNOWN to be buggy
> should be disabled until next upgrade. They pollute the mailing list with
> so many identical errors that this discourages one from looking through the
> logs.
>

+1

Mike McCandless

http://blog.mikemccandless.com

>


RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-16 Thread Uwe Schindler
Actually it’s printed in reproduce line. Sorry for the false alarm.

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

 <https://www.thetaphi.de> https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Uwe Schindler  
Sent: Thursday, January 16, 2020 9:29 AM
To: dev@lucene.apache.org
Subject: RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

$ cat lucene.build.properties

tests.jvms=6

tests.multiplier=3

 

-

UWE SCHINDLER

Software Architecture, Apache Lucene, Elasticsearch

PANGAEA - Data Publisher for Earth & Environmental Science

MARUM (FVG Ost building) - University of Bremen

Room 0210, Leobener Str. 2, D-28359 Bremen

Tel.: +49 421 218 65595

Fax:  +49 421 218 65505

 <https://www.pangaea.de/> https://www.pangaea.de/

E-mail: uschind...@pangaea.de <mailto:uschind...@pangaea.de> 

 

From: Dawid Weiss mailto:dawid.we...@gmail.com> > 
Sent: Thursday, January 16, 2020 9:24 AM
To: Lucene Dev mailto:dev@lucene.apache.org> >
Subject: Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

 

With new CPU, builds seem up to 2 times faster. FYI, the Linux builds are 
running with tests.multiplicator=3 to better trigger JVM failures. You have to 
remind about that when you reproduce failures - it looks like this is not 
printed in reproduce lines.

 

It is printed with gradle builds (any non-defaults are). The option is 
"tests.multiplier" actually; check if you don't have a typo there?

 

But we can also drop Solaris builds, as they only work with 8.x - no Java 11 
support on Solaris anymore by Oracle. Any comments?

 

As somebody who tries to look at each failure sent to the mailing list (at 
least recently...) I still think the EA builds that are KNOWN to be buggy 
should be disabled until next upgrade. They pollute the mailing list with so 
many identical errors that this discourages one from looking through the logs.

 

Dawid 



RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-16 Thread Uwe Schindler
Actually the Solaris builds are sometimes hanging forever (more often with new 
hardware). So they don’t pollute mailinglist they just block executor. I have 
to manually kill them.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

 <https://www.thetaphi.de> https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Dawid Weiss  
Sent: Thursday, January 16, 2020 9:24 AM
To: Lucene Dev 
Subject: Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

 

With new CPU, builds seem up to 2 times faster. FYI, the Linux builds are 
running with tests.multiplicator=3 to better trigger JVM failures. You have to 
remind about that when you reproduce failures - it looks like this is not 
printed in reproduce lines.

 

It is printed with gradle builds (any non-defaults are). The option is 
"tests.multiplier" actually; check if you don't have a typo there?

 

But we can also drop Solaris builds, as they only work with 8.x - no Java 11 
support on Solaris anymore by Oracle. Any comments?

 

As somebody who tries to look at each failure sent to the mailing list (at 
least recently...) I still think the EA builds that are KNOWN to be buggy 
should be disabled until next upgrade. They pollute the mailing list with so 
many identical errors that this discourages one from looking through the logs.

 

Dawid 



RE: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-16 Thread Uwe Schindler
$ cat lucene.build.properties

tests.jvms=6

tests.multiplier=3

 

-

UWE SCHINDLER

Software Architecture, Apache Lucene, Elasticsearch

PANGAEA - Data Publisher for Earth & Environmental Science

MARUM (FVG Ost building) - University of Bremen

Room 0210, Leobener Str. 2, D-28359 Bremen

Tel.: +49 421 218 65595

Fax:  +49 421 218 65505

 <https://www.pangaea.de/> https://www.pangaea.de/

E-mail: uschind...@pangaea.de

 

From: Dawid Weiss  
Sent: Thursday, January 16, 2020 9:24 AM
To: Lucene Dev 
Subject: Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 
ECC, 2x1 Terabyte NVME SSD as RAID

 

 

With new CPU, builds seem up to 2 times faster. FYI, the Linux builds are 
running with tests.multiplicator=3 to better trigger JVM failures. You have to 
remind about that when you reproduce failures - it looks like this is not 
printed in reproduce lines.

 

It is printed with gradle builds (any non-defaults are). The option is 
"tests.multiplier" actually; check if you don't have a typo there?

 

But we can also drop Solaris builds, as they only work with 8.x - no Java 11 
support on Solaris anymore by Oracle. Any comments?

 

As somebody who tries to look at each failure sent to the mailing list (at 
least recently...) I still think the EA builds that are KNOWN to be buggy 
should be disabled until next upgrade. They pollute the mailing list with so 
many identical errors that this discourages one from looking through the logs.

 

Dawid 



Re: New Policeman Jenkins Hardware: AMD Ryzen 7 3700X Octa-Core, DDR4 ECC, 2x1 Terabyte NVME SSD as RAID

2020-01-16 Thread Dawid Weiss
> With new CPU, builds seem up to 2 times faster. FYI, the Linux builds are
> running with tests.multiplicator=3 to better trigger JVM failures. You have
> to remind about that when you reproduce failures - it looks like this is
> not printed in reproduce lines.
>

It is printed with gradle builds (any non-defaults are). The option is
"tests.multiplier" actually; check if you don't have a typo there?


> But we can also drop Solaris builds, as they only work with 8.x - no Java
> 11 support on Solaris anymore by Oracle. Any comments?


As somebody who tries to look at each failure sent to the mailing list (at
least recently...) I still think the EA builds that are KNOWN to be buggy
should be disabled until next upgrade. They pollute the mailing list with
so many identical errors that this discourages one from looking through the
logs.

Dawid