Re: Question regarding pipeline performances
Hi Emmanuel, yes, the Jenkins machines are rather slow, especially the H2x and H3x nodes. I run lscpu and free -m on some, most show Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU E5-2620 v2 (from Sept 2013) and 96 GB RAM But I assume those are highly overprovisioned VMs with spinning disks only. If you want more info you have to ask infra. See also the recent discussion with Shawn [1]. But I don't mind, they are provided "for free" to the ASF and we can use them. Kind Regards, Stefan [1] https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E On 6/24/19 5:18 PM, Emmanuel Lécharny wrote: > Hi ! > > I'm just wondering on which kind of machine are the pipelines running? > The tests are taking almost forever. For instance, a test were we add > 1000 entries and search using an index takes 9 seconds on my computer > (ok, it's a fast one), and more 300 seconds on the pipeline.. > > ... > > [INFO] Running > org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT > > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 314.49 s - in > org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT > > > ... > > > That's more than 3O times slower ! > > > - > To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org > For additional commands, e-mail: dev-h...@directory.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org
Re: Question regarding pipeline performances
Many thanks, Stefan. It makes sense now. I have modified the test that add 1000 entries before running so that those additions are made within a single transaction. That should speedup the addition considerably (on my laptop, the test now takes 6 seconds instead of 10, and most of the time is spent elsewhere - the entries addition takes less than 1 second compared to 4 seconds -). I expect to see the test running in way less time than 300 seconds now. I also have disabled the recording feature (we don't revert anything), which brings another huge speedup: the test now runs in 1.4 seconds on my computer Let's see what it gives in the pipeline result :-) Side note: this is exactly what I want to do for the Fortress tests: use external transaction to gather injections of data in one single database update. On 25/06/2019 00:04, Stefan Seelmann wrote: Hi Emmanuel, yes, the Jenkins machines are rather slow, especially the H2x and H3x nodes. I run lscpu and free -m on some, most show Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU E5-2620 v2 (from Sept 2013) and 96 GB RAM But I assume those are highly overprovisioned VMs with spinning disks only. If you want more info you have to ask infra. See also the recent discussion with Shawn [1]. But I don't mind, they are provided "for free" to the ASF and we can use them. Kind Regards, Stefan [1] https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E On 6/24/19 5:18 PM, Emmanuel Lécharny wrote: Hi ! I'm just wondering on which kind of machine are the pipelines running? The tests are taking almost forever. For instance, a test were we add 1000 entries and search using an index takes 9 seconds on my computer (ok, it's a fast one), and more 300 seconds on the pipeline.. ... [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 314.49 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT ... That's more than 3O times slower ! - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org
Re: Question regarding pipeline performances
A bit better (3x faster): [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.514 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT On 25/06/2019 03:05, Emmanuel Lécharny wrote: Many thanks, Stefan. It makes sense now. I have modified the test that add 1000 entries before running so that those additions are made within a single transaction. That should speedup the addition considerably (on my laptop, the test now takes 6 seconds instead of 10, and most of the time is spent elsewhere - the entries addition takes less than 1 second compared to 4 seconds -). I expect to see the test running in way less time than 300 seconds now. I also have disabled the recording feature (we don't revert anything), which brings another huge speedup: the test now runs in 1.4 seconds on my computer Let's see what it gives in the pipeline result :-) Side note: this is exactly what I want to do for the Fortress tests: use external transaction to gather injections of data in one single database update. On 25/06/2019 00:04, Stefan Seelmann wrote: Hi Emmanuel, yes, the Jenkins machines are rather slow, especially the H2x and H3x nodes. I run lscpu and free -m on some, most show Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU E5-2620 v2 (from Sept 2013) and 96 GB RAM But I assume those are highly overprovisioned VMs with spinning disks only. If you want more info you have to ask infra. See also the recent discussion with Shawn [1]. But I don't mind, they are provided "for free" to the ASF and we can use them. Kind Regards, Stefan [1] https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E On 6/24/19 5:18 PM, Emmanuel Lécharny wrote: Hi ! I'm just wondering on which kind of machine are the pipelines running? The tests are taking almost forever. For instance, a test were we add 1000 entries and search using an index takes 9 seconds on my computer (ok, it's a fast one), and more 300 seconds on the pipeline.. ... [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 314.49 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT ... That's more than 3O times slower ! - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org
Re: Question regarding pipeline performances
Trying something more atm : adding txn around the revert operation. That should speedup all the tests that uses the revert feature. On 25/06/2019 08:18, Emmanuel Lécharny wrote: A bit better (3x faster): [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.514 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT On 25/06/2019 03:05, Emmanuel Lécharny wrote: Many thanks, Stefan. It makes sense now. I have modified the test that add 1000 entries before running so that those additions are made within a single transaction. That should speedup the addition considerably (on my laptop, the test now takes 6 seconds instead of 10, and most of the time is spent elsewhere - the entries addition takes less than 1 second compared to 4 seconds -). I expect to see the test running in way less time than 300 seconds now. I also have disabled the recording feature (we don't revert anything), which brings another huge speedup: the test now runs in 1.4 seconds on my computer Let's see what it gives in the pipeline result :-) Side note: this is exactly what I want to do for the Fortress tests: use external transaction to gather injections of data in one single database update. On 25/06/2019 00:04, Stefan Seelmann wrote: Hi Emmanuel, yes, the Jenkins machines are rather slow, especially the H2x and H3x nodes. I run lscpu and free -m on some, most show Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU E5-2620 v2 (from Sept 2013) and 96 GB RAM But I assume those are highly overprovisioned VMs with spinning disks only. If you want more info you have to ask infra. See also the recent discussion with Shawn [1]. But I don't mind, they are provided "for free" to the ASF and we can use them. Kind Regards, Stefan [1] https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E On 6/24/19 5:18 PM, Emmanuel Lécharny wrote: Hi ! I'm just wondering on which kind of machine are the pipelines running? The tests are taking almost forever. For instance, a test were we add 1000 entries and search using an index takes 9 seconds on my computer (ok, it's a fast one), and more 300 seconds on the pipeline.. ... [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 314.49 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT ... That's more than 3O times slower ! - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org
Re: Question regarding pipeline performances
Ok, commited a big change: reverts are now processed within a global transaction, which saves a LOT of time. On my machine, building apacheds takes 12 mins vs 22 mins before this simple change. I'm waiting for the pipeline to be completed (probably after my lunch) but I expect it to be way faster. On 25/06/2019 10:13, Emmanuel Lécharny wrote: Trying something more atm : adding txn around the revert operation. That should speedup all the tests that uses the revert feature. On 25/06/2019 08:18, Emmanuel Lécharny wrote: A bit better (3x faster): [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.514 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT On 25/06/2019 03:05, Emmanuel Lécharny wrote: Many thanks, Stefan. It makes sense now. I have modified the test that add 1000 entries before running so that those additions are made within a single transaction. That should speedup the addition considerably (on my laptop, the test now takes 6 seconds instead of 10, and most of the time is spent elsewhere - the entries addition takes less than 1 second compared to 4 seconds -). I expect to see the test running in way less time than 300 seconds now. I also have disabled the recording feature (we don't revert anything), which brings another huge speedup: the test now runs in 1.4 seconds on my computer Let's see what it gives in the pipeline result :-) Side note: this is exactly what I want to do for the Fortress tests: use external transaction to gather injections of data in one single database update. On 25/06/2019 00:04, Stefan Seelmann wrote: Hi Emmanuel, yes, the Jenkins machines are rather slow, especially the H2x and H3x nodes. I run lscpu and free -m on some, most show Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU E5-2620 v2 (from Sept 2013) and 96 GB RAM But I assume those are highly overprovisioned VMs with spinning disks only. If you want more info you have to ask infra. See also the recent discussion with Shawn [1]. But I don't mind, they are provided "for free" to the ASF and we can use them. Kind Regards, Stefan [1] https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E On 6/24/19 5:18 PM, Emmanuel Lécharny wrote: Hi ! I'm just wondering on which kind of machine are the pipelines running? The tests are taking almost forever. For instance, a test were we add 1000 entries and search using an index takes 9 seconds on my computer (ok, it's a fast one), and more 300 seconds on the pipeline.. ... [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 314.49 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT ... That's more than 3O times slower ! - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org
Re: Question regarding pipeline performances
Not too bad :-) Linux Java 8 : [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 17:49 min On 25/06/2019 12:41, Emmanuel Lécharny wrote: Ok, commited a big change: reverts are now processed within a global transaction, which saves a LOT of time. On my machine, building apacheds takes 12 mins vs 22 mins before this simple change. I'm waiting for the pipeline to be completed (probably after my lunch) but I expect it to be way faster. On 25/06/2019 10:13, Emmanuel Lécharny wrote: Trying something more atm : adding txn around the revert operation. That should speedup all the tests that uses the revert feature. On 25/06/2019 08:18, Emmanuel Lécharny wrote: A bit better (3x faster): [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.514 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT On 25/06/2019 03:05, Emmanuel Lécharny wrote: Many thanks, Stefan. It makes sense now. I have modified the test that add 1000 entries before running so that those additions are made within a single transaction. That should speedup the addition considerably (on my laptop, the test now takes 6 seconds instead of 10, and most of the time is spent elsewhere - the entries addition takes less than 1 second compared to 4 seconds -). I expect to see the test running in way less time than 300 seconds now. I also have disabled the recording feature (we don't revert anything), which brings another huge speedup: the test now runs in 1.4 seconds on my computer Let's see what it gives in the pipeline result :-) Side note: this is exactly what I want to do for the Fortress tests: use external transaction to gather injections of data in one single database update. On 25/06/2019 00:04, Stefan Seelmann wrote: Hi Emmanuel, yes, the Jenkins machines are rather slow, especially the H2x and H3x nodes. I run lscpu and free -m on some, most show Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU E5-2620 v2 (from Sept 2013) and 96 GB RAM But I assume those are highly overprovisioned VMs with spinning disks only. If you want more info you have to ask infra. See also the recent discussion with Shawn [1]. But I don't mind, they are provided "for free" to the ASF and we can use them. Kind Regards, Stefan [1] https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E On 6/24/19 5:18 PM, Emmanuel Lécharny wrote: Hi ! I'm just wondering on which kind of machine are the pipelines running? The tests are taking almost forever. For instance, a test were we add 1000 entries and search using an index takes 9 seconds on my computer (ok, it's a fast one), and more 300 seconds on the pipeline.. ... [INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 314.49 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT ... That's more than 3O times slower ! - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org - To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org
