I have a fix for the second one and I move the permanent one to exhaustive here: https://gerrit.cloudera.org/#/c/13967/.
I agree we don't need to test all those UDFs in a serial fashion - it's still useful to make sure we can load a variety of hive UDFs but I think we could just do that in test_udfs.py. Actually I think a lot of the permanent UDFs don't need to be custom cluster tests in principle - if we always set --local_library_dir for the test cluster and created the UDFs in a unique database, I think we could do all of the tests in the parallel suite, except for the ones that stop and start the cluster. On Tue, Aug 6, 2019 at 6:31 AM Tamas Mate <tm...@cloudera.com> wrote: > From the longest tests these two looks like an easy ~5-8 min win. What do > you think? > > 1) test_permanent_udfs.py::test_java_udfs_hive_integration - runs for ~5 > minutes consistently, I think it is not necessary to execute it with this > amount of UDFS > > https://github.com/apache/impala/blob/master/tests/custom_cluster/test_permanent_udfs.py#L187 > > 2) test_breakpad.py::test_abort_writes_minidump - runs for 4 minutes most > probably due to the CLUSTER_WAIT_TIMEOUT_IN_SECONDS, this could be set > lower for this test > > https://github.com/apache/impala/blob/master/tests/custom_cluster/test_breakpad.py#L155 > > https://github.com/apache/impala/blob/master/tests/common/impala_cluster.py#L67 > > On Tue, Aug 6, 2019 at 2:10 PM Tamas Mate <tm...@cloudera.com> wrote: > > > The source file was corrupted, that is why the sentry curiosity came up. > > Updated the Gist > > <https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe> with > > the new report. > > > > On Mon, Aug 5, 2019 at 8:35 PM Tamas Mate <tm...@cloudera.com> wrote: > > > >> No, possibly the mail server rejected it without notification. Uploaded > >> as a secret Gist > >> <https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe>. > >> > >> On Mon, Aug 5, 2019 at 7:44 PM Tim Armstrong <tarmstr...@cloudera.com> > >> wrote: > >> > >>> Did you forget to attach it? > >>> > >>> On Mon, Aug 5, 2019 at 10:32 AM Tamas Mate <tm...@cloudera.com> wrote: > >>> > >>> > Hi Tim, All, > >>> > > >>> > I made a quick log analysis on the linked job. Just blindly checked > the > >>> > differences between the consecutive lines to look for hotspots, so > slow > >>> > operations that are logging continuously remain hidden. There are > some > >>> > tests that running for minutes, will start checking them later. Also, > >>> there > >>> > is this curiosity, I haven't looked into it yet either: > >>> > 21:15:09 --> Starting the Sentry Policy Server > >>> > 21:21:45 Stopping Sentry > >>> > 21:27:56 19/08/04 21:06:23 INFO testutil.SentryServicePinger: Sentry > >>> > Service ping succeeded. > >>> > 21:27:56 --> Starting Ranger Server > >>> > 21:30:39 Restarting Apache Ranger Admin > >>> > 21:30:39 Apache Ranger Admin Service is not running > >>> > 21:34:16 Starting Apache Ranger Admin Service > >>> > 21:34:16 Apache Ranger Admin Service with pid 41746 has started. > >>> > > >>> > Attached the report to this mail. > >>> > > >>> > Best regards, > >>> > Tamas > >>> > > >>> > On Mon, Aug 5, 2019 at 8:28 AM Tim Armstrong < > tarmstr...@cloudera.com> > >>> > wrote: > >>> > > >>> >> Hi All, > >>> >> Just letting you know I tweaked the precommit testing job - > >>> >> https://jenkins.impala.io/job/parallel-all-tests - so that it runs > >>> the > >>> >> backend and frontend tests separately from the other tests. > >>> >> > >>> >> If you have ideas for reducing build+test runtime, please share or > >>> >> implement them. The frontend authorisation tests are running way too > >>> long, > >>> >> but the long pole is still the other job with the Python tests. I > >>> looked > >>> >> at > >>> >> a recent job ( > >>> >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/ > >>> >> < > >>> > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/consoleFull > >>> >) > >>> >> and > >>> >> here's the time taken per step: > >>> >> > >>> >> 0:00:20 Checked out repos > >>> >> 0:01:04 System setup done > >>> >> 0:00:27 Hadoop-lzo built > >>> >> 0:00:33 python bootstrap > >>> >> 0:00:47 download toolchain and cdh components > >>> >> 0:29:36 build impala > >>> >> 0:34:42 load data > >>> >> 0:45:08 serial tests > >>> >> 0:03:45 mini stress tests (all skipped or xfailed) > >>> >> 0:56:16 parallel tests > >>> >> 1:37:49 custom cluster tests > >>> >> 0:15:55 auth tests > >>> >> 0:01:48 archiving > >>> >> > >>> >> - Tim > >>> >> > >>> > > >>> > > >>> > >> > >> > >> -- > >> Tamas Mate > >> Senior Customer Operations Engineer > >> Cloudera > >> > > > > > > -- > > Tamas Mate > > Senior Customer Operations Engineer > > Cloudera > > > > > -- > Tamas Mate > Senior Customer Operations Engineer > Cloudera >
