yeah, totally weird. i'm actually going to take this moment and clean up the build scripts for both of these jobs. there's a lot of years-old cruft that i'll delete and make things more readable.
On Sun, Dec 8, 2019 at 7:50 PM Sean Owen <sro...@gmail.com> wrote: > > Hm, so they look pretty similar except for minor differences in the > actual script run. Is there any reason this should be different? Would > it be reasonable to try making the 'new' one work like the 'old' one > if the former isn't working? > > But I still can't figure out why it causes the same odd error every > time on this one PR, which is a minor change to tooltips in the UI. I > haven't seen other manually-triggered PR builds fail this way. Really > mysterious so far! > > https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4964/testReport/ > > > Old: > > #!/bin/bash > > set -e # fail on any non-zero exit code > set -x > > export AMPLAB_JENKINS=1 > export PATH="$PATH:/home/anaconda/envs/py3k/bin" > > # Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT > incremental compiler seems to > # ignore our JAVA_HOME and use the system javac instead. > export PATH="$JAVA_HOME/bin:$PATH" > > # Add a pre-downloaded version of Maven to the path so that we avoid > the flaky download step. > export > PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH" > > echo "fixing target dir permissions" > chmod -R +w target/* || true # stupid hack by sknapp to ensure that > the chmod always exits w/0 and doesn't bork the script > > echo "running git clean -fdx" > git clean -fdx > > # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention > export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER" > mkdir -p "$HOME" > export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2" > export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2" > > > ./dev/run-tests-jenkins > > > # Hack to ensure that at least one JVM suite always runs in order to > prevent spurious errors from the > # Jenkins JUnit test reporter plugin > ./build/sbt unsafe/test > /dev/null 2>&1 > > > > New: > > #!/bin/bash > > set -e > export AMPLAB_JENKINS=1 > export PATH="$PATH:/home/anaconda/envs/py3k/bin" > git clean -fdx > > # Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT > incremental compiler seems to > # ignore our JAVA_HOME and use the system javac instead. > export PATH="$JAVA_HOME/bin:$PATH" > > # Add a pre-downloaded version of Maven to the path so that we avoid > the flaky download step. > export > PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH" > > # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention > export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER" > mkdir -p "$HOME" > export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2" > export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2" > > # This is required for tests of backport patches. > # We need to download the run-tests-codes.sh file because it's > imported by run-tests-jenkins. > # When running tests on branch-1.0 (and earlier), the older version of > run-tests won't set CURRENT_BLOCK, so > # the Jenkins scripts will report all failures as "some tests failed" > rather than a more specific > # error message. > if [ ! -f "dev/run-tests-jenkins" ]; then > wget > https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-jenkins > wget > https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-codes.sh > mv run-tests-jenkins dev/ > mv run-tests-codes.sh dev/ > chmod 755 dev/run-tests-jenkins > chmod 755 dev/run-tests-codes.sh > fi > > ./dev/run-tests-jenkins > > > On Wed, Dec 4, 2019 at 5:53 PM Shane Knapp <skn...@berkeley.edu> wrote: > > > > ++yin huai for more insight in to the NewSparkPullRequestBuilder job... > > > > tbh, i never (or still) really understand the exact use for that job, > > except that it's triggered by https://spark-prs.appspot.com/ > > > > shane > > > > > > On Wed, Dec 4, 2019 at 3:34 PM Sean Owen <sro...@gmail.com> wrote: > > > > > > BTW does anyone know why there are two PR builder jobs? I'm confused > > > about why different ones would execute. > > > > > > Yes I see NewSparkPullRequestBuilder failing on a variety of PRs. > > > I don't think it has anything to do with Hive; these PRs touch > > > different parts of code but all not related to this failure. > > > > > > On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <dongjoon.h...@gmail.com> > > > wrote: > > > > > > > > Hi, Sean. > > > > > > > > It seems that there is no failure on your other SQL PR. > > > > > > > > https://github.com/apache/spark/pull/26748 > > > > > > > > Does the sequential failure happen only at `NewSparkPullRequestBuilder`? > > > > Since `NewSparkPullRequestBuilder` is not the same with > > > > `SparkPullRequestBuilder`, > > > > there might be a root cause inside it if it happens only at > > > > `NewSparkPullRequestBuilder`. > > > > > > > > For `org.apache.hive.service.ServiceException: Failed to Start > > > > HiveServer2`, > > > > I've observed them before, but the root cause might be different from > > > > this one. > > > > > > > > BTW, to reduce the scope of investigation, could you try with > > > > `[hive-1.2]` tag in your PR? > > > > > > > > Bests, > > > > Dongjoon. > > > > > > > > > > > > On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <sro...@gmail.com> wrote: > > > >> > > > >> I'm seeing consistent failures in the PR builder when touching SQL > > > >> code: > > > >> > > > >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/ > > > >> > > > >> > > > >> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's > > > >> own GetSchemasOperation(SparkGetSchemasOperation)14 ms2 > > > >> > > > >> org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It > > > >> is not a test it is a sbt.testing.SuiteSelector) > > > >> > > > >> Looks like this has failed about 6 builds in the past few days. Has > > > >> anyone seen this / has a clue what's causing it? errors are like ... > > > >> > > > >> java.sql.SQLException: No suitable driver found for > > > >> jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue > > > >> > > > >> > > > >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class > > > >> org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not > > > >> org.apache.hadoop.hive.metastore.MetaStoreFilterHook > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
