It's concerning to me that 1) the Go dependency resolution via gogradle is flaky, and 2) that it can block other languages.
I suppose 2) makes sense since it's part of the container bootstrapping code, but that makes 1) a serious problem, of which I wasn't aware. I should have time to investigate this in the next two weeks. On Mon, 25 Mar 2019 at 18:08, Michael Luckey <adude3...@gmail.com> wrote: > Just for the record, > > using a vm here, because did not yet get all task running on my mac, and > did not want to mess with my setup. > > So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores > and further > > sudo apt update > > sudo apt install gcc > > sudo apt install make > > sudo apt install perl > > sudo apt install curl > > sudo apt install openjdk-8-jdk > > sudo apt install python > > sudo apt install -y software-properties-common > > sudo add-apt-repository ppa:deadsnakes/ppa > > sudo apt update > > sudo apt install python3.5 > > sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent > software-properties-common > > curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key > add - > > sudo apt-key fingerprint 0EBFCD88 > > sudo add-apt-repository "deb [arch=amd64] > https://download.docker.com/linux/ubuntu \ > > $(lsb_release -cs) \ > > stable" > > sudo apt-get update > > sudo apt-get install docker-ce docker-ce-cli containerd.io > > sudo groupadd docker > > sudo usermod -aG docker $USER > > git config --global user.email "d...@spam.me" > > git config --global user.name "Some Guy" > > curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py > > sudo python get-pip.py > > rm get-pip.py > > sudo pip install --upgrade virtualenv > > sudo pip install cython > > sudo apt-get install python-dev > > sudo apt-get install python3-distutils > > sudo apt-get install python3-dev # for python3.x installs > > > git clone https://github.com/apache/beam.git cd beam/ ./gradlew build > > Nothing else changed/added. (hopefully, need to reassure myself here) > > Unfortunately, this is failing. Need to exclude those python tests (and of > course website, which usually fails on lira links) > > So I might be missing some env settings for gap, dunno. Probably missed > some docs. > > > > On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey <adude3...@gmail.com> > wrote: > >> Thanks Udi for trying that! >> >> In fact, the go dependency resolution is flaky. Did not look into that, >> but just rerunning usually works. Of course, less than optimal, but, >> well... >> >> Running build target is of course just an aggregation of task to run. And >> unfortunately just running that >> >> ./gradlew :beam-sdks-python:testPy2Gcp >> >> stalls on my (virtual) machine. >> >> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri <eh...@google.com> wrote: >> >>> Okay, `./gradlew build` failed pretty quickly for me: >>> >>> > Task :beam-sdks-go:resolveBuildDependencies FAILED >>> cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', >>> urls=[https://code.googlesource.com/gocloud] does not exist in >>> /usr/local/google/home/ehudm/.gradle/go/repo/ >>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be >>> performed. >>> >>> https://gradle.com/s/x5zqbc5zwd3bg >>> >>> (Now I remember why I stopped using `build` :/) >>> >>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri <eh...@google.com> wrote: >>> >>>> It shouldn't stall. That's a bug. >>>> OTOH, I never use the `build` target. >>>> I'll try running that myself. >>>> >>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey <adude3...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> trying to run './gradlew build' on vanilla setup, my build >>>>> consistently stalls during execution of python gcp tests, e.g. on both of >>>>> - > :beam-sdks-python:testPy2Gcp >>>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp >>>>> >>>>> Console output: >>>>> #### snip #### >>>>> test_big_query_standard_sql >>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>> ... SKIP: IT is skipped because --test-pipeline-options is not specified >>>>> test_big_query_standard_sql_kms_key >>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>> ... SKIP: This test requires BQ Dataflow native source support for KMS, >>>>> which is not available yet. >>>>> test_multiple_destinations_transform >>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... >>>>> SKIP: >>>>> IT is skipped because --test-pipeline-options is not specified >>>>> test_one_job_fails_all_jobs_fail >>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... >>>>> SKIP: >>>>> IT is skipped because --test-pipeline-options is not specified >>>>> test_records_traverse_transform_with_mocks >>>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ... >>>>> >>>>> output ends here, would expect a failed or ok here. >>>>> >>>>> >>>>> Afterwards no progress - even waiting for hours. Any idea, what might >>>>> be causing this? Do I need to add some GCP properties for this task ? >>>>> >>>>> Any ideas, what I am doing wrong? >>>>> >>>>> best, >>>>> >>>>> michel >>>>> >>>>>
