Reading the error from the gradle scan, it largely looks like some part of the GCP dependencies for the build depends on a package, where the commit version is no longer around. The main issue with gogradle is that it's entirely distinct from the usual Go workflow, which means deps users use are likely to be different to what's in the lock file.
This work will be tracked in https://issues.apache.org/jira/browse/BEAM-5379 GoGradle hasn't moved to support the new-go way of handling deps, so my inclination is to simplify to simple scripts for Gradle that shell out the to Go tool for handling Go dep management, over trying to fix GoGradle. On Tue, 26 Mar 2019 at 09:43, Udi Meiri <eh...@google.com> wrote: > Robert, from what I recall it's not flaky for me - it consistently fails. > Let me know if there's a way to get more logging about this error. > > On Mon, Mar 25, 2019, 19:50 Robert Burke <rob...@frantil.com> wrote: > >> It's concerning to me that 1) the Go dependency resolution via gogradle >> is flaky, and 2) that it can block other languages. >> >> I suppose 2) makes sense since it's part of the container bootstrapping >> code, but that makes 1) a serious problem, of which I wasn't aware. >> I should have time to investigate this in the next two weeks. >> >> On Mon, 25 Mar 2019 at 18:08, Michael Luckey <adude3...@gmail.com> wrote: >> >>> Just for the record, >>> >>> using a vm here, because did not yet get all task running on my mac, and >>> did not want to mess with my setup. >>> >>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores >>> and further >>> >>> sudo apt update >>> >>> sudo apt install gcc >>> >>> sudo apt install make >>> >>> sudo apt install perl >>> >>> sudo apt install curl >>> >>> sudo apt install openjdk-8-jdk >>> >>> sudo apt install python >>> >>> sudo apt install -y software-properties-common >>> >>> sudo add-apt-repository ppa:deadsnakes/ppa >>> >>> sudo apt update >>> >>> sudo apt install python3.5 >>> >>> sudo apt-get install apt-transport-https ca-certificates curl >>> gnupg-agent software-properties-common >>> >>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key >>> add - >>> >>> sudo apt-key fingerprint 0EBFCD88 >>> >>> sudo add-apt-repository "deb [arch=amd64] >>> https://download.docker.com/linux/ubuntu \ >>> >>> $(lsb_release -cs) \ >>> >>> stable" >>> >>> sudo apt-get update >>> >>> sudo apt-get install docker-ce docker-ce-cli containerd.io >>> >>> sudo groupadd docker >>> >>> sudo usermod -aG docker $USER >>> >>> git config --global user.email "d...@spam.me" >>> >>> git config --global user.name "Some Guy" >>> >>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >>> >>> sudo python get-pip.py >>> >>> rm get-pip.py >>> >>> sudo pip install --upgrade virtualenv >>> >>> sudo pip install cython >>> >>> sudo apt-get install python-dev >>> >>> sudo apt-get install python3-distutils >>> >>> sudo apt-get install python3-dev # for python3.x installs >>> >>> >>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew build >>> >>> Nothing else changed/added. (hopefully, need to reassure myself here) >>> >>> Unfortunately, this is failing. Need to exclude those python tests (and >>> of course website, which usually fails on lira links) >>> >>> So I might be missing some env settings for gap, dunno. Probably missed >>> some docs. >>> >>> >>> >>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey <adude3...@gmail.com> >>> wrote: >>> >>>> Thanks Udi for trying that! >>>> >>>> In fact, the go dependency resolution is flaky. Did not look into that, >>>> but just rerunning usually works. Of course, less than optimal, but, >>>> well... >>>> >>>> Running build target is of course just an aggregation of task to run. >>>> And unfortunately just running that >>>> >>>> ./gradlew :beam-sdks-python:testPy2Gcp >>>> >>>> stalls on my (virtual) machine. >>>> >>>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri <eh...@google.com> wrote: >>>> >>>>> Okay, `./gradlew build` failed pretty quickly for me: >>>>> >>>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED >>>>> cloud.google.com/go: >>>>> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[ >>>>> https://code.googlesource.com/gocloud] does not exist in >>>>> /usr/local/google/home/ehudm/.gradle/go/repo/ >>>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will >>>>> be performed. >>>>> >>>>> https://gradle.com/s/x5zqbc5zwd3bg >>>>> >>>>> (Now I remember why I stopped using `build` :/) >>>>> >>>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri <eh...@google.com> wrote: >>>>> >>>>>> It shouldn't stall. That's a bug. >>>>>> OTOH, I never use the `build` target. >>>>>> I'll try running that myself. >>>>>> >>>>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey <adude3...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> trying to run './gradlew build' on vanilla setup, my build >>>>>>> consistently stalls during execution of python gcp tests, e.g. on both >>>>>>> of >>>>>>> - > :beam-sdks-python:testPy2Gcp >>>>>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp >>>>>>> >>>>>>> Console output: >>>>>>> #### snip #### >>>>>>> test_big_query_standard_sql >>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>> ... SKIP: IT is skipped because --test-pipeline-options is not specified >>>>>>> test_big_query_standard_sql_kms_key >>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>> ... SKIP: This test requires BQ Dataflow native source support for KMS, >>>>>>> which is not available yet. >>>>>>> test_multiple_destinations_transform >>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... >>>>>>> SKIP: >>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>> test_one_job_fails_all_jobs_fail >>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... >>>>>>> SKIP: >>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>> test_records_traverse_transform_with_mocks >>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ... >>>>>>> >>>>>>> output ends here, would expect a failed or ok here. >>>>>>> >>>>>>> >>>>>>> Afterwards no progress - even waiting for hours. Any idea, what >>>>>>> might be causing this? Do I need to add some GCP properties for this >>>>>>> task ? >>>>>>> >>>>>>> Any ideas, what I am doing wrong? >>>>>>> >>>>>>> best, >>>>>>> >>>>>>> michel >>>>>>> >>>>>>>
