@Udi Did you try to just delete the '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?
@Robert As said before, I am a bit scared about the implications. Shelling out is done by python, and from build perspective, this does not work very well, unfortunately. I.e. no caching, up-to-date checks etc... But of course, we need to play with this a bit more. On Tue, Mar 26, 2019 at 6:24 PM Robert Burke <rob...@frantil.com> wrote: > Reading the error from the gradle scan, it largely looks like some part of > the GCP dependencies for the build depends on a package, where the commit > version is no longer around. The main issue with gogradle is that it's > entirely distinct from the usual Go workflow, which means deps users use > are likely to be different to what's in the lock file. > > This work will be tracked in > https://issues.apache.org/jira/browse/BEAM-5379 > GoGradle hasn't moved to support the new-go way of handling deps, so my > inclination is to simplify to simple scripts for Gradle that shell out the > to Go tool for handling Go dep management, over trying to fix GoGradle. > > On Tue, 26 Mar 2019 at 09:43, Udi Meiri <eh...@google.com> wrote: > >> Robert, from what I recall it's not flaky for me - it consistently fails. >> Let me know if there's a way to get more logging about this error. >> >> On Mon, Mar 25, 2019, 19:50 Robert Burke <rob...@frantil.com> wrote: >> >>> It's concerning to me that 1) the Go dependency resolution via gogradle >>> is flaky, and 2) that it can block other languages. >>> >>> I suppose 2) makes sense since it's part of the container bootstrapping >>> code, but that makes 1) a serious problem, of which I wasn't aware. >>> I should have time to investigate this in the next two weeks. >>> >>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey <adude3...@gmail.com> >>> wrote: >>> >>>> Just for the record, >>>> >>>> using a vm here, because did not yet get all task running on my mac, >>>> and did not want to mess with my setup. >>>> >>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores >>>> and further >>>> >>>> sudo apt update >>>> >>>> sudo apt install gcc >>>> >>>> sudo apt install make >>>> >>>> sudo apt install perl >>>> >>>> sudo apt install curl >>>> >>>> sudo apt install openjdk-8-jdk >>>> >>>> sudo apt install python >>>> >>>> sudo apt install -y software-properties-common >>>> >>>> sudo add-apt-repository ppa:deadsnakes/ppa >>>> >>>> sudo apt update >>>> >>>> sudo apt install python3.5 >>>> >>>> sudo apt-get install apt-transport-https ca-certificates curl >>>> gnupg-agent software-properties-common >>>> >>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key >>>> add - >>>> >>>> sudo apt-key fingerprint 0EBFCD88 >>>> >>>> sudo add-apt-repository "deb [arch=amd64] >>>> https://download.docker.com/linux/ubuntu \ >>>> >>>> $(lsb_release -cs) \ >>>> >>>> stable" >>>> >>>> sudo apt-get update >>>> >>>> sudo apt-get install docker-ce docker-ce-cli containerd.io >>>> >>>> sudo groupadd docker >>>> >>>> sudo usermod -aG docker $USER >>>> >>>> git config --global user.email "d...@spam.me" >>>> >>>> git config --global user.name "Some Guy" >>>> >>>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >>>> >>>> sudo python get-pip.py >>>> >>>> rm get-pip.py >>>> >>>> sudo pip install --upgrade virtualenv >>>> >>>> sudo pip install cython >>>> >>>> sudo apt-get install python-dev >>>> >>>> sudo apt-get install python3-distutils >>>> >>>> sudo apt-get install python3-dev # for python3.x installs >>>> >>>> >>>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew build >>>> >>>> Nothing else changed/added. (hopefully, need to reassure myself here) >>>> >>>> Unfortunately, this is failing. Need to exclude those python tests (and >>>> of course website, which usually fails on lira links) >>>> >>>> So I might be missing some env settings for gap, dunno. Probably missed >>>> some docs. >>>> >>>> >>>> >>>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey <adude3...@gmail.com> >>>> wrote: >>>> >>>>> Thanks Udi for trying that! >>>>> >>>>> In fact, the go dependency resolution is flaky. Did not look into >>>>> that, but just rerunning usually works. Of course, less than optimal, but, >>>>> well... >>>>> >>>>> Running build target is of course just an aggregation of task to run. >>>>> And unfortunately just running that >>>>> >>>>> ./gradlew :beam-sdks-python:testPy2Gcp >>>>> >>>>> stalls on my (virtual) machine. >>>>> >>>>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri <eh...@google.com> wrote: >>>>> >>>>>> Okay, `./gradlew build` failed pretty quickly for me: >>>>>> >>>>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED >>>>>> cloud.google.com/go: >>>>>> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[ >>>>>> https://code.googlesource.com/gocloud] does not exist in >>>>>> /usr/local/google/home/ehudm/.gradle/go/repo/ >>>>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will >>>>>> be performed. >>>>>> >>>>>> https://gradle.com/s/x5zqbc5zwd3bg >>>>>> >>>>>> (Now I remember why I stopped using `build` :/) >>>>>> >>>>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri <eh...@google.com> wrote: >>>>>> >>>>>>> It shouldn't stall. That's a bug. >>>>>>> OTOH, I never use the `build` target. >>>>>>> I'll try running that myself. >>>>>>> >>>>>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey <adude3...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> trying to run './gradlew build' on vanilla setup, my build >>>>>>>> consistently stalls during execution of python gcp tests, e.g. on both >>>>>>>> of >>>>>>>> - > :beam-sdks-python:testPy2Gcp >>>>>>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp >>>>>>>> >>>>>>>> Console output: >>>>>>>> #### snip #### >>>>>>>> test_big_query_standard_sql >>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>> ... SKIP: IT is skipped because --test-pipeline-options is not >>>>>>>> specified >>>>>>>> test_big_query_standard_sql_kms_key >>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>> ... SKIP: This test requires BQ Dataflow native source support for KMS, >>>>>>>> which is not available yet. >>>>>>>> test_multiple_destinations_transform >>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... >>>>>>>> SKIP: >>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>> test_one_job_fails_all_jobs_fail >>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... >>>>>>>> SKIP: >>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>> test_records_traverse_transform_with_mocks >>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ... >>>>>>>> >>>>>>>> output ends here, would expect a failed or ok here. >>>>>>>> >>>>>>>> >>>>>>>> Afterwards no progress - even waiting for hours. Any idea, what >>>>>>>> might be causing this? Do I need to add some GCP properties for this >>>>>>>> task ? >>>>>>>> >>>>>>>> Any ideas, what I am doing wrong? >>>>>>>> >>>>>>>> best, >>>>>>>> >>>>>>>> michel >>>>>>>> >>>>>>>>
