I believe what happens is that testPy2Gcp actually runs integration tests that try to connect to GCP. Without having GCP cluster and configuration on your machine I'd expect these tests to fail.
I'd say we should remove testPy2Gcp task from "build" task and explicitly keep it as integration test. --Mikhail On Tue, Mar 26, 2019 at 3:12 PM Michael Luckey <adude3...@gmail.com> wrote: > > > On Tue, Mar 26, 2019 at 10:29 PM Udi Meiri <eh...@google.com> wrote: > >> Luckey, I couldn't recreate your issue, but I still haven't done a full >> build. >> I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a image >> (n1-standard-4 machine type). >> >> Ran the following: >> sudo apt-get update >> sudo apt-get install python-pip >> sudo apt-get install python-virtualenv >> git clone https://github.com/apache/beam.git >> cd beam >> ./gradlew :beam-sdks-python:testPy2Gcp >> [failed: no JAVA_HOME] >> sudo apt-get install openjdk-8-jdk >> ./gradlew :beam-sdks-python:testPy2Gcp >> >> Got: BUILD SUCCESSFUL in 7m 52s >> > > Nice. Thanks a lot for your help here. > > If I understand correctly, this VM is already located within gcp. Could it > already have some setup, which needs to be done on 'my' VM? For instance I > was contemplating about that test trying 'to call home', but as I am > (unfortunately ;) no googler and do not have any gcp specific setup, fails > here but misses to timeout? This is just some weird assumption, did not yet > look into the actual implementation. > > Which I seemingly need to do here :( > > >> Then I tried: >> ./gradlew build >> >> And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot disk >> is 10G total) >> > > Ouch :D > > >> >> On Tue, Mar 26, 2019 at 1:35 PM Robert Burke <rob...@frantil.com> wrote: >> >>> Michael, your concern is reasonable, especially with the experience with >>> python, though that does help me bootstrap this work. :) >>> >>> The go tools provide caching and avoid redoing work if the source files >>> haven't changed. This applies most particularly for `go build` and `go >>> test`. As long as the go code isn't changing at every invocation, this >>> should be fine. I'm not aware of the same being the case for the usual >>> python tools. >>> >>> The real trick is ensuring a valid and consistent environment for the >>> go code. >>> >>> The environment question becomes easier for everyone by moving to go >>> modules, which were designed to provide these kinds of consistent builds. >>> It also avoids needing a GOPATH set. Any directory is permitted, as long as >>> the go.mod is present. >>> >>> (The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet >>> in the repo.) >>> >>> The main blocker is see is updating the Jenkins machines to have the >>> latest version of Go (1.12) instead of 1.10, which doesn't support modules. >>> This only blocks a final submission, rather than the work fortunately. >>> >>> On Tue, Mar 26, 2019, 1:08 PM Udi Meiri <eh...@google.com> wrote: >>> >>>> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one >>>> package with issues). >>>> My ~/.bashrc has >>>> export GOPATH=$HOME/go >>>> so maybe that's making the difference in my setup. >>>> >>>> On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise <t...@apache.org> wrote: >>>> >>>>> Can this be addressed by having "clean" remove all state that gogradle >>>>> leaves behind? This staleness issue has bitten me a few times also and it >>>>> would be good to have a reliable way to deal with it, even if it involves >>>>> an extra clean. >>>>> >>>>> >>>>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey <adude3...@gmail.com> >>>>> wrote: >>>>> >>>>>> @Udi >>>>>> Did you try to just delete the >>>>>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' >>>>>> folder? >>>>>> >>>>>> @Robert >>>>>> As said before, I am a bit scared about the implications. Shelling >>>>>> out is done by python, and from build perspective, this does not work >>>>>> very >>>>>> well, unfortunately. I.e. no caching, up-to-date checks etc... >>>>>> >>>>>> But of course, we need to play with this a bit more. >>>>>> >>>>>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke <rob...@frantil.com> >>>>>> wrote: >>>>>> >>>>>>> Reading the error from the gradle scan, it largely looks like some >>>>>>> part of the GCP dependencies for the build depends on a package, where >>>>>>> the >>>>>>> commit version is no longer around. The main issue with gogradle is that >>>>>>> it's entirely distinct from the usual Go workflow, which means deps >>>>>>> users >>>>>>> use are likely to be different to what's in the lock file. >>>>>>> >>>>>>> This work will be tracked in >>>>>>> https://issues.apache.org/jira/browse/BEAM-5379 >>>>>>> GoGradle hasn't moved to support the new-go way of handling deps, so >>>>>>> my inclination is to simplify to simple scripts for Gradle that shell >>>>>>> out >>>>>>> the to Go tool for handling Go dep management, over trying to fix >>>>>>> GoGradle. >>>>>>> >>>>>>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri <eh...@google.com> wrote: >>>>>>> >>>>>>>> Robert, from what I recall it's not flaky for me - it consistently >>>>>>>> fails. Let me know if there's a way to get more logging about this >>>>>>>> error. >>>>>>>> >>>>>>>> On Mon, Mar 25, 2019, 19:50 Robert Burke <rob...@frantil.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> It's concerning to me that 1) the Go dependency resolution via >>>>>>>>> gogradle is flaky, and 2) that it can block other languages. >>>>>>>>> >>>>>>>>> I suppose 2) makes sense since it's part of the container >>>>>>>>> bootstrapping code, but that makes 1) a serious problem, of which I >>>>>>>>> wasn't >>>>>>>>> aware. >>>>>>>>> I should have time to investigate this in the next two weeks. >>>>>>>>> >>>>>>>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey <adude3...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Just for the record, >>>>>>>>>> >>>>>>>>>> using a vm here, because did not yet get all task running on my >>>>>>>>>> mac, and did not want to mess with my setup. >>>>>>>>>> >>>>>>>>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 >>>>>>>>>> cores and further >>>>>>>>>> >>>>>>>>>> sudo apt update >>>>>>>>>> >>>>>>>>>> sudo apt install gcc >>>>>>>>>> >>>>>>>>>> sudo apt install make >>>>>>>>>> >>>>>>>>>> sudo apt install perl >>>>>>>>>> >>>>>>>>>> sudo apt install curl >>>>>>>>>> >>>>>>>>>> sudo apt install openjdk-8-jdk >>>>>>>>>> >>>>>>>>>> sudo apt install python >>>>>>>>>> >>>>>>>>>> sudo apt install -y software-properties-common >>>>>>>>>> >>>>>>>>>> sudo add-apt-repository ppa:deadsnakes/ppa >>>>>>>>>> >>>>>>>>>> sudo apt update >>>>>>>>>> >>>>>>>>>> sudo apt install python3.5 >>>>>>>>>> >>>>>>>>>> sudo apt-get install apt-transport-https ca-certificates curl >>>>>>>>>> gnupg-agent software-properties-common >>>>>>>>>> >>>>>>>>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo >>>>>>>>>> apt-key add - >>>>>>>>>> >>>>>>>>>> sudo apt-key fingerprint 0EBFCD88 >>>>>>>>>> >>>>>>>>>> sudo add-apt-repository "deb [arch=amd64] >>>>>>>>>> https://download.docker.com/linux/ubuntu \ >>>>>>>>>> >>>>>>>>>> $(lsb_release -cs) \ >>>>>>>>>> >>>>>>>>>> stable" >>>>>>>>>> >>>>>>>>>> sudo apt-get update >>>>>>>>>> >>>>>>>>>> sudo apt-get install docker-ce docker-ce-cli containerd.io >>>>>>>>>> >>>>>>>>>> sudo groupadd docker >>>>>>>>>> >>>>>>>>>> sudo usermod -aG docker $USER >>>>>>>>>> >>>>>>>>>> git config --global user.email "d...@spam.me" >>>>>>>>>> >>>>>>>>>> git config --global user.name "Some Guy" >>>>>>>>>> >>>>>>>>>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >>>>>>>>>> >>>>>>>>>> sudo python get-pip.py >>>>>>>>>> >>>>>>>>>> rm get-pip.py >>>>>>>>>> >>>>>>>>>> sudo pip install --upgrade virtualenv >>>>>>>>>> >>>>>>>>>> sudo pip install cython >>>>>>>>>> >>>>>>>>>> sudo apt-get install python-dev >>>>>>>>>> >>>>>>>>>> sudo apt-get install python3-distutils >>>>>>>>>> >>>>>>>>>> sudo apt-get install python3-dev # for python3.x installs >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew >>>>>>>>>> build >>>>>>>>>> >>>>>>>>>> Nothing else changed/added. (hopefully, need to reassure myself >>>>>>>>>> here) >>>>>>>>>> >>>>>>>>>> Unfortunately, this is failing. Need to exclude those python >>>>>>>>>> tests (and of course website, which usually fails on lira links) >>>>>>>>>> >>>>>>>>>> So I might be missing some env settings for gap, dunno. Probably >>>>>>>>>> missed some docs. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey < >>>>>>>>>> adude3...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Udi for trying that! >>>>>>>>>>> >>>>>>>>>>> In fact, the go dependency resolution is flaky. Did not look >>>>>>>>>>> into that, but just rerunning usually works. Of course, less than >>>>>>>>>>> optimal, >>>>>>>>>>> but, well... >>>>>>>>>>> >>>>>>>>>>> Running build target is of course just an aggregation of task to >>>>>>>>>>> run. And unfortunately just running that >>>>>>>>>>> >>>>>>>>>>> ./gradlew :beam-sdks-python:testPy2Gcp >>>>>>>>>>> >>>>>>>>>>> stalls on my (virtual) machine. >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri <eh...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Okay, `./gradlew build` failed pretty quickly for me: >>>>>>>>>>>> >>>>>>>>>>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED >>>>>>>>>>>> cloud.google.com/go: >>>>>>>>>>>> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[ >>>>>>>>>>>> https://code.googlesource.com/gocloud] does not exist in >>>>>>>>>>>> /usr/local/google/home/ehudm/.gradle/go/repo/ >>>>>>>>>>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating >>>>>>>>>>>> will be performed. >>>>>>>>>>>> >>>>>>>>>>>> https://gradle.com/s/x5zqbc5zwd3bg >>>>>>>>>>>> >>>>>>>>>>>> (Now I remember why I stopped using `build` :/) >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri <eh...@google.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> It shouldn't stall. That's a bug. >>>>>>>>>>>>> OTOH, I never use the `build` target. >>>>>>>>>>>>> I'll try running that myself. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey < >>>>>>>>>>>>> adude3...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> trying to run './gradlew build' on vanilla setup, my build >>>>>>>>>>>>>> consistently stalls during execution of python gcp tests, e.g. >>>>>>>>>>>>>> on both of >>>>>>>>>>>>>> - > :beam-sdks-python:testPy2Gcp >>>>>>>>>>>>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp >>>>>>>>>>>>>> >>>>>>>>>>>>>> Console output: >>>>>>>>>>>>>> #### snip #### >>>>>>>>>>>>>> test_big_query_standard_sql >>>>>>>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>>>>>>> ... SKIP: IT is skipped because --test-pipeline-options is not >>>>>>>>>>>>>> specified >>>>>>>>>>>>>> test_big_query_standard_sql_kms_key >>>>>>>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>>>>>>> ... SKIP: This test requires BQ Dataflow native source support >>>>>>>>>>>>>> for KMS, >>>>>>>>>>>>>> which is not available yet. >>>>>>>>>>>>>> test_multiple_destinations_transform >>>>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) >>>>>>>>>>>>>> ... SKIP: >>>>>>>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>>>>>>> test_one_job_fails_all_jobs_fail >>>>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) >>>>>>>>>>>>>> ... SKIP: >>>>>>>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>>>>>>> test_records_traverse_transform_with_mocks >>>>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> >>>>>>>>>>>>>> output ends here, would expect a failed or ok here. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Afterwards no progress - even waiting for hours. Any idea, >>>>>>>>>>>>>> what might be causing this? Do I need to add some GCP properties >>>>>>>>>>>>>> for this >>>>>>>>>>>>>> task ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any ideas, what I am doing wrong? >>>>>>>>>>>>>> >>>>>>>>>>>>>> best, >>>>>>>>>>>>>> >>>>>>>>>>>>>> michel >>>>>>>>>>>>>> >>>>>>>>>>>>>>
