On Tue, Mar 26, 2019 at 10:29 PM Udi Meiri <eh...@google.com> wrote: > Luckey, I couldn't recreate your issue, but I still haven't done a full > build. > I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a image > (n1-standard-4 machine type). > > Ran the following: > sudo apt-get update > sudo apt-get install python-pip > sudo apt-get install python-virtualenv > git clone https://github.com/apache/beam.git > cd beam > ./gradlew :beam-sdks-python:testPy2Gcp > [failed: no JAVA_HOME] > sudo apt-get install openjdk-8-jdk > ./gradlew :beam-sdks-python:testPy2Gcp > > Got: BUILD SUCCESSFUL in 7m 52s >
Nice. Thanks a lot for your help here. If I understand correctly, this VM is already located within gcp. Could it already have some setup, which needs to be done on 'my' VM? For instance I was contemplating about that test trying 'to call home', but as I am (unfortunately ;) no googler and do not have any gcp specific setup, fails here but misses to timeout? This is just some weird assumption, did not yet look into the actual implementation. Which I seemingly need to do here :( > Then I tried: > ./gradlew build > > And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot disk > is 10G total) > Ouch :D > > On Tue, Mar 26, 2019 at 1:35 PM Robert Burke <rob...@frantil.com> wrote: > >> Michael, your concern is reasonable, especially with the experience with >> python, though that does help me bootstrap this work. :) >> >> The go tools provide caching and avoid redoing work if the source files >> haven't changed. This applies most particularly for `go build` and `go >> test`. As long as the go code isn't changing at every invocation, this >> should be fine. I'm not aware of the same being the case for the usual >> python tools. >> >> The real trick is ensuring a valid and consistent environment for the go >> code. >> >> The environment question becomes easier for everyone by moving to go >> modules, which were designed to provide these kinds of consistent builds. >> It also avoids needing a GOPATH set. Any directory is permitted, as long as >> the go.mod is present. >> >> (The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet in >> the repo.) >> >> The main blocker is see is updating the Jenkins machines to have the >> latest version of Go (1.12) instead of 1.10, which doesn't support modules. >> This only blocks a final submission, rather than the work fortunately. >> >> On Tue, Mar 26, 2019, 1:08 PM Udi Meiri <eh...@google.com> wrote: >> >>> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one >>> package with issues). >>> My ~/.bashrc has >>> export GOPATH=$HOME/go >>> so maybe that's making the difference in my setup. >>> >>> On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise <t...@apache.org> wrote: >>> >>>> Can this be addressed by having "clean" remove all state that gogradle >>>> leaves behind? This staleness issue has bitten me a few times also and it >>>> would be good to have a reliable way to deal with it, even if it involves >>>> an extra clean. >>>> >>>> >>>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey <adude3...@gmail.com> >>>> wrote: >>>> >>>>> @Udi >>>>> Did you try to just delete the >>>>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' >>>>> folder? >>>>> >>>>> @Robert >>>>> As said before, I am a bit scared about the implications. Shelling out >>>>> is done by python, and from build perspective, this does not work very >>>>> well, unfortunately. I.e. no caching, up-to-date checks etc... >>>>> >>>>> But of course, we need to play with this a bit more. >>>>> >>>>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke <rob...@frantil.com> >>>>> wrote: >>>>> >>>>>> Reading the error from the gradle scan, it largely looks like some >>>>>> part of the GCP dependencies for the build depends on a package, where >>>>>> the >>>>>> commit version is no longer around. The main issue with gogradle is that >>>>>> it's entirely distinct from the usual Go workflow, which means deps users >>>>>> use are likely to be different to what's in the lock file. >>>>>> >>>>>> This work will be tracked in >>>>>> https://issues.apache.org/jira/browse/BEAM-5379 >>>>>> GoGradle hasn't moved to support the new-go way of handling deps, so >>>>>> my inclination is to simplify to simple scripts for Gradle that shell out >>>>>> the to Go tool for handling Go dep management, over trying to fix >>>>>> GoGradle. >>>>>> >>>>>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri <eh...@google.com> wrote: >>>>>> >>>>>>> Robert, from what I recall it's not flaky for me - it consistently >>>>>>> fails. Let me know if there's a way to get more logging about this >>>>>>> error. >>>>>>> >>>>>>> On Mon, Mar 25, 2019, 19:50 Robert Burke <rob...@frantil.com> wrote: >>>>>>> >>>>>>>> It's concerning to me that 1) the Go dependency resolution via >>>>>>>> gogradle is flaky, and 2) that it can block other languages. >>>>>>>> >>>>>>>> I suppose 2) makes sense since it's part of the container >>>>>>>> bootstrapping code, but that makes 1) a serious problem, of which I >>>>>>>> wasn't >>>>>>>> aware. >>>>>>>> I should have time to investigate this in the next two weeks. >>>>>>>> >>>>>>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey <adude3...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Just for the record, >>>>>>>>> >>>>>>>>> using a vm here, because did not yet get all task running on my >>>>>>>>> mac, and did not want to mess with my setup. >>>>>>>>> >>>>>>>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 >>>>>>>>> cores and further >>>>>>>>> >>>>>>>>> sudo apt update >>>>>>>>> >>>>>>>>> sudo apt install gcc >>>>>>>>> >>>>>>>>> sudo apt install make >>>>>>>>> >>>>>>>>> sudo apt install perl >>>>>>>>> >>>>>>>>> sudo apt install curl >>>>>>>>> >>>>>>>>> sudo apt install openjdk-8-jdk >>>>>>>>> >>>>>>>>> sudo apt install python >>>>>>>>> >>>>>>>>> sudo apt install -y software-properties-common >>>>>>>>> >>>>>>>>> sudo add-apt-repository ppa:deadsnakes/ppa >>>>>>>>> >>>>>>>>> sudo apt update >>>>>>>>> >>>>>>>>> sudo apt install python3.5 >>>>>>>>> >>>>>>>>> sudo apt-get install apt-transport-https ca-certificates curl >>>>>>>>> gnupg-agent software-properties-common >>>>>>>>> >>>>>>>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo >>>>>>>>> apt-key add - >>>>>>>>> >>>>>>>>> sudo apt-key fingerprint 0EBFCD88 >>>>>>>>> >>>>>>>>> sudo add-apt-repository "deb [arch=amd64] >>>>>>>>> https://download.docker.com/linux/ubuntu \ >>>>>>>>> >>>>>>>>> $(lsb_release -cs) \ >>>>>>>>> >>>>>>>>> stable" >>>>>>>>> >>>>>>>>> sudo apt-get update >>>>>>>>> >>>>>>>>> sudo apt-get install docker-ce docker-ce-cli containerd.io >>>>>>>>> >>>>>>>>> sudo groupadd docker >>>>>>>>> >>>>>>>>> sudo usermod -aG docker $USER >>>>>>>>> >>>>>>>>> git config --global user.email "d...@spam.me" >>>>>>>>> >>>>>>>>> git config --global user.name "Some Guy" >>>>>>>>> >>>>>>>>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >>>>>>>>> >>>>>>>>> sudo python get-pip.py >>>>>>>>> >>>>>>>>> rm get-pip.py >>>>>>>>> >>>>>>>>> sudo pip install --upgrade virtualenv >>>>>>>>> >>>>>>>>> sudo pip install cython >>>>>>>>> >>>>>>>>> sudo apt-get install python-dev >>>>>>>>> >>>>>>>>> sudo apt-get install python3-distutils >>>>>>>>> >>>>>>>>> sudo apt-get install python3-dev # for python3.x installs >>>>>>>>> >>>>>>>>> >>>>>>>>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew >>>>>>>>> build >>>>>>>>> >>>>>>>>> Nothing else changed/added. (hopefully, need to reassure myself >>>>>>>>> here) >>>>>>>>> >>>>>>>>> Unfortunately, this is failing. Need to exclude those python tests >>>>>>>>> (and of course website, which usually fails on lira links) >>>>>>>>> >>>>>>>>> So I might be missing some env settings for gap, dunno. Probably >>>>>>>>> missed some docs. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey < >>>>>>>>> adude3...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thanks Udi for trying that! >>>>>>>>>> >>>>>>>>>> In fact, the go dependency resolution is flaky. Did not look into >>>>>>>>>> that, but just rerunning usually works. Of course, less than >>>>>>>>>> optimal, but, >>>>>>>>>> well... >>>>>>>>>> >>>>>>>>>> Running build target is of course just an aggregation of task to >>>>>>>>>> run. And unfortunately just running that >>>>>>>>>> >>>>>>>>>> ./gradlew :beam-sdks-python:testPy2Gcp >>>>>>>>>> >>>>>>>>>> stalls on my (virtual) machine. >>>>>>>>>> >>>>>>>>>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri <eh...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Okay, `./gradlew build` failed pretty quickly for me: >>>>>>>>>>> >>>>>>>>>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED >>>>>>>>>>> cloud.google.com/go: >>>>>>>>>>> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[ >>>>>>>>>>> https://code.googlesource.com/gocloud] does not exist in >>>>>>>>>>> /usr/local/google/home/ehudm/.gradle/go/repo/ >>>>>>>>>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating >>>>>>>>>>> will be performed. >>>>>>>>>>> >>>>>>>>>>> https://gradle.com/s/x5zqbc5zwd3bg >>>>>>>>>>> >>>>>>>>>>> (Now I remember why I stopped using `build` :/) >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri <eh...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> It shouldn't stall. That's a bug. >>>>>>>>>>>> OTOH, I never use the `build` target. >>>>>>>>>>>> I'll try running that myself. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey <adude3...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> trying to run './gradlew build' on vanilla setup, my build >>>>>>>>>>>>> consistently stalls during execution of python gcp tests, e.g. on >>>>>>>>>>>>> both of >>>>>>>>>>>>> - > :beam-sdks-python:testPy2Gcp >>>>>>>>>>>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp >>>>>>>>>>>>> >>>>>>>>>>>>> Console output: >>>>>>>>>>>>> #### snip #### >>>>>>>>>>>>> test_big_query_standard_sql >>>>>>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>>>>>> ... SKIP: IT is skipped because --test-pipeline-options is not >>>>>>>>>>>>> specified >>>>>>>>>>>>> test_big_query_standard_sql_kms_key >>>>>>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>>>>>> ... SKIP: This test requires BQ Dataflow native source support >>>>>>>>>>>>> for KMS, >>>>>>>>>>>>> which is not available yet. >>>>>>>>>>>>> test_multiple_destinations_transform >>>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) >>>>>>>>>>>>> ... SKIP: >>>>>>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>>>>>> test_one_job_fails_all_jobs_fail >>>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) >>>>>>>>>>>>> ... SKIP: >>>>>>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>>>>>> test_records_traverse_transform_with_mocks >>>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) >>>>>>>>>>>>> ... >>>>>>>>>>>>> >>>>>>>>>>>>> output ends here, would expect a failed or ok here. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Afterwards no progress - even waiting for hours. Any idea, >>>>>>>>>>>>> what might be causing this? Do I need to add some GCP properties >>>>>>>>>>>>> for this >>>>>>>>>>>>> task ? >>>>>>>>>>>>> >>>>>>>>>>>>> Any ideas, what I am doing wrong? >>>>>>>>>>>>> >>>>>>>>>>>>> best, >>>>>>>>>>>>> >>>>>>>>>>>>> michel >>>>>>>>>>>>> >>>>>>>>>>>>>
