Can this be addressed by having "clean" remove all state that gogradle leaves behind? This staleness issue has bitten me a few times also and it would be good to have a reliable way to deal with it, even if it involves an extra clean.
On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey <adude3...@gmail.com> wrote: > @Udi > Did you try to just delete the > '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder? > > @Robert > As said before, I am a bit scared about the implications. Shelling out is > done by python, and from build perspective, this does not work very well, > unfortunately. I.e. no caching, up-to-date checks etc... > > But of course, we need to play with this a bit more. > > On Tue, Mar 26, 2019 at 6:24 PM Robert Burke <rob...@frantil.com> wrote: > >> Reading the error from the gradle scan, it largely looks like some part >> of the GCP dependencies for the build depends on a package, where the >> commit version is no longer around. The main issue with gogradle is that >> it's entirely distinct from the usual Go workflow, which means deps users >> use are likely to be different to what's in the lock file. >> >> This work will be tracked in >> https://issues.apache.org/jira/browse/BEAM-5379 >> GoGradle hasn't moved to support the new-go way of handling deps, so my >> inclination is to simplify to simple scripts for Gradle that shell out the >> to Go tool for handling Go dep management, over trying to fix GoGradle. >> >> On Tue, 26 Mar 2019 at 09:43, Udi Meiri <eh...@google.com> wrote: >> >>> Robert, from what I recall it's not flaky for me - it consistently >>> fails. Let me know if there's a way to get more logging about this error. >>> >>> On Mon, Mar 25, 2019, 19:50 Robert Burke <rob...@frantil.com> wrote: >>> >>>> It's concerning to me that 1) the Go dependency resolution via gogradle >>>> is flaky, and 2) that it can block other languages. >>>> >>>> I suppose 2) makes sense since it's part of the container bootstrapping >>>> code, but that makes 1) a serious problem, of which I wasn't aware. >>>> I should have time to investigate this in the next two weeks. >>>> >>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey <adude3...@gmail.com> >>>> wrote: >>>> >>>>> Just for the record, >>>>> >>>>> using a vm here, because did not yet get all task running on my mac, >>>>> and did not want to mess with my setup. >>>>> >>>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 >>>>> cores and further >>>>> >>>>> sudo apt update >>>>> >>>>> sudo apt install gcc >>>>> >>>>> sudo apt install make >>>>> >>>>> sudo apt install perl >>>>> >>>>> sudo apt install curl >>>>> >>>>> sudo apt install openjdk-8-jdk >>>>> >>>>> sudo apt install python >>>>> >>>>> sudo apt install -y software-properties-common >>>>> >>>>> sudo add-apt-repository ppa:deadsnakes/ppa >>>>> >>>>> sudo apt update >>>>> >>>>> sudo apt install python3.5 >>>>> >>>>> sudo apt-get install apt-transport-https ca-certificates curl >>>>> gnupg-agent software-properties-common >>>>> >>>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo >>>>> apt-key add - >>>>> >>>>> sudo apt-key fingerprint 0EBFCD88 >>>>> >>>>> sudo add-apt-repository "deb [arch=amd64] >>>>> https://download.docker.com/linux/ubuntu \ >>>>> >>>>> $(lsb_release -cs) \ >>>>> >>>>> stable" >>>>> >>>>> sudo apt-get update >>>>> >>>>> sudo apt-get install docker-ce docker-ce-cli containerd.io >>>>> >>>>> sudo groupadd docker >>>>> >>>>> sudo usermod -aG docker $USER >>>>> >>>>> git config --global user.email "d...@spam.me" >>>>> >>>>> git config --global user.name "Some Guy" >>>>> >>>>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >>>>> >>>>> sudo python get-pip.py >>>>> >>>>> rm get-pip.py >>>>> >>>>> sudo pip install --upgrade virtualenv >>>>> >>>>> sudo pip install cython >>>>> >>>>> sudo apt-get install python-dev >>>>> >>>>> sudo apt-get install python3-distutils >>>>> >>>>> sudo apt-get install python3-dev # for python3.x installs >>>>> >>>>> >>>>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew build >>>>> >>>>> Nothing else changed/added. (hopefully, need to reassure myself here) >>>>> >>>>> Unfortunately, this is failing. Need to exclude those python tests >>>>> (and of course website, which usually fails on lira links) >>>>> >>>>> So I might be missing some env settings for gap, dunno. Probably >>>>> missed some docs. >>>>> >>>>> >>>>> >>>>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey <adude3...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks Udi for trying that! >>>>>> >>>>>> In fact, the go dependency resolution is flaky. Did not look into >>>>>> that, but just rerunning usually works. Of course, less than optimal, >>>>>> but, >>>>>> well... >>>>>> >>>>>> Running build target is of course just an aggregation of task to run. >>>>>> And unfortunately just running that >>>>>> >>>>>> ./gradlew :beam-sdks-python:testPy2Gcp >>>>>> >>>>>> stalls on my (virtual) machine. >>>>>> >>>>>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri <eh...@google.com> wrote: >>>>>> >>>>>>> Okay, `./gradlew build` failed pretty quickly for me: >>>>>>> >>>>>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED >>>>>>> cloud.google.com/go: >>>>>>> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[ >>>>>>> https://code.googlesource.com/gocloud] does not exist in >>>>>>> /usr/local/google/home/ehudm/.gradle/go/repo/ >>>>>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will >>>>>>> be performed. >>>>>>> >>>>>>> https://gradle.com/s/x5zqbc5zwd3bg >>>>>>> >>>>>>> (Now I remember why I stopped using `build` :/) >>>>>>> >>>>>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri <eh...@google.com> wrote: >>>>>>> >>>>>>>> It shouldn't stall. That's a bug. >>>>>>>> OTOH, I never use the `build` target. >>>>>>>> I'll try running that myself. >>>>>>>> >>>>>>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey <adude3...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> trying to run './gradlew build' on vanilla setup, my build >>>>>>>>> consistently stalls during execution of python gcp tests, e.g. on >>>>>>>>> both of >>>>>>>>> - > :beam-sdks-python:testPy2Gcp >>>>>>>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp >>>>>>>>> >>>>>>>>> Console output: >>>>>>>>> #### snip #### >>>>>>>>> test_big_query_standard_sql >>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>> ... SKIP: IT is skipped because --test-pipeline-options is not >>>>>>>>> specified >>>>>>>>> test_big_query_standard_sql_kms_key >>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>> ... SKIP: This test requires BQ Dataflow native source support for >>>>>>>>> KMS, >>>>>>>>> which is not available yet. >>>>>>>>> test_multiple_destinations_transform >>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... >>>>>>>>> SKIP: >>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>> test_one_job_fails_all_jobs_fail >>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... >>>>>>>>> SKIP: >>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>> test_records_traverse_transform_with_mocks >>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) >>>>>>>>> ... >>>>>>>>> >>>>>>>>> output ends here, would expect a failed or ok here. >>>>>>>>> >>>>>>>>> >>>>>>>>> Afterwards no progress - even waiting for hours. Any idea, what >>>>>>>>> might be causing this? Do I need to add some GCP properties for this >>>>>>>>> task ? >>>>>>>>> >>>>>>>>> Any ideas, what I am doing wrong? >>>>>>>>> >>>>>>>>> best, >>>>>>>>> >>>>>>>>> michel >>>>>>>>> >>>>>>>>>
