Re: Build blocking on

[Michael Luckey]

Do not worry. It seems next to impossible to stumble upon this issue if
tests developed on 'prepared' machine. This seems to be a side effect of a
used library which seems difficult to expect.

Thanks for looking into that. FWIW I opened an issue and assigned it to you
[1]. Feel free to reach out to me if any information is missing.

best,

michel

[1] https://issues.apache.org/jira/browse/BEAM-6949

On 2019/04/01 00:53:40, Pablo Estrada  wrote:
> Hi Michael,>
> I wrote that test and much of that code. I'm quite sorry about the
trouble.>
> The test should use mocks and not hang when it's missing GCP
dependencies.>
> That sounds like a bug in the test. We can deactivate it while I figure
out>
> what's going wrong..>
> Best>
> -P.>
>
> On Sat, Mar 30, 2019, 2:55 PM Michael Luckey  wrote:>
>
> > After digging a bit deeper, I was able to verify, that those tests
block>
> > on authorization to GCP.>
> >>
> > Seems that, as I do not have any credentials set, and underlying
oauth2>
> > falls back to some local mode. This seems to start a webserver on port
8080>
> > and waiting there forever. Accessing that port forwards to some google,
but>
> > fails also miserably.>
> >>
> > Running>
> >>
> > python setup.py nosetests --tests>
> >>
 
apache_beam.io.gcp.bigquery_file_loads_test:TestBigQueryFileLoads.test_records_traverse_transform_with_mocks>
> >>
> >>
> > and hitting 'Ctrl-C' after it got stuck, results in following output:>
> >>
> > 'KeyboardInterrupt [while running>
> >>
\'WriteToBigQuery/BigQueryBatchFileLoads/RemoveTempTables/Delete\']\n>
> >> Your browser has been opened to visit:>
> >>>
> >>
https://accounts.google.com/o/oauth2/v2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery+https%3A%
>
> >> If your browser is on a different machine then exit and re-run this>
> >> application with the command-line parameter>
> >>   --noauth_local_webserver>
> >> Failed to find "code" in the query parameters of the redirect.>
> >> Invalid authorization: Try running with --noauth_local_webserver.>
> >>
> >>
> > I am a bit lost here on how to proceed.>
> >>
> >>
> > On Tue, Mar 26, 2019 at 11:48 PM Michael Luckey >
> > wrote:>
> >>
> >>>
> >>>
> >> On Tue, Mar 26, 2019 at 11:18 PM Mikhail Gryzykhin >
> >> wrote:>
> >>>
> >>> I believe what happens is that testPy2Gcp actually runs integration>
> >>> tests that try to connect to GCP.>
> 
> >>>
> >> Actually I was hoping for an explanation like this. Any suggestion how
I>
> >> could confirm that on my behalf?>
> >>>
> >>>
> >>> Without having GCP cluster and configuration on your machine I'd
expect>
> >>> these tests to fail.>
> 
> >>>
> >> Hmm... here I am actually unsure, what would be the best to handle
such>
> >> cases.>
> >>>
> >> If I understand correctly, we currently skip some tests which do not
meet>
> >> expectations, kind of 'can not run on your arch' thingies... So I am>
> >> undecided, whether I d prefer those tests to be skipped if gcp>
> >> configuration is missing>
> >>>
> >> pro>
> >> * dev is still able to run the tests (whichever task they are
associated>
> >> with) without having to separate the failures out. For instance,
these>
> >> 'testPy2Gcp' does actually execute 'some tests' - which might be
already>
> >> covered by some other calls... But I definitely do not like the idea,
to>
> >> put the burden on the developer to track which tasks/tests might be>
> >> executed on local machine. Unless this distinction is really coarse -
and>
> >> pre/postcommit is something I really would like to be able to run
locally...>
> >>>
> >>>
> >> con>
> >> * we definitely need to make sure, those tests are not accidentally>
> >> skipped on CI servers.>
> >>>
> >>>
> 
> >>> I'd say we should remove testPy2Gcp task from "build" task and>
> >>> explicitly keep it as integration test.>
> 
> >>> --Mikhail>
> 
> 
> >>> On Tue, Mar 26, 2019 at 3:12 PM Michael Luckey >
> >>> wrote:>
> 
> >
> >
>  On Tue, Mar 26, 2019 at 10:29 PM Udi Meiri 
wrote:>
> >
> > Luckey, I couldn't recreate your issue, but I still haven't done a>
> > full build.>
> > I created a new GCE VM with using the
ubuntu-1804-bionic-v20190212a>
> > image (n1-standard-4 machine type).>
> >>
> > Ran the following:>
> > sudo apt-get update>
> > sudo apt-get install python-pip>
> > sudo apt-get install python-virtualenv>
> > git clone https://github.com/apache/beam.git>
> > cd beam>
> > ./gradlew :beam-sdks-python:testPy2Gcp>
> > [failed: no JAVA_HOME]>
> > sudo apt-get install openjdk-8-jdk>
> > ./gradlew :beam-sdks-python:testPy2Gcp>
> >>
> > Got: BUILD SUCCESSFUL in 7m 52s>
> >>
> >
>  Nice. Thanks a lot for your help here.>
> >
>  If I understand correctly, this VM is already located within gcp.
Could>
>  it already have some setup, which needs to be done on 'my' VM? For
instance>
>  I was contemplating about that test trying 'to call home', but 

Re: Build blocking on

[Pablo Estrada]

Hi Michael,
I wrote that test and much of that code. I'm quite sorry about the trouble.
The test should use mocks and not hang when it's missing GCP dependencies.
That sounds like a bug in the test. We can deactivate it while I figure out
what's going wrong..
Best
-P.

On Sat, Mar 30, 2019, 2:55 PM Michael Luckey  wrote:

> After digging a bit deeper, I was able to verify, that those tests block
> on authorization to GCP.
>
> Seems that, as I do not have any credentials set, and underlying oauth2
> falls back to some local mode. This seems to start a webserver on port 8080
> and waiting there forever. Accessing that port forwards to some google, but
> fails also miserably.
>
> Running
>
> python setup.py nosetests --tests
>>  
>> apache_beam.io.gcp.bigquery_file_loads_test:TestBigQueryFileLoads.test_records_traverse_transform_with_mocks
>
>
> and hitting 'Ctrl-C' after it got stuck, results in following output:
>
> 'KeyboardInterrupt [while running
>> \'WriteToBigQuery/BigQueryBatchFileLoads/RemoveTempTables/Delete\']\n
>> Your browser has been opened to visit:
>>
>> https://accounts.google.com/o/oauth2/v2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery+https%3A%
>> If your browser is on a different machine then exit and re-run this
>> application with the command-line parameter
>>   --noauth_local_webserver
>> Failed to find "code" in the query parameters of the redirect.
>> Invalid authorization: Try running with --noauth_local_webserver.
>
>
> I am a bit lost here on how to proceed.
>
>
> On Tue, Mar 26, 2019 at 11:48 PM Michael Luckey 
> wrote:
>
>>
>>
>> On Tue, Mar 26, 2019 at 11:18 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> I believe what happens is that testPy2Gcp actually runs integration
>>> tests that try to connect to GCP.
>>>
>>
>> Actually I was hoping for an explanation like this. Any suggestion how I
>> could confirm that on my behalf?
>>
>>
>>> Without having GCP cluster and configuration on your machine I'd expect
>>> these tests to fail.
>>>
>>
>> Hmm... here I am actually unsure, what would be the best to handle such
>> cases.
>>
>> If I understand correctly, we currently skip some tests which do not meet
>> expectations, kind of 'can not run on your arch' thingies... So I am
>> undecided, whether I d prefer those tests to be skipped if gcp
>> configuration is missing
>>
>> pro
>> * dev is still able to run the tests (whichever task they are associated
>> with) without having to separate the failures out. For instance, these
>> 'testPy2Gcp' does actually execute 'some tests' - which might be already
>> covered by some other calls... But I definitely do not like the idea, to
>> put the burden on the developer to track which tasks/tests might be
>> executed on local machine. Unless this distinction is really coarse - and
>> pre/postcommit is something I really would like to be able to run locally...
>>
>>
>> con
>> * we definitely need to make sure, those tests are not accidentally
>> skipped on CI servers.
>>
>>
>>>
>>> I'd say we should remove testPy2Gcp task from "build" task and
>>> explicitly keep it as integration test.
>>>
>>> --Mikhail
>>>
>>>
>>> On Tue, Mar 26, 2019 at 3:12 PM Michael Luckey 
>>> wrote:
>>>


 On Tue, Mar 26, 2019 at 10:29 PM Udi Meiri  wrote:

> Luckey, I couldn't recreate your issue, but I still haven't done a
> full build.
> I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a
> image (n1-standard-4 machine type).
>
> Ran the following:
> sudo apt-get update
> sudo apt-get install python-pip
> sudo apt-get install python-virtualenv
> git clone https://github.com/apache/beam.git
> cd beam
> ./gradlew :beam-sdks-python:testPy2Gcp
> [failed: no JAVA_HOME]
> sudo apt-get install openjdk-8-jdk
> ./gradlew :beam-sdks-python:testPy2Gcp
>
> Got: BUILD SUCCESSFUL in 7m 52s
>

 Nice. Thanks a lot for your help here.

 If I understand correctly, this VM is already located within gcp. Could
 it already have some setup, which needs to be done on 'my' VM? For instance
 I was contemplating about that test trying 'to call home', but as I am
 (unfortunately ;) no googler and do not have any gcp specific setup, fails
 here but misses to timeout? This is just some weird assumption, did not yet
 look into the actual implementation.

 Which I seemingly need to do here :(


> Then I tried:
> ./gradlew build
>
> And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot
> disk is 10G total)
>

 Ouch :D


>
> On Tue, Mar 26, 2019 at 1:35 PM Robert Burke 
> wrote:
>
>> Michael, your concern is reasonable, especially with the experience
>> with python, though that does help me bootstrap this work. :)
>>
>> The go tools provide caching and avoid redoing work if the source
>> files haven't changed. This applies most particularly 

Re: Build blocking on

[Michael Luckey]

After digging a bit deeper, I was able to verify, that those tests block on
authorization to GCP.

Seems that, as I do not have any credentials set, and underlying oauth2
falls back to some local mode. This seems to start a webserver on port 8080
and waiting there forever. Accessing that port forwards to some google, but
fails also miserably.

Running

python setup.py nosetests --tests
>  
> apache_beam.io.gcp.bigquery_file_loads_test:TestBigQueryFileLoads.test_records_traverse_transform_with_mocks


and hitting 'Ctrl-C' after it got stuck, results in following output:

'KeyboardInterrupt [while running
> \'WriteToBigQuery/BigQueryBatchFileLoads/RemoveTempTables/Delete\']\n
> Your browser has been opened to visit:
>
> https://accounts.google.com/o/oauth2/v2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery+https%3A%
> If your browser is on a different machine then exit and re-run this
> application with the command-line parameter
>   --noauth_local_webserver
> Failed to find "code" in the query parameters of the redirect.
> Invalid authorization: Try running with --noauth_local_webserver.


I am a bit lost here on how to proceed.


On Tue, Mar 26, 2019 at 11:48 PM Michael Luckey  wrote:

>
>
> On Tue, Mar 26, 2019 at 11:18 PM Mikhail Gryzykhin 
> wrote:
>
>> I believe what happens is that testPy2Gcp actually runs integration tests
>> that try to connect to GCP.
>>
>
> Actually I was hoping for an explanation like this. Any suggestion how I
> could confirm that on my behalf?
>
>
>> Without having GCP cluster and configuration on your machine I'd expect
>> these tests to fail.
>>
>
> Hmm... here I am actually unsure, what would be the best to handle such
> cases.
>
> If I understand correctly, we currently skip some tests which do not meet
> expectations, kind of 'can not run on your arch' thingies... So I am
> undecided, whether I d prefer those tests to be skipped if gcp
> configuration is missing
>
> pro
> * dev is still able to run the tests (whichever task they are associated
> with) without having to separate the failures out. For instance, these
> 'testPy2Gcp' does actually execute 'some tests' - which might be already
> covered by some other calls... But I definitely do not like the idea, to
> put the burden on the developer to track which tasks/tests might be
> executed on local machine. Unless this distinction is really coarse - and
> pre/postcommit is something I really would like to be able to run locally...
>
>
> con
> * we definitely need to make sure, those tests are not accidentally
> skipped on CI servers.
>
>
>>
>> I'd say we should remove testPy2Gcp task from "build" task and explicitly
>> keep it as integration test.
>>
>> --Mikhail
>>
>>
>> On Tue, Mar 26, 2019 at 3:12 PM Michael Luckey 
>> wrote:
>>
>>>
>>>
>>> On Tue, Mar 26, 2019 at 10:29 PM Udi Meiri  wrote:
>>>
 Luckey, I couldn't recreate your issue, but I still haven't done a full
 build.
 I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a
 image (n1-standard-4 machine type).

 Ran the following:
 sudo apt-get update
 sudo apt-get install python-pip
 sudo apt-get install python-virtualenv
 git clone https://github.com/apache/beam.git
 cd beam
 ./gradlew :beam-sdks-python:testPy2Gcp
 [failed: no JAVA_HOME]
 sudo apt-get install openjdk-8-jdk
 ./gradlew :beam-sdks-python:testPy2Gcp

 Got: BUILD SUCCESSFUL in 7m 52s

>>>
>>> Nice. Thanks a lot for your help here.
>>>
>>> If I understand correctly, this VM is already located within gcp. Could
>>> it already have some setup, which needs to be done on 'my' VM? For instance
>>> I was contemplating about that test trying 'to call home', but as I am
>>> (unfortunately ;) no googler and do not have any gcp specific setup, fails
>>> here but misses to timeout? This is just some weird assumption, did not yet
>>> look into the actual implementation.
>>>
>>> Which I seemingly need to do here :(
>>>
>>>
 Then I tried:
 ./gradlew build

 And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot
 disk is 10G total)

>>>
>>> Ouch :D
>>>
>>>

 On Tue, Mar 26, 2019 at 1:35 PM Robert Burke 
 wrote:

> Michael, your concern is reasonable, especially with the experience
> with python, though that does help me bootstrap this work. :)
>
> The go tools provide caching and avoid redoing work if the source
> files haven't changed. This applies most particularly for `go build` and
> `go test`. As long as the go code isn't changing at every invocation, this
> should be fine. I'm not aware of the same being the case for the usual
> python tools.
>
>  The real trick is ensuring a valid and consistent environment for the
> go code.
>
> The environment question becomes easier for everyone by moving to go
> modules, which were designed to provide these kinds of consistent builds.
> It also 

Re: Build blocking on

[Robert Burke]

Again very valid concerns! I wouldn't take that step lightly (eg. testing
every single go using gradle task we have, if not testing that explicit
case).

A happier path would be that gogradle "just works" with go modules, and we
can avoid the whole awkward double lockfile state, and version mismatches
between what GoGradle is using and testing on Jenkins vs the Pipeline
Author experience.

I'm not sure, and I won't be convinced until I have time to try it. I'm
just skeptical they do since the issue to support using them
 has
received no traction in the half a year it's been filed.

Given that GoGradle by it's nature is manually invoking the Go Tool, it's
entirely possible the step is to avoid the gradle lock file, and the built
in vendoring commands, and just switch to Go Modules, and since our usage
is very simple, as the go tool will continue to do it's own state
management. Again, it needs testing and that takes time.


On Tue, 26 Mar 2019 at 15:18, Michael Luckey  wrote:

> Of course, we could implement something here. But I am worried about the
> consequences. As gogradle writes into (user) global state this would have
> unexpected side effects.
>
> Consider a developer running Project A - which happens to also use
> gogradle - and during build of that project issues an innocent beam clean.
> Would be no fun to track all that failures arising out of those now deleted
> state.
>
> Unfortunately, I do not have a clear understanding yet, what's going on
> here, why that cache is put there and how it gets inconsistent. Need to
> look into that. (But might be obsoleted anyway by Roberts plans to drop
> gogradle for something different.)
>
> On Tue, Mar 26, 2019 at 7:28 PM Thomas Weise  wrote:
>
>> Can this be addressed by having "clean" remove all state that gogradle
>> leaves behind? This staleness issue has bitten me a few times also and it
>> would be good to have a reliable way to deal with it, even if it involves
>> an extra clean.
>>
>>
>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
>> wrote:
>>
>>> @Udi
>>> Did you try to just delete the
>>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?
>>>
>>> @Robert
>>> As said before, I am a bit scared about the implications. Shelling out
>>> is done by python, and from build perspective, this does not work very
>>> well, unfortunately. I.e. no caching, up-to-date checks etc...
>>>
>>> But of course, we need to play with this a bit more.
>>>
>>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke  wrote:
>>>
 Reading the error from the gradle scan, it largely looks like some part
 of the GCP dependencies for the build depends on a package, where the
 commit version is no longer around. The main issue with gogradle is that
 it's entirely distinct from the usual Go workflow, which means deps users
 use are likely to be different to what's in the lock file.

 This work will be tracked in
 https://issues.apache.org/jira/browse/BEAM-5379
 GoGradle hasn't moved to support the new-go way of handling deps, so my
 inclination is to simplify to simple scripts for Gradle that shell out the
 to Go tool for handling Go dep management, over trying to fix GoGradle.

 On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:

> Robert, from what I recall it's not flaky for me - it consistently
> fails. Let me know if there's a way to get more logging about this error.
>
> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>
>> It's concerning to me that 1) the Go dependency resolution via
>> gogradle is flaky, and 2) that it can block other languages.
>>
>> I suppose 2) makes sense since it's part of the container
>> bootstrapping code, but that makes 1) a serious problem, of which I 
>> wasn't
>> aware.
>> I should have time to investigate this in the next two weeks.
>>
>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
>> wrote:
>>
>>> Just for the record,
>>>
>>> using a vm here, because did not yet get all task running on my mac,
>>> and did not want to mess with my setup.
>>>
>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6
>>> cores and further
>>>
>>> sudo apt update
>>>
>>> sudo apt install gcc
>>>
>>> sudo apt install make
>>>
>>> sudo apt install perl
>>>
>>> sudo apt install curl
>>>
>>> sudo apt install openjdk-8-jdk
>>>
>>> sudo apt install python
>>>
>>> sudo apt install -y software-properties-common
>>>
>>> sudo add-apt-repository ppa:deadsnakes/ppa
>>>
>>> sudo apt update
>>>
>>> sudo apt install python3.5
>>>
>>> sudo apt-get install apt-transport-https ca-certificates curl
>>> gnupg-agent software-properties-common
>>>
>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo
>>> apt-key 

Re: Build blocking on

[Kenneth Knowles]

Exactly. What you have said is what we should move towards IMO.

On Tue, Mar 26, 2019 at 4:02 PM Michael Luckey  wrote:

>
>
> On Tue, Mar 26, 2019 at 11:40 PM Kenneth Knowles  wrote:
>
>> +1 to separating integration tests from "build". It should be able to
>> succeed without internet access (if deps are cached).
>>
>
> Big +1 here.
>
> I d even suggest the following:
> - ./gradlew build must succeed 'offline' (if deps cached)
> - no flaky tests on build target. Flaky tests should be 'offloaded' to
> preCommit (probably)
>
>   Main point here from my side would be user experience. If someone
> downloads released source code, she must be able to just do a './gradlew
> build publishToMavenLocal' without failing tests. Not fully understood
> python/go in this context, though.
>
> - no docker tests on build target
>
>   imho just to much of required infrastructure for a 'build'
>
>
>
>
>>
>> On Tue, Mar 26, 2019 at 3:18 PM Michael Luckey 
>> wrote:
>>
>>> Of course, we could implement something here. But I am worried about the
>>> consequences. As gogradle writes into (user) global state this would have
>>> unexpected side effects.
>>>
>>> Consider a developer running Project A - which happens to also use
>>> gogradle - and during build of that project issues an innocent beam clean.
>>> Would be no fun to track all that failures arising out of those now deleted
>>> state.
>>>
>>> Unfortunately, I do not have a clear understanding yet, what's going on
>>> here, why that cache is put there and how it gets inconsistent. Need to
>>> look into that. (But might be obsoleted anyway by Roberts plans to drop
>>> gogradle for something different.)
>>>
>>> On Tue, Mar 26, 2019 at 7:28 PM Thomas Weise  wrote:
>>>
 Can this be addressed by having "clean" remove all state that gogradle
 leaves behind? This staleness issue has bitten me a few times also and it
 would be good to have a reliable way to deal with it, even if it involves
 an extra clean.


 On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
 wrote:

> @Udi
> Did you try to just delete the
> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com'
> folder?
>
> @Robert
> As said before, I am a bit scared about the implications. Shelling out
> is done by python, and from build perspective, this does not work very
> well, unfortunately. I.e. no caching, up-to-date checks etc...
>
> But of course, we need to play with this a bit more.
>
> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke 
> wrote:
>
>> Reading the error from the gradle scan, it largely looks like some
>> part of the GCP dependencies for the build depends on a package, where 
>> the
>> commit version is no longer around. The main issue with gogradle is that
>> it's entirely distinct from the usual Go workflow, which means deps users
>> use are likely to be different to what's in the lock file.
>>
>> This work will be tracked in
>> https://issues.apache.org/jira/browse/BEAM-5379
>> GoGradle hasn't moved to support the new-go way of handling deps, so
>> my inclination is to simplify to simple scripts for Gradle that shell out
>> the to Go tool for handling Go dep management, over trying to fix 
>> GoGradle.
>>
>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>>
>>> Robert, from what I recall it's not flaky for me - it consistently
>>> fails. Let me know if there's a way to get more logging about this 
>>> error.
>>>
>>> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>>>
 It's concerning to me that 1) the Go dependency resolution via
 gogradle is flaky, and 2) that it can block other languages.

 I suppose 2) makes sense since it's part of the container
 bootstrapping code, but that makes 1) a serious problem, of which I 
 wasn't
 aware.
 I should have time to investigate this in the next two weeks.

 On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
 wrote:

> Just for the record,
>
> using a vm here, because did not yet get all task running on my
> mac, and did not want to mess with my setup.
>
> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6
> cores and further
>
> sudo apt update
>
> sudo apt install gcc
>
> sudo apt install make
>
> sudo apt install perl
>
> sudo apt install curl
>
> sudo apt install openjdk-8-jdk
>
> sudo apt install python
>
> sudo apt install -y software-properties-common
>
> sudo add-apt-repository ppa:deadsnakes/ppa
>
> sudo apt update
>
> sudo apt install python3.5
>
> sudo apt-get install apt-transport-https ca-certificates curl

Re: Build blocking on

[Michael Luckey]

On Tue, Mar 26, 2019 at 11:40 PM Kenneth Knowles  wrote:

> +1 to separating integration tests from "build". It should be able to
> succeed without internet access (if deps are cached).
>

Big +1 here.

I d even suggest the following:
- ./gradlew build must succeed 'offline' (if deps cached)
- no flaky tests on build target. Flaky tests should be 'offloaded' to
preCommit (probably)

  Main point here from my side would be user experience. If someone
downloads released source code, she must be able to just do a './gradlew
build publishToMavenLocal' without failing tests. Not fully understood
python/go in this context, though.

- no docker tests on build target

  imho just to much of required infrastructure for a 'build'




>
> On Tue, Mar 26, 2019 at 3:18 PM Michael Luckey 
> wrote:
>
>> Of course, we could implement something here. But I am worried about the
>> consequences. As gogradle writes into (user) global state this would have
>> unexpected side effects.
>>
>> Consider a developer running Project A - which happens to also use
>> gogradle - and during build of that project issues an innocent beam clean.
>> Would be no fun to track all that failures arising out of those now deleted
>> state.
>>
>> Unfortunately, I do not have a clear understanding yet, what's going on
>> here, why that cache is put there and how it gets inconsistent. Need to
>> look into that. (But might be obsoleted anyway by Roberts plans to drop
>> gogradle for something different.)
>>
>> On Tue, Mar 26, 2019 at 7:28 PM Thomas Weise  wrote:
>>
>>> Can this be addressed by having "clean" remove all state that gogradle
>>> leaves behind? This staleness issue has bitten me a few times also and it
>>> would be good to have a reliable way to deal with it, even if it involves
>>> an extra clean.
>>>
>>>
>>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
>>> wrote:
>>>
 @Udi
 Did you try to just delete the
 '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?

 @Robert
 As said before, I am a bit scared about the implications. Shelling out
 is done by python, and from build perspective, this does not work very
 well, unfortunately. I.e. no caching, up-to-date checks etc...

 But of course, we need to play with this a bit more.

 On Tue, Mar 26, 2019 at 6:24 PM Robert Burke 
 wrote:

> Reading the error from the gradle scan, it largely looks like some
> part of the GCP dependencies for the build depends on a package, where the
> commit version is no longer around. The main issue with gogradle is that
> it's entirely distinct from the usual Go workflow, which means deps users
> use are likely to be different to what's in the lock file.
>
> This work will be tracked in
> https://issues.apache.org/jira/browse/BEAM-5379
> GoGradle hasn't moved to support the new-go way of handling deps, so
> my inclination is to simplify to simple scripts for Gradle that shell out
> the to Go tool for handling Go dep management, over trying to fix 
> GoGradle.
>
> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>
>> Robert, from what I recall it's not flaky for me - it consistently
>> fails. Let me know if there's a way to get more logging about this error.
>>
>> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>>
>>> It's concerning to me that 1) the Go dependency resolution via
>>> gogradle is flaky, and 2) that it can block other languages.
>>>
>>> I suppose 2) makes sense since it's part of the container
>>> bootstrapping code, but that makes 1) a serious problem, of which I 
>>> wasn't
>>> aware.
>>> I should have time to investigate this in the next two weeks.
>>>
>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
>>> wrote:
>>>
 Just for the record,

 using a vm here, because did not yet get all task running on my
 mac, and did not want to mess with my setup.

 So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6
 cores and further

 sudo apt update

 sudo apt install gcc

 sudo apt install make

 sudo apt install perl

 sudo apt install curl

 sudo apt install openjdk-8-jdk

 sudo apt install python

 sudo apt install -y software-properties-common

 sudo add-apt-repository ppa:deadsnakes/ppa

 sudo apt update

 sudo apt install python3.5

 sudo apt-get install apt-transport-https ca-certificates curl
 gnupg-agent software-properties-common

 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo
 apt-key add -

 sudo apt-key fingerprint 0EBFCD88

 sudo add-apt-repository "deb [arch=amd64]
 

Re: Build blocking on

[Michael Luckey]

On Tue, Mar 26, 2019 at 11:18 PM Mikhail Gryzykhin 
wrote:

> I believe what happens is that testPy2Gcp actually runs integration tests
> that try to connect to GCP.
>

Actually I was hoping for an explanation like this. Any suggestion how I
could confirm that on my behalf?


> Without having GCP cluster and configuration on your machine I'd expect
> these tests to fail.
>

Hmm... here I am actually unsure, what would be the best to handle such
cases.

If I understand correctly, we currently skip some tests which do not meet
expectations, kind of 'can not run on your arch' thingies... So I am
undecided, whether I d prefer those tests to be skipped if gcp
configuration is missing

pro
* dev is still able to run the tests (whichever task they are associated
with) without having to separate the failures out. For instance, these
'testPy2Gcp' does actually execute 'some tests' - which might be already
covered by some other calls... But I definitely do not like the idea, to
put the burden on the developer to track which tasks/tests might be
executed on local machine. Unless this distinction is really coarse - and
pre/postcommit is something I really would like to be able to run locally...


con
* we definitely need to make sure, those tests are not accidentally skipped
on CI servers.


>
> I'd say we should remove testPy2Gcp task from "build" task and explicitly
> keep it as integration test.
>
> --Mikhail
>
>
> On Tue, Mar 26, 2019 at 3:12 PM Michael Luckey 
> wrote:
>
>>
>>
>> On Tue, Mar 26, 2019 at 10:29 PM Udi Meiri  wrote:
>>
>>> Luckey, I couldn't recreate your issue, but I still haven't done a full
>>> build.
>>> I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a
>>> image (n1-standard-4 machine type).
>>>
>>> Ran the following:
>>> sudo apt-get update
>>> sudo apt-get install python-pip
>>> sudo apt-get install python-virtualenv
>>> git clone https://github.com/apache/beam.git
>>> cd beam
>>> ./gradlew :beam-sdks-python:testPy2Gcp
>>> [failed: no JAVA_HOME]
>>> sudo apt-get install openjdk-8-jdk
>>> ./gradlew :beam-sdks-python:testPy2Gcp
>>>
>>> Got: BUILD SUCCESSFUL in 7m 52s
>>>
>>
>> Nice. Thanks a lot for your help here.
>>
>> If I understand correctly, this VM is already located within gcp. Could
>> it already have some setup, which needs to be done on 'my' VM? For instance
>> I was contemplating about that test trying 'to call home', but as I am
>> (unfortunately ;) no googler and do not have any gcp specific setup, fails
>> here but misses to timeout? This is just some weird assumption, did not yet
>> look into the actual implementation.
>>
>> Which I seemingly need to do here :(
>>
>>
>>> Then I tried:
>>> ./gradlew build
>>>
>>> And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot disk
>>> is 10G total)
>>>
>>
>> Ouch :D
>>
>>
>>>
>>> On Tue, Mar 26, 2019 at 1:35 PM Robert Burke  wrote:
>>>
 Michael, your concern is reasonable, especially with the experience
 with python, though that does help me bootstrap this work. :)

 The go tools provide caching and avoid redoing work if the source files
 haven't changed. This applies most particularly for `go build` and `go
 test`. As long as the go code isn't changing at every invocation, this
 should be fine. I'm not aware of the same being the case for the usual
 python tools.

  The real trick is ensuring a valid and consistent environment for the
 go code.

 The environment question becomes easier for everyone by moving to go
 modules, which were designed to provide these kinds of consistent builds.
 It also avoids needing a GOPATH set. Any directory is permitted, as long as
 the go.mod is present.

 (The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet
 in the repo.)

 The main blocker is see is updating the Jenkins machines to have the
 latest version of Go (1.12) instead of 1.10, which doesn't support modules.
 This only blocks a final submission, rather than the work fortunately.

 On Tue, Mar 26, 2019, 1:08 PM Udi Meiri  wrote:

> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one
> package with issues).
> My ~/.bashrc has
>   export GOPATH=$HOME/go
> so maybe that's making the difference in my setup.
>
> On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise  wrote:
>
>> Can this be addressed by having "clean" remove all state that
>> gogradle leaves behind? This staleness issue has bitten me a few times 
>> also
>> and it would be good to have a reliable way to deal with it, even if it
>> involves an extra clean.
>>
>>
>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
>> wrote:
>>
>>> @Udi
>>> Did you try to just delete the
>>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com'
>>> folder?
>>>
>>> @Robert
>>> As said before, I am a bit scared about the implications. 

Re: Build blocking on

[Kenneth Knowles]

+1 to separating integration tests from "build". It should be able to
succeed without internet access (if deps are cached).

On Tue, Mar 26, 2019 at 3:18 PM Michael Luckey  wrote:

> Of course, we could implement something here. But I am worried about the
> consequences. As gogradle writes into (user) global state this would have
> unexpected side effects.
>
> Consider a developer running Project A - which happens to also use
> gogradle - and during build of that project issues an innocent beam clean.
> Would be no fun to track all that failures arising out of those now deleted
> state.
>
> Unfortunately, I do not have a clear understanding yet, what's going on
> here, why that cache is put there and how it gets inconsistent. Need to
> look into that. (But might be obsoleted anyway by Roberts plans to drop
> gogradle for something different.)
>
> On Tue, Mar 26, 2019 at 7:28 PM Thomas Weise  wrote:
>
>> Can this be addressed by having "clean" remove all state that gogradle
>> leaves behind? This staleness issue has bitten me a few times also and it
>> would be good to have a reliable way to deal with it, even if it involves
>> an extra clean.
>>
>>
>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
>> wrote:
>>
>>> @Udi
>>> Did you try to just delete the
>>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?
>>>
>>> @Robert
>>> As said before, I am a bit scared about the implications. Shelling out
>>> is done by python, and from build perspective, this does not work very
>>> well, unfortunately. I.e. no caching, up-to-date checks etc...
>>>
>>> But of course, we need to play with this a bit more.
>>>
>>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke  wrote:
>>>
 Reading the error from the gradle scan, it largely looks like some part
 of the GCP dependencies for the build depends on a package, where the
 commit version is no longer around. The main issue with gogradle is that
 it's entirely distinct from the usual Go workflow, which means deps users
 use are likely to be different to what's in the lock file.

 This work will be tracked in
 https://issues.apache.org/jira/browse/BEAM-5379
 GoGradle hasn't moved to support the new-go way of handling deps, so my
 inclination is to simplify to simple scripts for Gradle that shell out the
 to Go tool for handling Go dep management, over trying to fix GoGradle.

 On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:

> Robert, from what I recall it's not flaky for me - it consistently
> fails. Let me know if there's a way to get more logging about this error.
>
> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>
>> It's concerning to me that 1) the Go dependency resolution via
>> gogradle is flaky, and 2) that it can block other languages.
>>
>> I suppose 2) makes sense since it's part of the container
>> bootstrapping code, but that makes 1) a serious problem, of which I 
>> wasn't
>> aware.
>> I should have time to investigate this in the next two weeks.
>>
>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
>> wrote:
>>
>>> Just for the record,
>>>
>>> using a vm here, because did not yet get all task running on my mac,
>>> and did not want to mess with my setup.
>>>
>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6
>>> cores and further
>>>
>>> sudo apt update
>>>
>>> sudo apt install gcc
>>>
>>> sudo apt install make
>>>
>>> sudo apt install perl
>>>
>>> sudo apt install curl
>>>
>>> sudo apt install openjdk-8-jdk
>>>
>>> sudo apt install python
>>>
>>> sudo apt install -y software-properties-common
>>>
>>> sudo add-apt-repository ppa:deadsnakes/ppa
>>>
>>> sudo apt update
>>>
>>> sudo apt install python3.5
>>>
>>> sudo apt-get install apt-transport-https ca-certificates curl
>>> gnupg-agent software-properties-common
>>>
>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo
>>> apt-key add -
>>>
>>> sudo apt-key fingerprint 0EBFCD88
>>>
>>> sudo add-apt-repository "deb [arch=amd64]
>>> https://download.docker.com/linux/ubuntu \
>>>
>>> $(lsb_release -cs) \
>>>
>>> stable"
>>>
>>> sudo apt-get update
>>>
>>> sudo apt-get install docker-ce docker-ce-cli containerd.io
>>>
>>> sudo groupadd docker
>>>
>>> sudo usermod -aG docker $USER
>>>
>>> git config --global user.email "d...@spam.me"
>>>
>>> git config --global user.name "Some Guy"
>>>
>>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
>>>
>>> sudo python get-pip.py
>>>
>>> rm get-pip.py
>>>
>>> sudo pip install --upgrade virtualenv
>>>
>>> sudo pip install cython
>>>
>>> sudo apt-get install python-dev
>>>
>>> sudo apt-get install python3-distutils
>>>

Re: Build blocking on

[Mikhail Gryzykhin]

I believe what happens is that testPy2Gcp actually runs integration tests
that try to connect to GCP. Without having GCP cluster and configuration on
your machine I'd expect these tests to fail.

I'd say we should remove testPy2Gcp task from "build" task and explicitly
keep it as integration test.

--Mikhail


On Tue, Mar 26, 2019 at 3:12 PM Michael Luckey  wrote:

>
>
> On Tue, Mar 26, 2019 at 10:29 PM Udi Meiri  wrote:
>
>> Luckey, I couldn't recreate your issue, but I still haven't done a full
>> build.
>> I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a image
>> (n1-standard-4 machine type).
>>
>> Ran the following:
>> sudo apt-get update
>> sudo apt-get install python-pip
>> sudo apt-get install python-virtualenv
>> git clone https://github.com/apache/beam.git
>> cd beam
>> ./gradlew :beam-sdks-python:testPy2Gcp
>> [failed: no JAVA_HOME]
>> sudo apt-get install openjdk-8-jdk
>> ./gradlew :beam-sdks-python:testPy2Gcp
>>
>> Got: BUILD SUCCESSFUL in 7m 52s
>>
>
> Nice. Thanks a lot for your help here.
>
> If I understand correctly, this VM is already located within gcp. Could it
> already have some setup, which needs to be done on 'my' VM? For instance I
> was contemplating about that test trying 'to call home', but as I am
> (unfortunately ;) no googler and do not have any gcp specific setup, fails
> here but misses to timeout? This is just some weird assumption, did not yet
> look into the actual implementation.
>
> Which I seemingly need to do here :(
>
>
>> Then I tried:
>> ./gradlew build
>>
>> And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot disk
>> is 10G total)
>>
>
> Ouch :D
>
>
>>
>> On Tue, Mar 26, 2019 at 1:35 PM Robert Burke  wrote:
>>
>>> Michael, your concern is reasonable, especially with the experience with
>>> python, though that does help me bootstrap this work. :)
>>>
>>> The go tools provide caching and avoid redoing work if the source files
>>> haven't changed. This applies most particularly for `go build` and `go
>>> test`. As long as the go code isn't changing at every invocation, this
>>> should be fine. I'm not aware of the same being the case for the usual
>>> python tools.
>>>
>>>  The real trick is ensuring a valid and consistent environment for the
>>> go code.
>>>
>>> The environment question becomes easier for everyone by moving to go
>>> modules, which were designed to provide these kinds of consistent builds.
>>> It also avoids needing a GOPATH set. Any directory is permitted, as long as
>>> the go.mod is present.
>>>
>>> (The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet
>>> in the repo.)
>>>
>>> The main blocker is see is updating the Jenkins machines to have the
>>> latest version of Go (1.12) instead of 1.10, which doesn't support modules.
>>> This only blocks a final submission, rather than the work fortunately.
>>>
>>> On Tue, Mar 26, 2019, 1:08 PM Udi Meiri  wrote:
>>>
 "rm -r ~/.gradle/go/repo/" worked for me (there was more than one
 package with issues).
 My ~/.bashrc has
   export GOPATH=$HOME/go
 so maybe that's making the difference in my setup.

 On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise  wrote:

> Can this be addressed by having "clean" remove all state that gogradle
> leaves behind? This staleness issue has bitten me a few times also and it
> would be good to have a reliable way to deal with it, even if it involves
> an extra clean.
>
>
> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
> wrote:
>
>> @Udi
>> Did you try to just delete the
>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com'
>> folder?
>>
>> @Robert
>> As said before, I am a bit scared about the implications. Shelling
>> out is done by python, and from build perspective, this does not work 
>> very
>> well, unfortunately. I.e. no caching, up-to-date checks etc...
>>
>> But of course, we need to play with this a bit more.
>>
>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke 
>> wrote:
>>
>>> Reading the error from the gradle scan, it largely looks like some
>>> part of the GCP dependencies for the build depends on a package, where 
>>> the
>>> commit version is no longer around. The main issue with gogradle is that
>>> it's entirely distinct from the usual Go workflow, which means deps 
>>> users
>>> use are likely to be different to what's in the lock file.
>>>
>>> This work will be tracked in
>>> https://issues.apache.org/jira/browse/BEAM-5379
>>> GoGradle hasn't moved to support the new-go way of handling deps, so
>>> my inclination is to simplify to simple scripts for Gradle that shell 
>>> out
>>> the to Go tool for handling Go dep management, over trying to fix 
>>> GoGradle.
>>>
>>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>>>
 Robert, from what I recall it's not flaky for me - it consistently

Re: Build blocking on

[Michael Luckey]

On Tue, Mar 26, 2019 at 10:29 PM Udi Meiri  wrote:

> Luckey, I couldn't recreate your issue, but I still haven't done a full
> build.
> I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a image
> (n1-standard-4 machine type).
>
> Ran the following:
> sudo apt-get update
> sudo apt-get install python-pip
> sudo apt-get install python-virtualenv
> git clone https://github.com/apache/beam.git
> cd beam
> ./gradlew :beam-sdks-python:testPy2Gcp
> [failed: no JAVA_HOME]
> sudo apt-get install openjdk-8-jdk
> ./gradlew :beam-sdks-python:testPy2Gcp
>
> Got: BUILD SUCCESSFUL in 7m 52s
>

Nice. Thanks a lot for your help here.

If I understand correctly, this VM is already located within gcp. Could it
already have some setup, which needs to be done on 'my' VM? For instance I
was contemplating about that test trying 'to call home', but as I am
(unfortunately ;) no googler and do not have any gcp specific setup, fails
here but misses to timeout? This is just some weird assumption, did not yet
look into the actual implementation.

Which I seemingly need to do here :(


> Then I tried:
> ./gradlew build
>
> And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot disk
> is 10G total)
>

Ouch :D


>
> On Tue, Mar 26, 2019 at 1:35 PM Robert Burke  wrote:
>
>> Michael, your concern is reasonable, especially with the experience with
>> python, though that does help me bootstrap this work. :)
>>
>> The go tools provide caching and avoid redoing work if the source files
>> haven't changed. This applies most particularly for `go build` and `go
>> test`. As long as the go code isn't changing at every invocation, this
>> should be fine. I'm not aware of the same being the case for the usual
>> python tools.
>>
>>  The real trick is ensuring a valid and consistent environment for the go
>> code.
>>
>> The environment question becomes easier for everyone by moving to go
>> modules, which were designed to provide these kinds of consistent builds.
>> It also avoids needing a GOPATH set. Any directory is permitted, as long as
>> the go.mod is present.
>>
>> (The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet in
>> the repo.)
>>
>> The main blocker is see is updating the Jenkins machines to have the
>> latest version of Go (1.12) instead of 1.10, which doesn't support modules.
>> This only blocks a final submission, rather than the work fortunately.
>>
>> On Tue, Mar 26, 2019, 1:08 PM Udi Meiri  wrote:
>>
>>> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one
>>> package with issues).
>>> My ~/.bashrc has
>>>   export GOPATH=$HOME/go
>>> so maybe that's making the difference in my setup.
>>>
>>> On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise  wrote:
>>>
 Can this be addressed by having "clean" remove all state that gogradle
 leaves behind? This staleness issue has bitten me a few times also and it
 would be good to have a reliable way to deal with it, even if it involves
 an extra clean.


 On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
 wrote:

> @Udi
> Did you try to just delete the
> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com'
> folder?
>
> @Robert
> As said before, I am a bit scared about the implications. Shelling out
> is done by python, and from build perspective, this does not work very
> well, unfortunately. I.e. no caching, up-to-date checks etc...
>
> But of course, we need to play with this a bit more.
>
> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke 
> wrote:
>
>> Reading the error from the gradle scan, it largely looks like some
>> part of the GCP dependencies for the build depends on a package, where 
>> the
>> commit version is no longer around. The main issue with gogradle is that
>> it's entirely distinct from the usual Go workflow, which means deps users
>> use are likely to be different to what's in the lock file.
>>
>> This work will be tracked in
>> https://issues.apache.org/jira/browse/BEAM-5379
>> GoGradle hasn't moved to support the new-go way of handling deps, so
>> my inclination is to simplify to simple scripts for Gradle that shell out
>> the to Go tool for handling Go dep management, over trying to fix 
>> GoGradle.
>>
>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>>
>>> Robert, from what I recall it's not flaky for me - it consistently
>>> fails. Let me know if there's a way to get more logging about this 
>>> error.
>>>
>>> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>>>
 It's concerning to me that 1) the Go dependency resolution via
 gogradle is flaky, and 2) that it can block other languages.

 I suppose 2) makes sense since it's part of the container
 bootstrapping code, but that makes 1) a serious problem, of which I 
 wasn't
 aware.
 I should have time to 

Re: Build blocking on

[Michael Luckey]

On Tue, Mar 26, 2019 at 9:35 PM Robert Burke  wrote:

> Michael, your concern is reasonable, especially with the experience with
> python, though that does help me bootstrap this work. :)
>
> The go tools provide caching and avoid redoing work if the source files
> haven't changed.
>

Of course, that's pretty cool. But I am currently unable to fully get the
consequences. It would still be better, if the build system (in our case
currently gradle) would track this, as it might chose to do different
processing. And of course, the benefits from using the build cache would
also be missed out.

To clarify, I am not arguing against using/delegating to native tools - in
fact gradle always delegates to native tools, e.g. javac - and also
gogradle is doing the same, I am just trying to point out, that we should
keep a close eye on the actual implementation to leverage all those gradle
goodies. Which is easily done wrong with introducing all kind of issues in
the build. And should not be done by simply calling out to some shell
script. For instance our python implementation seems far to often just
return a '-1' and requires some digging into logs to eventually find the
root cause. Which I believe could be done better.

What we would essentially try to do, is kind of reimplementing gogradle,
which is doable, of course, to get over the shortcomings of that plugin.


> This applies most particularly for `go build` and `go test`. As long as
> the go code isn't changing at every invocation, this should be fine. I'm
> not aware of the same being the case for the usual python tools.
>

>  The real trick is ensuring a valid and consistent environment for the go
> code.
>
> The environment question becomes easier for everyone by moving to go
> modules, which were designed to provide these kinds of consistent builds.
> It also avoids needing a GOPATH set. Any directory is permitted, as long as
> the go.mod is present.
>
> (The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet in
> the repo.)
>
> The main blocker is see is updating the Jenkins machines to have the
> latest version of Go (1.12) instead of 1.10, which doesn't support modules.
> This only blocks a final submission, rather than the work fortunately.
>

I d really love to assist you in getting that running. So if you think I
could be of help here, just ping me and we could give that a try.


>
> On Tue, Mar 26, 2019, 1:08 PM Udi Meiri  wrote:
>
>> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one package
>> with issues).
>> My ~/.bashrc has
>>   export GOPATH=$HOME/go
>> so maybe that's making the difference in my setup.
>>
>> On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise  wrote:
>>
>>> Can this be addressed by having "clean" remove all state that gogradle
>>> leaves behind? This staleness issue has bitten me a few times also and it
>>> would be good to have a reliable way to deal with it, even if it involves
>>> an extra clean.
>>>
>>>
>>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
>>> wrote:
>>>
 @Udi
 Did you try to just delete the
 '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?

 @Robert
 As said before, I am a bit scared about the implications. Shelling out
 is done by python, and from build perspective, this does not work very
 well, unfortunately. I.e. no caching, up-to-date checks etc...

 But of course, we need to play with this a bit more.

 On Tue, Mar 26, 2019 at 6:24 PM Robert Burke 
 wrote:

> Reading the error from the gradle scan, it largely looks like some
> part of the GCP dependencies for the build depends on a package, where the
> commit version is no longer around. The main issue with gogradle is that
> it's entirely distinct from the usual Go workflow, which means deps users
> use are likely to be different to what's in the lock file.
>
> This work will be tracked in
> https://issues.apache.org/jira/browse/BEAM-5379
> GoGradle hasn't moved to support the new-go way of handling deps, so
> my inclination is to simplify to simple scripts for Gradle that shell out
> the to Go tool for handling Go dep management, over trying to fix 
> GoGradle.
>
> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>
>> Robert, from what I recall it's not flaky for me - it consistently
>> fails. Let me know if there's a way to get more logging about this error.
>>
>> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>>
>>> It's concerning to me that 1) the Go dependency resolution via
>>> gogradle is flaky, and 2) that it can block other languages.
>>>
>>> I suppose 2) makes sense since it's part of the container
>>> bootstrapping code, but that makes 1) a serious problem, of which I 
>>> wasn't
>>> aware.
>>> I should have time to investigate this in the next two weeks.
>>>
>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
>>> 

Re: Build blocking on

[Udi Meiri]

Luckey, I couldn't recreate your issue, but I still haven't done a full
build.
I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a image
(n1-standard-4 machine type).

Ran the following:
sudo apt-get update
sudo apt-get install python-pip
sudo apt-get install python-virtualenv
git clone https://github.com/apache/beam.git
cd beam
./gradlew :beam-sdks-python:testPy2Gcp
[failed: no JAVA_HOME]
sudo apt-get install openjdk-8-jdk
./gradlew :beam-sdks-python:testPy2Gcp

Got: BUILD SUCCESSFUL in 7m 52s

Then I tried:
./gradlew build

And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot disk is
10G total)

On Tue, Mar 26, 2019 at 1:35 PM Robert Burke  wrote:

> Michael, your concern is reasonable, especially with the experience with
> python, though that does help me bootstrap this work. :)
>
> The go tools provide caching and avoid redoing work if the source files
> haven't changed. This applies most particularly for `go build` and `go
> test`. As long as the go code isn't changing at every invocation, this
> should be fine. I'm not aware of the same being the case for the usual
> python tools.
>
>  The real trick is ensuring a valid and consistent environment for the go
> code.
>
> The environment question becomes easier for everyone by moving to go
> modules, which were designed to provide these kinds of consistent builds.
> It also avoids needing a GOPATH set. Any directory is permitted, as long as
> the go.mod is present.
>
> (The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet in
> the repo.)
>
> The main blocker is see is updating the Jenkins machines to have the
> latest version of Go (1.12) instead of 1.10, which doesn't support modules.
> This only blocks a final submission, rather than the work fortunately.
>
> On Tue, Mar 26, 2019, 1:08 PM Udi Meiri  wrote:
>
>> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one package
>> with issues).
>> My ~/.bashrc has
>>   export GOPATH=$HOME/go
>> so maybe that's making the difference in my setup.
>>
>> On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise  wrote:
>>
>>> Can this be addressed by having "clean" remove all state that gogradle
>>> leaves behind? This staleness issue has bitten me a few times also and it
>>> would be good to have a reliable way to deal with it, even if it involves
>>> an extra clean.
>>>
>>>
>>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
>>> wrote:
>>>
 @Udi
 Did you try to just delete the
 '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?

 @Robert
 As said before, I am a bit scared about the implications. Shelling out
 is done by python, and from build perspective, this does not work very
 well, unfortunately. I.e. no caching, up-to-date checks etc...

 But of course, we need to play with this a bit more.

 On Tue, Mar 26, 2019 at 6:24 PM Robert Burke 
 wrote:

> Reading the error from the gradle scan, it largely looks like some
> part of the GCP dependencies for the build depends on a package, where the
> commit version is no longer around. The main issue with gogradle is that
> it's entirely distinct from the usual Go workflow, which means deps users
> use are likely to be different to what's in the lock file.
>
> This work will be tracked in
> https://issues.apache.org/jira/browse/BEAM-5379
> GoGradle hasn't moved to support the new-go way of handling deps, so
> my inclination is to simplify to simple scripts for Gradle that shell out
> the to Go tool for handling Go dep management, over trying to fix 
> GoGradle.
>
> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>
>> Robert, from what I recall it's not flaky for me - it consistently
>> fails. Let me know if there's a way to get more logging about this error.
>>
>> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>>
>>> It's concerning to me that 1) the Go dependency resolution via
>>> gogradle is flaky, and 2) that it can block other languages.
>>>
>>> I suppose 2) makes sense since it's part of the container
>>> bootstrapping code, but that makes 1) a serious problem, of which I 
>>> wasn't
>>> aware.
>>> I should have time to investigate this in the next two weeks.
>>>
>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
>>> wrote:
>>>
 Just for the record,

 using a vm here, because did not yet get all task running on my
 mac, and did not want to mess with my setup.

 So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6
 cores and further

 sudo apt update

 sudo apt install gcc

 sudo apt install make

 sudo apt install perl

 sudo apt install curl

 sudo apt install openjdk-8-jdk

 sudo apt install python

 sudo apt 

Re: Build blocking on

[Robert Burke]

Michael, your concern is reasonable, especially with the experience with
python, though that does help me bootstrap this work. :)

The go tools provide caching and avoid redoing work if the source files
haven't changed. This applies most particularly for `go build` and `go
test`. As long as the go code isn't changing at every invocation, this
should be fine. I'm not aware of the same being the case for the usual
python tools.

 The real trick is ensuring a valid and consistent environment for the go
code.

The environment question becomes easier for everyone by moving to go
modules, which were designed to provide these kinds of consistent builds.
It also avoids needing a GOPATH set. Any directory is permitted, as long as
the go.mod is present.

(The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet in
the repo.)

The main blocker is see is updating the Jenkins machines to have the latest
version of Go (1.12) instead of 1.10, which doesn't support modules. This
only blocks a final submission, rather than the work fortunately.

On Tue, Mar 26, 2019, 1:08 PM Udi Meiri  wrote:

> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one package
> with issues).
> My ~/.bashrc has
>   export GOPATH=$HOME/go
> so maybe that's making the difference in my setup.
>
> On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise  wrote:
>
>> Can this be addressed by having "clean" remove all state that gogradle
>> leaves behind? This staleness issue has bitten me a few times also and it
>> would be good to have a reliable way to deal with it, even if it involves
>> an extra clean.
>>
>>
>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
>> wrote:
>>
>>> @Udi
>>> Did you try to just delete the
>>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?
>>>
>>> @Robert
>>> As said before, I am a bit scared about the implications. Shelling out
>>> is done by python, and from build perspective, this does not work very
>>> well, unfortunately. I.e. no caching, up-to-date checks etc...
>>>
>>> But of course, we need to play with this a bit more.
>>>
>>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke  wrote:
>>>
 Reading the error from the gradle scan, it largely looks like some part
 of the GCP dependencies for the build depends on a package, where the
 commit version is no longer around. The main issue with gogradle is that
 it's entirely distinct from the usual Go workflow, which means deps users
 use are likely to be different to what's in the lock file.

 This work will be tracked in
 https://issues.apache.org/jira/browse/BEAM-5379
 GoGradle hasn't moved to support the new-go way of handling deps, so my
 inclination is to simplify to simple scripts for Gradle that shell out the
 to Go tool for handling Go dep management, over trying to fix GoGradle.

 On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:

> Robert, from what I recall it's not flaky for me - it consistently
> fails. Let me know if there's a way to get more logging about this error.
>
> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>
>> It's concerning to me that 1) the Go dependency resolution via
>> gogradle is flaky, and 2) that it can block other languages.
>>
>> I suppose 2) makes sense since it's part of the container
>> bootstrapping code, but that makes 1) a serious problem, of which I 
>> wasn't
>> aware.
>> I should have time to investigate this in the next two weeks.
>>
>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
>> wrote:
>>
>>> Just for the record,
>>>
>>> using a vm here, because did not yet get all task running on my mac,
>>> and did not want to mess with my setup.
>>>
>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6
>>> cores and further
>>>
>>> sudo apt update
>>>
>>> sudo apt install gcc
>>>
>>> sudo apt install make
>>>
>>> sudo apt install perl
>>>
>>> sudo apt install curl
>>>
>>> sudo apt install openjdk-8-jdk
>>>
>>> sudo apt install python
>>>
>>> sudo apt install -y software-properties-common
>>>
>>> sudo add-apt-repository ppa:deadsnakes/ppa
>>>
>>> sudo apt update
>>>
>>> sudo apt install python3.5
>>>
>>> sudo apt-get install apt-transport-https ca-certificates curl
>>> gnupg-agent software-properties-common
>>>
>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo
>>> apt-key add -
>>>
>>> sudo apt-key fingerprint 0EBFCD88
>>>
>>> sudo add-apt-repository "deb [arch=amd64]
>>> https://download.docker.com/linux/ubuntu \
>>>
>>> $(lsb_release -cs) \
>>>
>>> stable"
>>>
>>> sudo apt-get update
>>>
>>> sudo apt-get install docker-ce docker-ce-cli containerd.io
>>>
>>> sudo groupadd docker
>>>
>>> sudo usermod -aG docker $USER
>>>
>>> git 

Re: Build blocking on

[Udi Meiri]

"rm -r ~/.gradle/go/repo/" worked for me (there was more than one package
with issues).
My ~/.bashrc has
  export GOPATH=$HOME/go
so maybe that's making the difference in my setup.

On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise  wrote:

> Can this be addressed by having "clean" remove all state that gogradle
> leaves behind? This staleness issue has bitten me a few times also and it
> would be good to have a reliable way to deal with it, even if it involves
> an extra clean.
>
>
> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey 
> wrote:
>
>> @Udi
>> Did you try to just delete the
>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?
>>
>> @Robert
>> As said before, I am a bit scared about the implications. Shelling out is
>> done by python, and from build perspective, this does not work very well,
>> unfortunately. I.e. no caching, up-to-date checks etc...
>>
>> But of course, we need to play with this a bit more.
>>
>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke  wrote:
>>
>>> Reading the error from the gradle scan, it largely looks like some part
>>> of the GCP dependencies for the build depends on a package, where the
>>> commit version is no longer around. The main issue with gogradle is that
>>> it's entirely distinct from the usual Go workflow, which means deps users
>>> use are likely to be different to what's in the lock file.
>>>
>>> This work will be tracked in
>>> https://issues.apache.org/jira/browse/BEAM-5379
>>> GoGradle hasn't moved to support the new-go way of handling deps, so my
>>> inclination is to simplify to simple scripts for Gradle that shell out the
>>> to Go tool for handling Go dep management, over trying to fix GoGradle.
>>>
>>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>>>
 Robert, from what I recall it's not flaky for me - it consistently
 fails. Let me know if there's a way to get more logging about this error.

 On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:

> It's concerning to me that 1) the Go dependency resolution via
> gogradle is flaky, and 2) that it can block other languages.
>
> I suppose 2) makes sense since it's part of the container
> bootstrapping code, but that makes 1) a serious problem, of which I wasn't
> aware.
> I should have time to investigate this in the next two weeks.
>
> On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
> wrote:
>
>> Just for the record,
>>
>> using a vm here, because did not yet get all task running on my mac,
>> and did not want to mess with my setup.
>>
>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6
>> cores and further
>>
>> sudo apt update
>>
>> sudo apt install gcc
>>
>> sudo apt install make
>>
>> sudo apt install perl
>>
>> sudo apt install curl
>>
>> sudo apt install openjdk-8-jdk
>>
>> sudo apt install python
>>
>> sudo apt install -y software-properties-common
>>
>> sudo add-apt-repository ppa:deadsnakes/ppa
>>
>> sudo apt update
>>
>> sudo apt install python3.5
>>
>> sudo apt-get install apt-transport-https ca-certificates curl
>> gnupg-agent software-properties-common
>>
>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo
>> apt-key add -
>>
>> sudo apt-key fingerprint 0EBFCD88
>>
>> sudo add-apt-repository "deb [arch=amd64]
>> https://download.docker.com/linux/ubuntu \
>>
>> $(lsb_release -cs) \
>>
>> stable"
>>
>> sudo apt-get update
>>
>> sudo apt-get install docker-ce docker-ce-cli containerd.io
>>
>> sudo groupadd docker
>>
>> sudo usermod -aG docker $USER
>>
>> git config --global user.email "d...@spam.me"
>>
>> git config --global user.name "Some Guy"
>>
>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
>>
>> sudo python get-pip.py
>>
>> rm get-pip.py
>>
>> sudo pip install --upgrade virtualenv
>>
>> sudo pip install cython
>>
>> sudo apt-get install python-dev
>>
>> sudo apt-get install python3-distutils
>>
>> sudo apt-get install python3-dev # for python3.x installs
>>
>>
>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew
>> build
>>
>> Nothing else changed/added. (hopefully, need to reassure myself here)
>>
>> Unfortunately, this is failing. Need to exclude those python tests
>> (and of course website, which usually fails on lira links)
>>
>> So I might be missing some env settings for gap, dunno. Probably
>> missed some docs.
>>
>>
>>
>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey 
>> wrote:
>>
>>> Thanks Udi for trying that!
>>>
>>> In fact, the go dependency resolution is flaky. Did not look into
>>> that, but just rerunning usually works. Of course, less than optimal, 
>>> but,

Re: Build blocking on

[Thomas Weise]

Can this be addressed by having "clean" remove all state that gogradle
leaves behind? This staleness issue has bitten me a few times also and it
would be good to have a reliable way to deal with it, even if it involves
an extra clean.


On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey  wrote:

> @Udi
> Did you try to just delete the
> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?
>
> @Robert
> As said before, I am a bit scared about the implications. Shelling out is
> done by python, and from build perspective, this does not work very well,
> unfortunately. I.e. no caching, up-to-date checks etc...
>
> But of course, we need to play with this a bit more.
>
> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke  wrote:
>
>> Reading the error from the gradle scan, it largely looks like some part
>> of the GCP dependencies for the build depends on a package, where the
>> commit version is no longer around. The main issue with gogradle is that
>> it's entirely distinct from the usual Go workflow, which means deps users
>> use are likely to be different to what's in the lock file.
>>
>> This work will be tracked in
>> https://issues.apache.org/jira/browse/BEAM-5379
>> GoGradle hasn't moved to support the new-go way of handling deps, so my
>> inclination is to simplify to simple scripts for Gradle that shell out the
>> to Go tool for handling Go dep management, over trying to fix GoGradle.
>>
>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>>
>>> Robert, from what I recall it's not flaky for me - it consistently
>>> fails. Let me know if there's a way to get more logging about this error.
>>>
>>> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>>>
 It's concerning to me that 1) the Go dependency resolution via gogradle
 is flaky, and 2) that it can block other languages.

 I suppose 2) makes sense since it's part of the container bootstrapping
 code, but that makes 1) a serious problem, of which I wasn't aware.
 I should have time to investigate this in the next two weeks.

 On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
 wrote:

> Just for the record,
>
> using a vm here, because did not yet get all task running on my mac,
> and did not want to mess with my setup.
>
> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6
> cores and further
>
> sudo apt update
>
> sudo apt install gcc
>
> sudo apt install make
>
> sudo apt install perl
>
> sudo apt install curl
>
> sudo apt install openjdk-8-jdk
>
> sudo apt install python
>
> sudo apt install -y software-properties-common
>
> sudo add-apt-repository ppa:deadsnakes/ppa
>
> sudo apt update
>
> sudo apt install python3.5
>
> sudo apt-get install apt-transport-https ca-certificates curl
> gnupg-agent software-properties-common
>
> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo
> apt-key add -
>
> sudo apt-key fingerprint 0EBFCD88
>
> sudo add-apt-repository "deb [arch=amd64]
> https://download.docker.com/linux/ubuntu \
>
> $(lsb_release -cs) \
>
> stable"
>
> sudo apt-get update
>
> sudo apt-get install docker-ce docker-ce-cli containerd.io
>
> sudo groupadd docker
>
> sudo usermod -aG docker $USER
>
> git config --global user.email "d...@spam.me"
>
> git config --global user.name "Some Guy"
>
> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
>
> sudo python get-pip.py
>
> rm get-pip.py
>
> sudo pip install --upgrade virtualenv
>
> sudo pip install cython
>
> sudo apt-get install python-dev
>
> sudo apt-get install python3-distutils
>
> sudo apt-get install python3-dev # for python3.x installs
>
>
> git clone https://github.com/apache/beam.git cd beam/ ./gradlew build
>
> Nothing else changed/added. (hopefully, need to reassure myself here)
>
> Unfortunately, this is failing. Need to exclude those python tests
> (and of course website, which usually fails on lira links)
>
> So I might be missing some env settings for gap, dunno. Probably
> missed some docs.
>
>
>
> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey 
> wrote:
>
>> Thanks Udi for trying that!
>>
>> In fact, the go dependency resolution is flaky. Did not look into
>> that, but just rerunning usually works. Of course, less than optimal, 
>> but,
>> well...
>>
>> Running build target is of course just an aggregation of task to run.
>> And unfortunately just running that
>>
>> ./gradlew  :beam-sdks-python:testPy2Gcp
>>
>> stalls on my (virtual) machine.
>>
>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:
>>
>>> Okay, `./gradlew build` failed pretty quickly for me:
>>>
>>> > Task 

Re: Build blocking on

[Michael Luckey]

@Udi
Did you try to just delete the
'/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder?

@Robert
As said before, I am a bit scared about the implications. Shelling out is
done by python, and from build perspective, this does not work very well,
unfortunately. I.e. no caching, up-to-date checks etc...

But of course, we need to play with this a bit more.

On Tue, Mar 26, 2019 at 6:24 PM Robert Burke  wrote:

> Reading the error from the gradle scan, it largely looks like some part of
> the GCP dependencies for the build depends on a package, where the commit
> version is no longer around. The main issue with gogradle is that it's
> entirely distinct from the usual Go workflow, which means deps users use
> are likely to be different to what's in the lock file.
>
> This work will be tracked in
> https://issues.apache.org/jira/browse/BEAM-5379
> GoGradle hasn't moved to support the new-go way of handling deps, so my
> inclination is to simplify to simple scripts for Gradle that shell out the
> to Go tool for handling Go dep management, over trying to fix GoGradle.
>
> On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:
>
>> Robert, from what I recall it's not flaky for me - it consistently fails.
>> Let me know if there's a way to get more logging about this error.
>>
>> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>>
>>> It's concerning to me that 1) the Go dependency resolution via gogradle
>>> is flaky, and 2) that it can block other languages.
>>>
>>> I suppose 2) makes sense since it's part of the container bootstrapping
>>> code, but that makes 1) a serious problem, of which I wasn't aware.
>>> I should have time to investigate this in the next two weeks.
>>>
>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey 
>>> wrote:
>>>
 Just for the record,

 using a vm here, because did not yet get all task running on my mac,
 and did not want to mess with my setup.

 So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores
 and further

 sudo apt update

 sudo apt install gcc

 sudo apt install make

 sudo apt install perl

 sudo apt install curl

 sudo apt install openjdk-8-jdk

 sudo apt install python

 sudo apt install -y software-properties-common

 sudo add-apt-repository ppa:deadsnakes/ppa

 sudo apt update

 sudo apt install python3.5

 sudo apt-get install apt-transport-https ca-certificates curl
 gnupg-agent software-properties-common

 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key
 add -

 sudo apt-key fingerprint 0EBFCD88

 sudo add-apt-repository "deb [arch=amd64]
 https://download.docker.com/linux/ubuntu \

 $(lsb_release -cs) \

 stable"

 sudo apt-get update

 sudo apt-get install docker-ce docker-ce-cli containerd.io

 sudo groupadd docker

 sudo usermod -aG docker $USER

 git config --global user.email "d...@spam.me"

 git config --global user.name "Some Guy"

 curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

 sudo python get-pip.py

 rm get-pip.py

 sudo pip install --upgrade virtualenv

 sudo pip install cython

 sudo apt-get install python-dev

 sudo apt-get install python3-distutils

 sudo apt-get install python3-dev # for python3.x installs


 git clone https://github.com/apache/beam.git cd beam/ ./gradlew build

 Nothing else changed/added. (hopefully, need to reassure myself here)

 Unfortunately, this is failing. Need to exclude those python tests (and
 of course website, which usually fails on lira links)

 So I might be missing some env settings for gap, dunno. Probably missed
 some docs.



 On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey 
 wrote:

> Thanks Udi for trying that!
>
> In fact, the go dependency resolution is flaky. Did not look into
> that, but just rerunning usually works. Of course, less than optimal, but,
> well...
>
> Running build target is of course just an aggregation of task to run.
> And unfortunately just running that
>
> ./gradlew  :beam-sdks-python:testPy2Gcp
>
> stalls on my (virtual) machine.
>
> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:
>
>> Okay, `./gradlew build` failed pretty quickly for me:
>>
>> > Task :beam-sdks-go:resolveBuildDependencies FAILED
>> cloud.google.com/go:
>> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[
>> https://code.googlesource.com/gocloud] does not exist in
>> /usr/local/google/home/ehudm/.gradle/go/repo/
>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will
>> be performed.
>>
>> https://gradle.com/s/x5zqbc5zwd3bg
>>
>> (Now I remember why I stopped using `build` :/)

Re: Build blocking on

[Robert Burke]

Reading the error from the gradle scan, it largely looks like some part of
the GCP dependencies for the build depends on a package, where the commit
version is no longer around. The main issue with gogradle is that it's
entirely distinct from the usual Go workflow, which means deps users use
are likely to be different to what's in the lock file.

This work will be tracked in https://issues.apache.org/jira/browse/BEAM-5379
GoGradle hasn't moved to support the new-go way of handling deps, so my
inclination is to simplify to simple scripts for Gradle that shell out the
to Go tool for handling Go dep management, over trying to fix GoGradle.

On Tue, 26 Mar 2019 at 09:43, Udi Meiri  wrote:

> Robert, from what I recall it's not flaky for me - it consistently fails.
> Let me know if there's a way to get more logging about this error.
>
> On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:
>
>> It's concerning to me that 1) the Go dependency resolution via gogradle
>> is flaky, and 2) that it can block other languages.
>>
>> I suppose 2) makes sense since it's part of the container bootstrapping
>> code, but that makes 1) a serious problem, of which I wasn't aware.
>> I should have time to investigate this in the next two weeks.
>>
>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey  wrote:
>>
>>> Just for the record,
>>>
>>> using a vm here, because did not yet get all task running on my mac, and
>>> did not want to mess with my setup.
>>>
>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores
>>> and further
>>>
>>> sudo apt update
>>>
>>> sudo apt install gcc
>>>
>>> sudo apt install make
>>>
>>> sudo apt install perl
>>>
>>> sudo apt install curl
>>>
>>> sudo apt install openjdk-8-jdk
>>>
>>> sudo apt install python
>>>
>>> sudo apt install -y software-properties-common
>>>
>>> sudo add-apt-repository ppa:deadsnakes/ppa
>>>
>>> sudo apt update
>>>
>>> sudo apt install python3.5
>>>
>>> sudo apt-get install apt-transport-https ca-certificates curl
>>> gnupg-agent software-properties-common
>>>
>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key
>>> add -
>>>
>>> sudo apt-key fingerprint 0EBFCD88
>>>
>>> sudo add-apt-repository "deb [arch=amd64]
>>> https://download.docker.com/linux/ubuntu \
>>>
>>> $(lsb_release -cs) \
>>>
>>> stable"
>>>
>>> sudo apt-get update
>>>
>>> sudo apt-get install docker-ce docker-ce-cli containerd.io
>>>
>>> sudo groupadd docker
>>>
>>> sudo usermod -aG docker $USER
>>>
>>> git config --global user.email "d...@spam.me"
>>>
>>> git config --global user.name "Some Guy"
>>>
>>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
>>>
>>> sudo python get-pip.py
>>>
>>> rm get-pip.py
>>>
>>> sudo pip install --upgrade virtualenv
>>>
>>> sudo pip install cython
>>>
>>> sudo apt-get install python-dev
>>>
>>> sudo apt-get install python3-distutils
>>>
>>> sudo apt-get install python3-dev # for python3.x installs
>>>
>>>
>>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew build
>>>
>>> Nothing else changed/added. (hopefully, need to reassure myself here)
>>>
>>> Unfortunately, this is failing. Need to exclude those python tests (and
>>> of course website, which usually fails on lira links)
>>>
>>> So I might be missing some env settings for gap, dunno. Probably missed
>>> some docs.
>>>
>>>
>>>
>>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey 
>>> wrote:
>>>
 Thanks Udi for trying that!

 In fact, the go dependency resolution is flaky. Did not look into that,
 but just rerunning usually works. Of course, less than optimal, but,
 well...

 Running build target is of course just an aggregation of task to run.
 And unfortunately just running that

 ./gradlew  :beam-sdks-python:testPy2Gcp

 stalls on my (virtual) machine.

 On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:

> Okay, `./gradlew build` failed pretty quickly for me:
>
> > Task :beam-sdks-go:resolveBuildDependencies FAILED
> cloud.google.com/go:
> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[
> https://code.googlesource.com/gocloud] does not exist in
> /usr/local/google/home/ehudm/.gradle/go/repo/
> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will
> be performed.
>
> https://gradle.com/s/x5zqbc5zwd3bg
>
> (Now I remember why I stopped using `build` :/)
>
> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:
>
>> It shouldn't stall. That's a bug.
>> OTOH, I never use the `build` target.
>> I'll try running that myself.
>>
>> On Mon, Mar 25, 2019, 07:24 Michael Luckey 
>> wrote:
>>
>>> Hi,
>>>
>>> trying to run './gradlew build' on vanilla setup, my build
>>> consistently stalls during execution of python gcp tests, e.g. on both 
>>> of
>>> - > :beam-sdks-python:testPy2Gcp
>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>>>
>>> Console output:
>>> 

Re: Build blocking on

[Udi Meiri]

Robert, from what I recall it's not flaky for me - it consistently fails.
Let me know if there's a way to get more logging about this error.

On Mon, Mar 25, 2019, 19:50 Robert Burke  wrote:

> It's concerning to me that 1) the Go dependency resolution via gogradle is
> flaky, and 2) that it can block other languages.
>
> I suppose 2) makes sense since it's part of the container bootstrapping
> code, but that makes 1) a serious problem, of which I wasn't aware.
> I should have time to investigate this in the next two weeks.
>
> On Mon, 25 Mar 2019 at 18:08, Michael Luckey  wrote:
>
>> Just for the record,
>>
>> using a vm here, because did not yet get all task running on my mac, and
>> did not want to mess with my setup.
>>
>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores
>> and further
>>
>> sudo apt update
>>
>> sudo apt install gcc
>>
>> sudo apt install make
>>
>> sudo apt install perl
>>
>> sudo apt install curl
>>
>> sudo apt install openjdk-8-jdk
>>
>> sudo apt install python
>>
>> sudo apt install -y software-properties-common
>>
>> sudo add-apt-repository ppa:deadsnakes/ppa
>>
>> sudo apt update
>>
>> sudo apt install python3.5
>>
>> sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent
>> software-properties-common
>>
>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key
>> add -
>>
>> sudo apt-key fingerprint 0EBFCD88
>>
>> sudo add-apt-repository "deb [arch=amd64]
>> https://download.docker.com/linux/ubuntu \
>>
>> $(lsb_release -cs) \
>>
>> stable"
>>
>> sudo apt-get update
>>
>> sudo apt-get install docker-ce docker-ce-cli containerd.io
>>
>> sudo groupadd docker
>>
>> sudo usermod -aG docker $USER
>>
>> git config --global user.email "d...@spam.me"
>>
>> git config --global user.name "Some Guy"
>>
>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
>>
>> sudo python get-pip.py
>>
>> rm get-pip.py
>>
>> sudo pip install --upgrade virtualenv
>>
>> sudo pip install cython
>>
>> sudo apt-get install python-dev
>>
>> sudo apt-get install python3-distutils
>>
>> sudo apt-get install python3-dev # for python3.x installs
>>
>>
>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew build
>>
>> Nothing else changed/added. (hopefully, need to reassure myself here)
>>
>> Unfortunately, this is failing. Need to exclude those python tests (and
>> of course website, which usually fails on lira links)
>>
>> So I might be missing some env settings for gap, dunno. Probably missed
>> some docs.
>>
>>
>>
>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey 
>> wrote:
>>
>>> Thanks Udi for trying that!
>>>
>>> In fact, the go dependency resolution is flaky. Did not look into that,
>>> but just rerunning usually works. Of course, less than optimal, but,
>>> well...
>>>
>>> Running build target is of course just an aggregation of task to run.
>>> And unfortunately just running that
>>>
>>> ./gradlew  :beam-sdks-python:testPy2Gcp
>>>
>>> stalls on my (virtual) machine.
>>>
>>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:
>>>
 Okay, `./gradlew build` failed pretty quickly for me:

 > Task :beam-sdks-go:resolveBuildDependencies FAILED
 cloud.google.com/go:
 commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[
 https://code.googlesource.com/gocloud] does not exist in
 /usr/local/google/home/ehudm/.gradle/go/repo/
 cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
 performed.

 https://gradle.com/s/x5zqbc5zwd3bg

 (Now I remember why I stopped using `build` :/)

 On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:

> It shouldn't stall. That's a bug.
> OTOH, I never use the `build` target.
> I'll try running that myself.
>
> On Mon, Mar 25, 2019, 07:24 Michael Luckey 
> wrote:
>
>> Hi,
>>
>> trying to run './gradlew build' on vanilla setup, my build
>> consistently stalls during execution of python gcp tests, e.g. on both of
>> - > :beam-sdks-python:testPy2Gcp
>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>>
>> Console output:
>>  snip 
>> test_big_query_standard_sql
>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>> ... SKIP: IT is skipped because --test-pipeline-options is not specified
>> test_big_query_standard_sql_kms_key
>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>> ... SKIP: This test requires BQ Dataflow native source support for KMS,
>> which is not available yet.
>> test_multiple_destinations_transform
>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... 
>> SKIP:
>> IT is skipped because --test-pipeline-options is not specified
>> test_one_job_fails_all_jobs_fail
>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... 
>> SKIP:
>> IT is skipped because --test-pipeline-options is not 

Re: Build blocking on

[Robert Burke]

It's concerning to me that 1) the Go dependency resolution via gogradle is
flaky, and 2) that it can block other languages.

I suppose 2) makes sense since it's part of the container bootstrapping
code, but that makes 1) a serious problem, of which I wasn't aware.
I should have time to investigate this in the next two weeks.

On Mon, 25 Mar 2019 at 18:08, Michael Luckey  wrote:

> Just for the record,
>
> using a vm here, because did not yet get all task running on my mac, and
> did not want to mess with my setup.
>
> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores
> and further
>
> sudo apt update
>
> sudo apt install gcc
>
> sudo apt install make
>
> sudo apt install perl
>
> sudo apt install curl
>
> sudo apt install openjdk-8-jdk
>
> sudo apt install python
>
> sudo apt install -y software-properties-common
>
> sudo add-apt-repository ppa:deadsnakes/ppa
>
> sudo apt update
>
> sudo apt install python3.5
>
> sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent
> software-properties-common
>
> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key
> add -
>
> sudo apt-key fingerprint 0EBFCD88
>
> sudo add-apt-repository "deb [arch=amd64]
> https://download.docker.com/linux/ubuntu \
>
> $(lsb_release -cs) \
>
> stable"
>
> sudo apt-get update
>
> sudo apt-get install docker-ce docker-ce-cli containerd.io
>
> sudo groupadd docker
>
> sudo usermod -aG docker $USER
>
> git config --global user.email "d...@spam.me"
>
> git config --global user.name "Some Guy"
>
> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
>
> sudo python get-pip.py
>
> rm get-pip.py
>
> sudo pip install --upgrade virtualenv
>
> sudo pip install cython
>
> sudo apt-get install python-dev
>
> sudo apt-get install python3-distutils
>
> sudo apt-get install python3-dev # for python3.x installs
>
>
> git clone https://github.com/apache/beam.git cd beam/ ./gradlew build
>
> Nothing else changed/added. (hopefully, need to reassure myself here)
>
> Unfortunately, this is failing. Need to exclude those python tests (and of
> course website, which usually fails on lira links)
>
> So I might be missing some env settings for gap, dunno. Probably missed
> some docs.
>
>
>
> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey 
> wrote:
>
>> Thanks Udi for trying that!
>>
>> In fact, the go dependency resolution is flaky. Did not look into that,
>> but just rerunning usually works. Of course, less than optimal, but,
>> well...
>>
>> Running build target is of course just an aggregation of task to run. And
>> unfortunately just running that
>>
>> ./gradlew  :beam-sdks-python:testPy2Gcp
>>
>> stalls on my (virtual) machine.
>>
>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:
>>
>>> Okay, `./gradlew build` failed pretty quickly for me:
>>>
>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED
>>> cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d',
>>> urls=[https://code.googlesource.com/gocloud] does not exist in
>>> /usr/local/google/home/ehudm/.gradle/go/repo/
>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
>>> performed.
>>>
>>> https://gradle.com/s/x5zqbc5zwd3bg
>>>
>>> (Now I remember why I stopped using `build` :/)
>>>
>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:
>>>
 It shouldn't stall. That's a bug.
 OTOH, I never use the `build` target.
 I'll try running that myself.

 On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:

> Hi,
>
> trying to run './gradlew build' on vanilla setup, my build
> consistently stalls during execution of python gcp tests, e.g. on both of
> - > :beam-sdks-python:testPy2Gcp
> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>
> Console output:
>  snip 
> test_big_query_standard_sql
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... SKIP: IT is skipped because --test-pipeline-options is not specified
> test_big_query_standard_sql_kms_key
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... SKIP: This test requires BQ Dataflow native source support for KMS,
> which is not available yet.
> test_multiple_destinations_transform
> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... 
> SKIP:
> IT is skipped because --test-pipeline-options is not specified
> test_one_job_fails_all_jobs_fail
> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... 
> SKIP:
> IT is skipped because --test-pipeline-options is not specified
> test_records_traverse_transform_with_mocks
> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...
>
> output ends here, would expect a failed or ok here.
>
>
> Afterwards no progress - even waiting for hours. Any idea, what might
> be causing this? Do I need to add some GCP properties for this task ?
>

Re: Build blocking on

[Michael Luckey]

Just for the record,

using a vm here, because did not yet get all task running on my mac, and
did not want to mess with my setup.

So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores and
further

sudo apt update

sudo apt install gcc

sudo apt install make

sudo apt install perl

sudo apt install curl

sudo apt install openjdk-8-jdk

sudo apt install python

sudo apt install -y software-properties-common

sudo add-apt-repository ppa:deadsnakes/ppa

sudo apt update

sudo apt install python3.5

sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent
software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo apt-key fingerprint 0EBFCD88

sudo add-apt-repository "deb [arch=amd64]
https://download.docker.com/linux/ubuntu \

$(lsb_release -cs) \

stable"

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io

sudo groupadd docker

sudo usermod -aG docker $USER

git config --global user.email "d...@spam.me"

git config --global user.name "Some Guy"

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

sudo python get-pip.py

rm get-pip.py

sudo pip install --upgrade virtualenv

sudo pip install cython

sudo apt-get install python-dev

sudo apt-get install python3-distutils

sudo apt-get install python3-dev # for python3.x installs


git clone https://github.com/apache/beam.git cd beam/ ./gradlew build

Nothing else changed/added. (hopefully, need to reassure myself here)

Unfortunately, this is failing. Need to exclude those python tests (and of
course website, which usually fails on lira links)

So I might be missing some env settings for gap, dunno. Probably missed
some docs.



On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey  wrote:

> Thanks Udi for trying that!
>
> In fact, the go dependency resolution is flaky. Did not look into that,
> but just rerunning usually works. Of course, less than optimal, but,
> well...
>
> Running build target is of course just an aggregation of task to run. And
> unfortunately just running that
>
> ./gradlew  :beam-sdks-python:testPy2Gcp
>
> stalls on my (virtual) machine.
>
> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:
>
>> Okay, `./gradlew build` failed pretty quickly for me:
>>
>> > Task :beam-sdks-go:resolveBuildDependencies FAILED
>> cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d',
>> urls=[https://code.googlesource.com/gocloud] does not exist in
>> /usr/local/google/home/ehudm/.gradle/go/repo/
>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
>> performed.
>>
>> https://gradle.com/s/x5zqbc5zwd3bg
>>
>> (Now I remember why I stopped using `build` :/)
>>
>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:
>>
>>> It shouldn't stall. That's a bug.
>>> OTOH, I never use the `build` target.
>>> I'll try running that myself.
>>>
>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:
>>>
 Hi,

 trying to run './gradlew build' on vanilla setup, my build consistently
 stalls during execution of python gcp tests, e.g. on both of
 - > :beam-sdks-python:testPy2Gcp
 - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp

 Console output:
  snip 
 test_big_query_standard_sql
 (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
 ... SKIP: IT is skipped because --test-pipeline-options is not specified
 test_big_query_standard_sql_kms_key
 (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
 ... SKIP: This test requires BQ Dataflow native source support for KMS,
 which is not available yet.
 test_multiple_destinations_transform
 (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
 IT is skipped because --test-pipeline-options is not specified
 test_one_job_fails_all_jobs_fail
 (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
 IT is skipped because --test-pipeline-options is not specified
 test_records_traverse_transform_with_mocks
 (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...

 output ends here, would expect a failed or ok here.


 Afterwards no progress - even waiting for hours. Any idea, what might
 be causing this? Do I need to add some GCP properties for this task ?

 Any ideas, what I am doing wrong?

 best,

 michel

Re: Build blocking on

[Michael Luckey]

Thanks Udi for trying that!

In fact, the go dependency resolution is flaky. Did not look into that, but
just rerunning usually works. Of course, less than optimal, but, well...

Running build target is of course just an aggregation of task to run. And
unfortunately just running that

./gradlew  :beam-sdks-python:testPy2Gcp

stalls on my (virtual) machine.

On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:

> Okay, `./gradlew build` failed pretty quickly for me:
>
> > Task :beam-sdks-go:resolveBuildDependencies FAILED
> cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d',
> urls=[https://code.googlesource.com/gocloud] does not exist in
> /usr/local/google/home/ehudm/.gradle/go/repo/
> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
> performed.
>
> https://gradle.com/s/x5zqbc5zwd3bg
>
> (Now I remember why I stopped using `build` :/)
>
> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:
>
>> It shouldn't stall. That's a bug.
>> OTOH, I never use the `build` target.
>> I'll try running that myself.
>>
>> On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:
>>
>>> Hi,
>>>
>>> trying to run './gradlew build' on vanilla setup, my build consistently
>>> stalls during execution of python gcp tests, e.g. on both of
>>> - > :beam-sdks-python:testPy2Gcp
>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>>>
>>> Console output:
>>>  snip 
>>> test_big_query_standard_sql
>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>>> ... SKIP: IT is skipped because --test-pipeline-options is not specified
>>> test_big_query_standard_sql_kms_key
>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>>> ... SKIP: This test requires BQ Dataflow native source support for KMS,
>>> which is not available yet.
>>> test_multiple_destinations_transform
>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
>>> IT is skipped because --test-pipeline-options is not specified
>>> test_one_job_fails_all_jobs_fail
>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
>>> IT is skipped because --test-pipeline-options is not specified
>>> test_records_traverse_transform_with_mocks
>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...
>>>
>>> output ends here, would expect a failed or ok here.
>>>
>>>
>>> Afterwards no progress - even waiting for hours. Any idea, what might be
>>> causing this? Do I need to add some GCP properties for this task ?
>>>
>>> Any ideas, what I am doing wrong?
>>>
>>> best,
>>>
>>> michel
>>>
>>>

Re: Build blocking on

[Udi Meiri]

It shouldn't stall. That's a bug.
OTOH, I never use the `build` target.
I'll try running that myself.

On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:

> Hi,
>
> trying to run './gradlew build' on vanilla setup, my build consistently
> stalls during execution of python gcp tests, e.g. on both of
> - > :beam-sdks-python:testPy2Gcp
> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>
> Console output:
>  snip 
> test_big_query_standard_sql
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... SKIP: IT is skipped because --test-pipeline-options is not specified
> test_big_query_standard_sql_kms_key
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... SKIP: This test requires BQ Dataflow native source support for KMS,
> which is not available yet.
> test_multiple_destinations_transform
> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
> IT is skipped because --test-pipeline-options is not specified
> test_one_job_fails_all_jobs_fail
> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
> IT is skipped because --test-pipeline-options is not specified
> test_records_traverse_transform_with_mocks
> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...
>
> output ends here, would expect a failed or ok here.
>
>
> Afterwards no progress - even waiting for hours. Any idea, what might be
> causing this? Do I need to add some GCP properties for this task ?
>
> Any ideas, what I am doing wrong?
>
> best,
>
> michel
>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Build blocking on

[Udi Meiri]

Okay, `./gradlew build` failed pretty quickly for me:

> Task :beam-sdks-go:resolveBuildDependencies FAILED
cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d',
urls=[https://code.googlesource.com/gocloud] does not exist in
/usr/local/google/home/ehudm/.gradle/go/repo/
cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
performed.

https://gradle.com/s/x5zqbc5zwd3bg

(Now I remember why I stopped using `build` :/)

On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:

> It shouldn't stall. That's a bug.
> OTOH, I never use the `build` target.
> I'll try running that myself.
>
> On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:
>
>> Hi,
>>
>> trying to run './gradlew build' on vanilla setup, my build consistently
>> stalls during execution of python gcp tests, e.g. on both of
>> - > :beam-sdks-python:testPy2Gcp
>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>>
>> Console output:
>>  snip 
>> test_big_query_standard_sql
>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>> ... SKIP: IT is skipped because --test-pipeline-options is not specified
>> test_big_query_standard_sql_kms_key
>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>> ... SKIP: This test requires BQ Dataflow native source support for KMS,
>> which is not available yet.
>> test_multiple_destinations_transform
>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
>> IT is skipped because --test-pipeline-options is not specified
>> test_one_job_fails_all_jobs_fail
>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
>> IT is skipped because --test-pipeline-options is not specified
>> test_records_traverse_transform_with_mocks
>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...
>>
>> output ends here, would expect a failed or ok here.
>>
>>
>> Afterwards no progress - even waiting for hours. Any idea, what might be
>> causing this? Do I need to add some GCP properties for this task ?
>>
>> Any ideas, what I am doing wrong?
>>
>> best,
>>
>> michel
>>
>>


smime.p7s
Description: S/MIME Cryptographic Signature