subject:"Dependencies and Datastore"

Re: Dependencies and Datastore

2018-01-31 Thread Jacob Marble

Thanks for that, Joshua.

Late last night I moved my BigTable-related code (depending on
beam:beam-sdks-java-io-google-cloud-platform) into a separate pom/project
from GCS-related code (depending on google-cloud-storage).

pom A:

+- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
|  +- com.google.api:gax-grpc:jar:0.20.0:compile
|  |  +- io.grpc:grpc-protobuf:jar:1.2.0:compile
|  |  +- com.google.api:api-common:jar:1.1.0:compile
|  |  +- com.google.api:gax:jar:1.3.1:compile
|  |  \- org.threeten:threetenbp:jar:1.3.3:compile

pom B:

|  +- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
|  \-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
| +- com.google.api:gax-grpc:jar:0.20.0:compile
| |  +- io.grpc:grpc-protobuf:jar:1.2.0:compile
| |  \- org.threeten:threetenbp:jar:1.3.3:compile
...
+- com.google.cloud:google-cloud-storage:jar:1.4.0:compile
|  +- com.google.cloud:google-cloud-core:jar:1.4.0:compile
|  |  +- org.json:json:jar:20160810:compile
|  |  +- com.google.api:api-common:jar:1.1.0:compile
|  |  +- com.google.api:gax:jar:1.5.0:compile

Jacob

On Wed, Jan 31, 2018 at 3:24 AM, Joshua Fox  wrote:

> The problem does not occur for me now following a dependency update.
>
> Here are the parts of mvn dependency:tree that involve gax
>
>  |  +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:
> jar:2.2.0:compile
>  |  |  +- com.google.api:gax-grpc:jar:0.20.0:compile
>
> ...
>
>  +- com.google.code.gson:gson:jar:2.8.2:compile
>  \- com.google.api:gax:jar:1.15.0:compile
>
> Those that involve Beam
>
>  +- org.apache.beam:beam-runners-google-cloud-dataflow-java:
> jar:2.2.0:compile
>  |  +- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
>  |  +- org.apache.beam:beam-sdks-java-extensions-google-cloud-
> platform-core:jar:2.2.0:compile
>  |  +- org.apache.beam:beam-sdks-common-runner-api:jar:2.2.0:compile
> ...
>
>  |  +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:
> jar:2.2.0:compile
>  |  |  +- com.google.api:gax-grpc:jar:0.20.0:compile
>
>
> Here are the parts of my pom that involve gax and Beam
>
>
> 
> org.apache.beam
> beam-runners-google-cloud-dataflow-java
> 2.2.0
> 
>
>
> 
> com.google.api
> gax
> 1.15.0
> 
>
>
>
>
> On Wed, Jan 31, 2018 at 2:55 AM, Jacob Marble  wrote:
>
>> Josh, what did you do to work around this?
>>
>> This suddenly crept up on a production pipeline yesterday, without
>> anything changing on our side (we do rebuild at every run).
>>
>> Jacob
>>
>> On Fri, Dec 8, 2017 at 6:46 PM, Chamikara Jayalath 
>> wrote:
>>
>>> Created https://issues.apache.org/jira/browse/BEAM-3321 to update
>>> the gax-grpc dependency of Beam.
>>>
>>> - Cham
>>>
>>> On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:

 +chamikara@


 On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:
>
> I use Cloud Datastore API to check for Kinds in the Datastore, then
> use Dataflow -- now upgrading to Beam -- to copy one Datastore to 
> another..
>
> After adding beam-sdks-java-io-google-cloud-platform to my pom, I
> start getting this when initializing the Cloud Datastore API
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/api/gax/retrying/ResultRetryAlgorithm
> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
> Factory.create(DatastoreOptions.java:51)
> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
> Factory.create(DatastoreOptions.java:45)
> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>
> It is caused by  gax dependencies.
>
> Specifically, before I add beam-sdks-java-io-google-cloud-platform there
> is this  gax dependency
>
> +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
> |  +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
> |  |  +- com.google.api:gax:jar:1.15.0:compile
>
>
>
> and after I add  it there is this
>
> +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:
> 2.2.0:compile
> |  +- com.google.api:gax-grpc:jar:0.20.0:compile
> |  |  +- com.google.api:gax:jar:1.3.1:compile
>
>
> If I add gax 1.15.0 to my pom explicitly, I get
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/api/gax/retrying/ExceptionRetryAlgorithm
> at java.lang.ClassLoader.defineClass1(Native Method)
> ..
> at com.google.cloud.BaseService.(BaseService.java:48)
> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
> Factory.create(DatastoreOptions.java:51)
> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
> Factory.create(DatastoreOptions.java:45)
> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>
> Clearly Datastore and Beam should work together. Yet there have been
> dependency problems between th

Re: Dependencies and Datastore

2018-01-31 Thread Joshua Fox

The problem does not occur for me now following a dependency update.

Here are the parts of mvn dependency:tree that involve gax

 |  +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
 |  |  +- com.google.api:gax-grpc:jar:0.20.0:compile

...

 +- com.google.code.gson:gson:jar:2.8.2:compile
 \- com.google.api:gax:jar:1.15.0:compile

Those that involve Beam

 +-
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.2.0:compile
 |  +- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
 |  +-
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.2.0:compile
 |  +- org.apache.beam:beam-sdks-common-runner-api:jar:2.2.0:compile
...

 |  +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
 |  |  +- com.google.api:gax-grpc:jar:0.20.0:compile


Here are the parts of my pom that involve gax and Beam



org.apache.beam
beam-runners-google-cloud-dataflow-java
2.2.0




com.google.api
gax
1.15.0





On Wed, Jan 31, 2018 at 2:55 AM, Jacob Marble  wrote:

> Josh, what did you do to work around this?
>
> This suddenly crept up on a production pipeline yesterday, without
> anything changing on our side (we do rebuild at every run).
>
> Jacob
>
> On Fri, Dec 8, 2017 at 6:46 PM, Chamikara Jayalath 
> wrote:
>
>> Created https://issues.apache.org/jira/browse/BEAM-3321 to update
>> the gax-grpc dependency of Beam.
>>
>> - Cham
>>
>> On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:
>>>
>>> +chamikara@
>>>
>>>
>>> On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:

 I use Cloud Datastore API to check for Kinds in the Datastore, then
 use Dataflow -- now upgrading to Beam -- to copy one Datastore to another..

 After adding beam-sdks-java-io-google-cloud-platform to my pom, I
 start getting this when initializing the Cloud Datastore API

 Exception in thread "main" java.lang.NoClassDefFoundError:
 com/google/api/gax/retrying/ResultRetryAlgorithm
 at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
 Factory.create(DatastoreOptions.java:51)
 at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
 Factory.create(DatastoreOptions.java:45)
 at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)

 It is caused by  gax dependencies.

 Specifically, before I add beam-sdks-java-io-google-cloud-platform there
 is this  gax dependency

 +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
 |  +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
 |  |  +- com.google.api:gax:jar:1.15.0:compile



 and after I add  it there is this

 +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:
 2.2.0:compile
 |  +- com.google.api:gax-grpc:jar:0.20.0:compile
 |  |  +- com.google.api:gax:jar:1.3.1:compile


 If I add gax 1.15.0 to my pom explicitly, I get

 Exception in thread "main" java.lang.NoClassDefFoundError:
 com/google/api/gax/retrying/ExceptionRetryAlgorithm
 at java.lang.ClassLoader.defineClass1(Native Method)
 ..
 at com.google.cloud.BaseService.(BaseService.java:48)
 at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
 Factory.create(DatastoreOptions.java:51)
 at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
 Factory.create(DatastoreOptions.java:45)
 at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)

 Clearly Datastore and Beam should work together. Yet there have been
 dependency problems between the two for  a while. See this discussion
  from 1 year ago.

 How can I resolve this?

>>>
>


-- 


*JOSHUA FOX*
Principal Software Architect | Freightos



*T (Israel): *+972-545691165 | *T (US)*:  +1-3123400953
Smooth shipping.

Re: Dependencies and Datastore

2018-01-30 Thread Jacob Marble

Also weird, things work fine (no need to force gax to any version other
than 1.3.1) when I build and run from my workstation, but fail on our CI/CD
worker.

Jacob

On Tue, Jan 30, 2018 at 4:55 PM, Jacob Marble  wrote:

> Josh, what did you do to work around this?
>
> This suddenly crept up on a production pipeline yesterday, without
> anything changing on our side (we do rebuild at every run).
>
> Jacob
>
> On Fri, Dec 8, 2017 at 6:46 PM, Chamikara Jayalath 
> wrote:
>
>> Created https://issues.apache.org/jira/browse/BEAM-3321 to update
>> the gax-grpc dependency of Beam.
>>
>> - Cham
>>
>> On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:
>>>
>>> +chamikara@
>>>
>>>
>>> On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:

 I use Cloud Datastore API to check for Kinds in the Datastore, then
 use Dataflow -- now upgrading to Beam -- to copy one Datastore to another..

 After adding beam-sdks-java-io-google-cloud-platform to my pom, I
 start getting this when initializing the Cloud Datastore API

 Exception in thread "main" java.lang.NoClassDefFoundError:
 com/google/api/gax/retrying/ResultRetryAlgorithm
 at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
 Factory.create(DatastoreOptions.java:51)
 at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
 Factory.create(DatastoreOptions.java:45)
 at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)

 It is caused by  gax dependencies.

 Specifically, before I add beam-sdks-java-io-google-cloud-platform there
 is this  gax dependency

 +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
 |  +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
 |  |  +- com.google.api:gax:jar:1.15.0:compile

 and after I add  it there is this

 +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:
 2.2.0:compile
 |  +- com.google.api:gax-grpc:jar:0.20.0:compile
 |  |  +- com.google.api:gax:jar:1.3.1:compile

 If I add gax 1.15.0 to my pom explicitly, I get

 Exception in thread "main" java.lang.NoClassDefFoundError:
 com/google/api/gax/retrying/ExceptionRetryAlgorithm
 at java.lang.ClassLoader.defineClass1(Native Method)
 ..
 at com.google.cloud.BaseService.(BaseService.java:48)
 at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
 Factory.create(DatastoreOptions.java:51)
 at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
 Factory.create(DatastoreOptions.java:45)
 at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)

 Clearly Datastore and Beam should work together. Yet there have been
 dependency problems between the two for  a while. See this discussion
  from 1 year ago.

 How can I resolve this?

>>>
>

Re: Dependencies and Datastore

2018-01-30 Thread Jacob Marble

Josh, what did you do to work around this?

This suddenly crept up on a production pipeline yesterday, without anything
changing on our side (we do rebuild at every run).

Jacob

On Fri, Dec 8, 2017 at 6:46 PM, Chamikara Jayalath 
wrote:

> Created https://issues.apache.org/jira/browse/BEAM-3321 to update
> the gax-grpc dependency of Beam.
>
> - Cham
>
> On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:
>>
>> +chamikara@
>>
>>
>> On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:
>>>
>>> I use Cloud Datastore API to check for Kinds in the Datastore, then  use
>>> Dataflow -- now upgrading to Beam -- to copy one Datastore to another..
>>>
>>> After adding beam-sdks-java-io-google-cloud-platform to my pom, I start
>>> getting this when initializing the Cloud Datastore API
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> com/google/api/gax/retrying/ResultRetryAlgorithm
>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>> Factory.create(DatastoreOptions.java:51)
>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>> Factory.create(DatastoreOptions.java:45)
>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>
>>> It is caused by  gax dependencies.
>>>
>>> Specifically, before I add beam-sdks-java-io-google-cloud-platform there
>>> is this  gax dependency
>>>
>>> +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
>>> |  +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
>>> |  |  +- com.google.api:gax:jar:1.15.0:compile
>>>
>>>
>>>
>>> and after I add  it there is this
>>>
>>> +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:
>>> 2.2.0:compile
>>> |  +- com.google.api:gax-grpc:jar:0.20.0:compile
>>> |  |  +- com.google.api:gax:jar:1.3.1:compile
>>>
>>>
>>> If I add gax 1.15.0 to my pom explicitly, I get
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> com/google/api/gax/retrying/ExceptionRetryAlgorithm
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> ..
>>> at com.google.cloud.BaseService.(BaseService.java:48)
>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>> Factory.create(DatastoreOptions.java:51)
>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>> Factory.create(DatastoreOptions.java:45)
>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>
>>> Clearly Datastore and Beam should work together. Yet there have been
>>> dependency problems between the two for  a while. See this discussion
>>>  from 1 year ago.
>>>
>>> How can I resolve this?
>>>
>>

Re: Dependencies and Datastore

2017-12-08 Thread Chamikara Jayalath

Created https://issues.apache.org/jira/browse/BEAM-3321 to update 
the gax-grpc dependency of Beam.

- Cham

On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:
>
> +chamikara@
>
> On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:
>>
>> I use Cloud Datastore API to check for Kinds in the Datastore, then  use 
>> Dataflow -- now upgrading to Beam -- to copy one Datastore to another..
>>
>> After adding beam-sdks-java-io-google-cloud-platform to my pom, I start 
>> getting this when initializing the Cloud Datastore API
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError: 
>> com/google/api/gax/retrying/ResultRetryAlgorithm
>> at 
>> com.google.cloud.datastore.DatastoreOptions$DefaultDatastoreFactory.create(DatastoreOptions.java:51)
>> at 
>> com.google.cloud.datastore.DatastoreOptions$DefaultDatastoreFactory.create(DatastoreOptions.java:45)
>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>
>> It is caused by  gax dependencies.
>>
>> Specifically, before I add beam-sdks-java-io-google-cloud-platform there 
>> is this  gax dependency 
>>
>> +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
>> |  +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
>> |  |  +- com.google.api:gax:jar:1.15.0:compile
>>
>>
>>
>> and after I add  it there is this
>>
>> +- 
>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
>> |  +- com.google.api:gax-grpc:jar:0.20.0:compile
>> |  |  +- com.google.api:gax:jar:1.3.1:compile
>>
>>
>> If I add gax 1.15.0 to my pom explicitly, I get
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError: 
>> com/google/api/gax/retrying/ExceptionRetryAlgorithm
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> .. 
>> at com.google.cloud.BaseService.(BaseService.java:48)
>> at 
>> com.google.cloud.datastore.DatastoreOptions$DefaultDatastoreFactory.create(DatastoreOptions.java:51)
>> at 
>> com.google.cloud.datastore.DatastoreOptions$DefaultDatastoreFactory.create(DatastoreOptions.java:45)
>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>
>> Clearly Datastore and Beam should work together. Yet there have been 
>> dependency problems between the two for  a while. See this discussion 
>>  from 1 year ago.
>>
>> How can I resolve this?
>>
>

Dependencies and Datastore

2017-12-08 Thread Joshua Fox

I use Cloud Datastore API to check for Kinds in the Datastore, then  use
Dataflow -- now upgrading to Beam -- to copy one Datastore to another..

After adding beam-sdks-java-io-google-cloud-platform to my pom, I start
getting this when initializing the Cloud Datastore API

Exception in thread "main" java.lang.NoClassDefFoundError:
com/google/api/gax/retrying/ResultRetryAlgorithm
at com.google.cloud.datastore.DatastoreOptions$DefaultDatastoreFactory.
create(DatastoreOptions.java:51)
at com.google.cloud.datastore.DatastoreOptions$DefaultDatastoreFactory.
create(DatastoreOptions.java:45)
at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)

It is caused by  gax dependencies.

Specifically, before I add beam-sdks-java-io-google-cloud-platform there is
this  gax dependency

+- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
|  +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
|  |  +- com.google.api:gax:jar:1.15.0:compile



and after I add  it there is this

+- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
|  +- com.google.api:gax-grpc:jar:0.20.0:compile
|  |  +- com.google.api:gax:jar:1.3.1:compile


If I add gax 1.15.0 to my pom explicitly, I get

Exception in thread "main" java.lang.NoClassDefFoundError:
com/google/api/gax/retrying/ExceptionRetryAlgorithm
at java.lang.ClassLoader.defineClass1(Native Method)
..
at com.google.cloud.BaseService.(BaseService.java:48)
at
com.google.cloud.datastore.DatastoreOptions$DefaultDatastoreFactory.create(DatastoreOptions.java:51)
at
com.google.cloud.datastore.DatastoreOptions$DefaultDatastoreFactory.create(DatastoreOptions.java:45)
at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)

Clearly Datastore and Beam should work together. Yet there have been
dependency problems between the two for  a while. See this discussion
 from 1 year ago.

How can I resolve this?

Re: Dependencies and Datastore

Re: Dependencies and Datastore

Re: Dependencies and Datastore

Re: Dependencies and Datastore

Re: Dependencies and Datastore

Dependencies and Datastore

6 matches

Site Navigation

Mail list logo

Footer information