Re: CassandraIO breakage

2019-04-18 Thread Alexey Romanenko
How it fails? No issue for me with local build against master. > On 17 Apr 2019, at 21:48, Reuven Lax wrote: > > Did something break with CassandraIO? It no longer seems to compile.

Re: Go SDK status

2019-04-18 Thread Thomas Weise
Hi Robert, Thanks a bunch for providing this comprehensive update. This is exactly the kind of perspective I was looking for, even when overall it means that for potential users of the Go SDK it is even sooner than what I might have hoped for. For more context, my interest was primarily on the st

Re: CassandraIO breakage

2019-04-18 Thread Jean-Baptiste Onofré
Let me check if it works on my machine. Regards JB On 17/04/2019 21:48, Reuven Lax wrote: > Did something break with CassandraIO? It no longer seems to compile. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: CassandraIO breakage

2019-04-18 Thread Jean-Baptiste Onofré
It builds fine on my machine. Let me check on Jenkins. Regards JB On 17/04/2019 21:48, Reuven Lax wrote: > Did something break with CassandraIO? It no longer seems to compile. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: CassandraIO breakage

2019-04-18 Thread Reuven Lax
Interesting. Let me try a full rebuild, maybe some dependency is not getting rebuilt. I was getting errors like this: /Users/relax/beam/sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraServiceImpl.java:74: error: incompatible types: ValueProvider> cannot be converted

Re: SNAPSHOTS have not been updated since february

2019-04-18 Thread Ismaël Mejía
Any progress on this? On Wed, Mar 27, 2019 at 5:38 AM Daniel Oliveira wrote: > > I made a bug for this specific issue (artifacts not publishing to the Apache > Maven repo): https://issues.apache.org/jira/browse/BEAM-6919 > > While I was gathering info for the bug report I also noticed +Yifan Zou

Re: SNAPSHOTS have not been updated since february

2019-04-18 Thread Ismaël Mejía
And is there a way we can detect SNAPSHOTS not been published daily in the future? On Thu, Apr 18, 2019 at 6:37 PM Ismaël Mejía wrote: > > Any progress on this? > > On Wed, Mar 27, 2019 at 5:38 AM Daniel Oliveira > wrote: > > > > I made a bug for this specific issue (artifacts not publishing to

Re: SNAPSHOTS have not been updated since february

2019-04-18 Thread Boyuan Zhang
There is a test target https://builds.apache.org/job/beam_Release_NightlySnapshot/ in beam, which builds and pushes snapshot to maven every day. Current failure is like, the jenkin machine cannot publish artifacts into maven owing to some weird permission issue. I think +Yifan Zou is working on i

Re: SNAPSHOTS have not been updated since february

2019-04-18 Thread Lukasz Cwik
The permissions issue is that the credentials needed to publish to the maven repository are only deployed on machines managed by Apache Infra. Now that the machines have been given back to each project to manage Yifan was investigating some other way to get the permissions on to the machine. On Th

Re: SNAPSHOTS have not been updated since february

2019-04-18 Thread Yifan Zou
The origin build nodes were updated in Jan 24 and the nexus credentials were removed from the filesystem because they are not supposed to be on external build nodes (nodes Infra does not own). We now need to set up the role account on the new Beam JNLP nodes. I am still contacting Infra to bring th

Artifact staging in cross-language pipelines

2019-04-18 Thread Maximilian Michels
Hi everyone, We have previously merged support for configuring transforms across languages. Please see Cham's summary on the discussion [1]. There is also a design document [2]. Subsequently, we've added wrappers for cross-language transforms to the Python SDK, i.e. GenerateSequence, ReadFro

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Lukasz Cwik
We can expose the artifact staging endpoint and artifact token to allow the expansion service to upload any resources its environment may need. For example, the expansion service for the Beam Java SDK would be able to upload jars. In the "docker" environment, the Apache Beam Java SDK harness conta

Hazelcast Jet Runner

2019-04-18 Thread Jozsef Bartok
Hi. We at Hazelcast Jet have been working for a while now to implement a Java Beam Runner (non-portable) based on Hazelcast Jet ( https://jet.hazelcast.org/). The process is still ongoing ( https://github.com/hazelcast/hazelcast-jet-beam-runner), but we are aiming for a fully functional, reliable R

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Chamikara Jayalath
I think there are two kind of dependencies we have to consider. (1) Dependencies that are needed to expand the transform. These have to be provided when we start the expansion service so that available external transforms are correctly registered with the expansion service. (2) Dependencies that

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Lukasz Cwik
Note that Max did ask whether making the expansion service do the staging made sense, and my first line was agreeing with that direction and expanding on how it could be done (so this is really Max's idea or from whomever he got the idea from). I believe a lot of the value of the expansion service

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Chamikara Jayalath
On Thu, Apr 18, 2019 at 2:12 PM Lukasz Cwik wrote: > Note that Max did ask whether making the expansion service do the staging > made sense, and my first line was agreeing with that direction and > expanding on how it could be done (so this is really Max's idea or from > whomever he got the idea

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Ankur Goenka
I agree that the Expansion service knows about the artifacts required for a cross language transform and having a prepackage folder/Zip for transforms based on language makes sense. One think to note here is that expansion service might not have the same access privilege as the pipeline author and

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Lukasz Cwik
I can understand the concern about credentials, the same access concern will exist for several cross language transforms (mostly IOs) since some will need access to credentials to read/write to an external service. Are there any ideas on how credential propagation could work to these IOs? Can we u

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Chamikara Jayalath
Thanks for raising the concern about credentials Ankur, I agree that this is a significant issue. On Thu, Apr 18, 2019 at 4:23 PM Lukasz Cwik wrote: > I can understand the concern about credentials, the same access concern > will exist for several cross language transforms (mostly IOs) since som

investigating python precommit wordcount_it failure

2019-04-18 Thread Udi Meiri
in https://issues.apache.org/jira/browse/BEAM-7111 If anyone has state please lmk smime.p7s Description: S/MIME Cryptographic Signature

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Chamikara Jayalath
On Thu, Apr 18, 2019 at 5:21 PM Chamikara Jayalath wrote: > Thanks for raising the concern about credentials Ankur, I agree that this > is a significant issue. > > On Thu, Apr 18, 2019 at 4:23 PM Lukasz Cwik wrote: > >> I can understand the concern about credentials, the same access concern >> w

Re: investigating python precommit wordcount_it failure

2019-04-18 Thread Udi Meiri
Correction: it's a postcommit failure On Thu, Apr 18, 2019 at 5:43 PM Udi Meiri wrote: > in https://issues.apache.org/jira/browse/BEAM-7111 > > If anyone has state please lmk > smime.p7s Description: S/MIME Cryptographic Signature

Re: investigating python precommit wordcount_it failure

2019-04-18 Thread Valentyn Tymofieiev
I am working on a postcommit worcount it failure in BEAM-7063. On Thu, Apr 18, 2019 at 6:05 PM Udi Meiri wrote: > Correction: it's a postcommit failure > > On Thu, Apr 18, 2019 at 5:43 PM Udi Meiri wrote: > >> in https://issues.apache.org/jira/browse/BEAM-7111 >> >> If anyone has state please l

Re: Artifact staging in cross-language pipelines

2019-04-18 Thread Thomas Weise
Good discussion :) Initially the expansion service was considered a user responsibility, but I think that isn't necessarily the case. I can also see the expansion service provided as part of the infrastructure and the user not wanting to deal with it at all. For example, users may want to write Py