Re: Build failing on the Apex runner

2017-04-08 Thread Thomas Weise
I'm not sure what the issue is, but that commit shouldn't affect the test listed and the Apex tests pass when I run them. Thomas On Sat, Apr 8, 2017 at 11:15 PM, Jean-Baptiste Onofré wrote: > Hii huys, > > FYI, the build is failing on the Apex runner tests (due to > NullPointerException): > >

Build failing on the Apex runner

2017-04-08 Thread Jean-Baptiste Onofré
Hii huys, FYI, the build is failing on the Apex runner tests (due to NullPointerException): ParDoTranslatorTest.testSerialization:225 » NullPointer ApexStateInternalsTest.initStateInternals:83 » NullPointer ApexStateInternalsTest.initStateInternals:83 » NullPointer ApexStateInternalsTes

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

2017-04-08 Thread Thomas Weise
Nice work Aljoscha! Update WRT ApexRunner: We merged some prep work in the ParDoOperator to weed out remnants of OldDoFn. I have almost all the changes ready to add the support for Splittable DoFn (for most part those follow the Flink runner changes). The final piece missing to support the feature

Re: [PROPOSAL] Standard IO Metrics

2017-04-08 Thread Jean-Baptiste Onofré
Thanks Aviem, let me take a look on the document. Regards JB On 04/08/2017 09:04 PM, Aviem Zur wrote: Hi all, We are currently in the process of introducing IO metrics to Beam. Questions have been raised as to what the metrics names should be, and if they should be standard across different

[PROPOSAL] Standard IO Metrics

2017-04-08 Thread Aviem Zur
Hi all, We are currently in the process of introducing IO metrics to Beam. Questions have been raised as to what the metrics names should be, and if they should be standard across different IOs. I've written this up as a proposal found here: https://s.apache.org/standard-io-metrics As usual, th

Re: [DISCUSSION] PAssert success/failure count validation for all runners

2017-04-08 Thread Aljoscha Krettek
@kenn What’s the design you’re mentioning? (I probably missed it because I’m not completely up to data on the Jiras and ML because of Flink Forward preparations) > On 7. Apr 2017, at 12:42, Kenneth Knowles wrote: > > We also have a design that improves the signal even without metrics, so I'm >

Re: Proposed Splittable DoFn API changes

2017-04-08 Thread Aljoscha Krettek
+1 I was too busy with traveling and preparations for Flink Forward but I wanted to retroactively confirm that these are good changes. :-) > On 7. Apr 2017, at 22:43, Jean-Baptiste Onofré wrote: > > Hi Eugene, > > thanks for the update and nice example. > > I plan to start to refactor/experim

Re: IO ITs: Hosting Docker images

2017-04-08 Thread Ted Yu
+1 > On Apr 7, 2017, at 10:46 PM, Jean-Baptiste Onofré wrote: > > Hi Stephen, > > I think we should go to 1 and 4: > > 1. Try to use existing images providing what we need. If we don't find > existing image, we can always ask and help other community to provide so. > 4. If we don't find a sui