Re: splitIntoBundles vs. generateInitialSplits

Indeed, take a look at https://issues.apache.org/jira/browse/BEAM-1272. On Tue, Mar 21, 2017 at 8:20 AM Jean-Baptiste Onofré wrote: > It makes sense. > > Regards > JB > > On 03/20/2017 11:14 PM, Ismaël Mejía wrote: > > This is an forgotten one, Stas did you create a JIRA about this one? I > > th

Re: why Source#validate() is not declared to throw any exception

I think it would make sense to allow the validate method to throw Exception. On Mon, Mar 20, 2017, 11:21 PM Jean-Baptiste Onofré wrote: > Hi Ted, > > validate() is supposed to throw runtime exception (IllegalStateException, > RuntimeException, ...) to "traverse" the executor. > > Regards > JB >

Re: why Source#validate() is not declared to throw any exception

Hi Ted, validate() is supposed to throw runtime exception (IllegalStateException, RuntimeException, ...) to "traverse" the executor. Regards JB On 03/21/2017 01:56 AM, Ted Yu wrote: Hi, I was reading HDFSFileSource.java where: @Override public void validate() { ... } catch (IOExce

Re: splitIntoBundles vs. generateInitialSplits

It makes sense. Regards JB On 03/20/2017 11:14 PM, Ismaël Mejía wrote: This is an forgotten one, Stas did you create a JIRA about this one? I think this change should be also tagged as First version release, because this is an API change and can break stuff if we do it later on. On Wed, Jan 11

why Source#validate() is not declared to throw any exception

Hi, I was reading HDFSFileSource.java where: @Override public void validate() { ... } catch (IOException | InterruptedException e) { throw new RuntimeException(e); } Why is validate() not declared to throw any exception ? If validation doesn't pass, there is nothing to cle

Re: Docker image dependencies

I have somehow forgotten this one. > Basically - I'm trying to keep number of tools at a minimum while still > providing good support for the functionality we need. Does docker-compose > provide something beyond the functionality that k8 does? I'm not familiar > with docker-compose, but looking at

Re: [ANNOUNCEMENT] New committers, March 2017 edition!

Thanks everyone, Feels great to be part of the team. Congratulations to the other new committers ! -Ismaël On Mon, Mar 20, 2017 at 2:50 PM, Tyler Akidau wrote: > Welcome! > > On Mon, Mar 20, 2017, 02:25 Jean-Baptiste Onofré wrote: > >> Welcome aboard, and congrats ! >> >> Really happy to count

Re: splitIntoBundles vs. generateInitialSplits

This is an forgotten one, Stas did you create a JIRA about this one? I think this change should be also tagged as First version release, because this is an API change and can break stuff if we do it later on. On Wed, Jan 11, 2017 at 4:30 PM, Jean-Baptiste Onofré wrote: > Hi Eugene and Stas, > > J

Re: Beam File System in the Python SDK

Thanks for the feedback Tibor. I think in the first iteration we can probably assume that the underlying filesystem is taking care of permission enforcement. Once we have a few sources using the FS API we can may be revisit this as we'll surely learn a few things from that. Thoughts ? On Sun, Mar

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Meeting notes: Me and Thomas had a video call and we pretty much walked through the implementation of SDF in the runner-agnostic part and in the direct runner. Flink and Apex are pretty similar, so likely https://github.com/apache/beam/pull/2235 (the Flink PR) will give a very good guideline as to

RE: user beam in pre-spark cluster have some proplems

Hi Today, I user Eclipse debug code,When I submit to spark cluster,After a moment,I see some error message,It as below: 2017-03-20 16:37:44 ERROR org.apache.spark.Logging$class.logError(Logging.scala:74) task-result-getter-3 [Task 31.0 in stage 109.0 (TID 848) had a not serializable result: org

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Thomas - yes, 9:30 works, shall we do that? JB - excellent! You can start experimenting already, using direct runner! On Mon, Mar 20, 2017, 2:26 AM Jean-Baptiste Onofré wrote: > Hi Eugene, > > Thanks for the meeting notes ! > > I will be in the next call and Ismaël also provided to me some upda

Re: [ANNOUNCEMENT] New committers, March 2017 edition!

Welcome! On Mon, Mar 20, 2017, 02:25 Jean-Baptiste Onofré wrote: > Welcome aboard, and congrats ! > > Really happy to count you all in the team ;) > > Regards > JB > > On 03/17/2017 10:13 PM, Davor Bonaci wrote: > > Please join me and the rest of Beam PMC in welcoming the following > > contribut

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Eugene, Thanks for the meeting notes ! I will be in the next call and Ismaël also provided to me some updates. I will sync with Amit on Spark runner and start to experiment and test SDF on the JMS IO. Thanks ! Regards JB On 03/17/2017 04:36 PM, Eugene Kirpichov wrote: Meeting notes from

Re: [ANNOUNCEMENT] New committers, March 2017 edition!

Welcome aboard, and congrats ! Really happy to count you all in the team ;) Regards JB On 03/17/2017 10:13 PM, Davor Bonaci wrote: Please join me and the rest of Beam PMC in welcoming the following contributors as our newest committers. They have significantly contributed to the project in dif

Build failed in Jenkins: beam_Release_NightlySnapshot #363

See Changes: [amitsela33] [BEAM-1752, BEAM-1582] execute tests that recover from checkpoint in -- [...truncated 1.16 MB...] 2017-03-20T07:19:33.101 [INFO] 2017