SerializableFunction composition operator

Does the Java API have a composition operator for SerializableFunction?  It seems like these two compositions in particular might tend to pop up from time to time: myFunction(KV::getValue(x)) myFunction(KV::getKey(x)) If there isn't already one, would there be interest in a contribution of

Re: low availability in the coming 4 weeks

Congratulations Mingmin! Take your time with your new baby/ Mingmin Xu 于2017年5月25日周四 上午11:33写道： > Hello everyone, > > I'll take 4 weeks off to take care of my new born baby. I'm very glad that > James Xu agrees to take my role in Beam SQL feature. > > Ps, I'll consolidate the

Re: connector for AWS Aurora

Thanks Lukasz and Sourabh. Yes I will use jdbcio. On May 24, 2017 1:51 PM, "Sourabh Bajaj" wrote: You can connect to AWS Aurora using JDBC so you might want to experiment using the JdbcIO and seeing if that fits your use case. -Sourabh On Wed, May 24, 2017 at

Re: Please add me into beam slack channel

Thanks. On Thu, May 25, 2017 at 8:37 AM, James wrote: > DONE. > > On Thu, May 25, 2017 at 10:57 AM, tarush grover > wrote: > > > Hi Team, > > > > I saw on the website that beam community has a slack channel, kindly add > me > > also into that

Re: Please add me into beam slack channel

DONE. On Thu, May 25, 2017 at 10:57 AM, tarush grover wrote: > Hi Team, > > I saw on the website that beam community has a slack channel, kindly add me > also into that channel for easy communication. > > Regards, > Tarush >

Please add me into beam slack channel

Hi Team, I saw on the website that beam community has a slack channel, kindly add me also into that channel for easy communication. Regards, Tarush

Re: Make runner implementation smoother?

Yes, we have a Slack channel. I've sent you an invite. Kenn On Wed, May 24, 2017 at 3:33 PM, Romain Manni-Bucau wrote: > Hi Kenneth, > > thanks a lot, this doc is really helpful. Will try to move a bit forward > in my tests when I'll get time (hopefully end of the week

Re: Make runner implementation smoother?

Hi Kenneth, thanks a lot, this doc is really helpful. Will try to move a bit forward in my tests when I'll get time (hopefully end of the week or next one). PS: saw some history on google but not sure what was the complete outcome: if there an IRC/slack/git-based chat for beam? Know ASF (infra)

Re: Make runner implementation smoother?

Hi Romain, These are great topics for discussion, and very good timing. I strongly recommend joining dev@beam, as that is where most discussion about implementing Beam runners takes place. On that list, I recently shared a guide for implementing a Beam runner at

Re: Dynamic file-based sinks

Hmm, on one hand this looks syntactically very appealing, on the other hand, it's icky to have a function return a PTransform at runtime, only to have some information be immediately extracted from that transform. Moreover, not all TextIO.Write transforms will be legal to return - e.g. most likely

Re: connector for AWS Aurora

We have a list of built/in-progress IOs here: https://github.com/apache/beam-site/blob/asf-site/src/ documentation/io/built-in.md Aurora doesn't seem to be listed. Please add a JIRA representing the issue and update the built-in.md linking the inprogress development. On Wed, May 24, 2017 at

connector for AWS Aurora

Hello, We are planning to develop connector for AWS Aurora. Please, let us know if anybody is working on it. Madhu Borkar

Re: [New Proposal] Hive connector using native api

one comment I had that I realized was worth bringing back to the mailing list: The Write transform here does batching using startBundle/finishBundle, but I suspect it'd be better to use the GroupIntoBatches transform before doing the actual write. I *think* our general guidance in the future

Re: Dynamic file-based sinks

Did you see that I modified the second proposal so that users can map DestinationT to the actual PTransform (i.e. DestinationT->TextIO or DestinationT->AvroIO). This means that users do not have to deal with FileBasedSink or even know it exists. I prefer the second approach for two reason: 1. It

Re: Dynamic file-based sinks

I commented a little in the doc I want to reply on list because this is a really great feature. The two alternatives, as I understand them, both include mapping your elements to an intermediate DestinationT that you can group by before writing. Then the big picture decision is whether to map each

RE: [New Proposal] Hive connector using native api

Thanks all for your review and comments. I will raise a PR against https://issues.apache.org/jira/browse/BEAM-2357 Regards, Seshadri 408 601 7548 -Original Message- From: Ismaël Mejía [mailto:ieme...@gmail.com] Sent: Wednesday, May 24, 2017 2:57 AM To: dev@beam.apache.org Subject: Re:

Re: [New Proposal] Hive connector using native api

Hello, I created a new JIRA for this native implementation of the IO so feel free to PR the 'native' implementation using this ticket. https://issues.apache.org/jira/browse/BEAM-2357 We will discuss all the small details in the PR. The old JIRA (BEAM-1158) will still be there just to add the

Jenkins build is back to stable : beam_Release_NightlySnapshot #425

See

Re: [New Proposal] Hive connector using native api

Hi, It looks good. I just saw some issues: - javadoc is not correct in HiveIO (it says write() for read ;)). - estimated size is global to the table (doesn't consider the filter). It's not a big deal, but it should be documented. - you don't use the desired bundle size provided by the runner