Re: A 15x speed-up in local Python DirectRunner execution

Yes it does work for Java pipelines, modulo https://github.com/apache/beam/pull/4211 . I'm actually not sure what the performance characteristics are; but I'm sure it's not as dramatic as improvement (if any) compared to what we see in Python. It's great for development though. On Fri, Feb 16,

Re: A 15x speed-up in local Python DirectRunner execution

Does the same runner work for Java pipelines? (I assume so, given that it uses portability framework.) If so, does it provide similar speedup? On Fri, Feb 16, 2018 at 7:37 PM Robert Bradshaw wrote: > If there are no concerns, I say let's merge this. > > On Fri, Feb 16, 2018

Re: A 15x speed-up in local Python DirectRunner execution

If there are no concerns, I say let's merge this. On Fri, Feb 16, 2018 at 9:39 AM, Charles Chen wrote: > I hope those interested have had time to test this out. I have sent out > https://github.com/apache/beam/pull/4696 to switch to using this fast runner > as the default

Re: A 15x speed-up in local Python DirectRunner execution

I hope those interested have had time to test this out. I have sent out https://github.com/apache/beam/pull/4696 to switch to using this fast runner as the default DirectRunner for local execution. Let me know if there are any concerns. On Tue, Feb 13, 2018 at 12:17 PM Charles Chen

Re: A 15x speed-up in local Python DirectRunner execution

This is now checked into master. You can use it by setting --runner=SwitchingDirectRunner. Please let us know if you run into any issues. On Thu, Feb 8, 2018 at 10:30 AM Romain Manni-Bucau wrote: > Very interesting! Sounds like a sane way for beam future and I'm very >

Re: A 15x speed-up in local Python DirectRunner execution

Very interesting! Sounds like a sane way for beam future and I'm very happy it is consistent with the current Java experience: no need to interlace runners at the end, it makes design, code and user experience way better than trying to put everything in the direct runner :). Le 8 févr. 2018

Re: A 15x speed-up in local Python DirectRunner execution

Amazing improvement, Charles. Thanks for the effort! On Thu, Feb 8, 2018 at 10:14 AM Eugene Kirpichov wrote: > Sounds awesome, congratulations and thanks for making this happen! > > On Thu, Feb 8, 2018 at 10:07 AM Raghu Angadi wrote: > >> This is

Re: A 15x speed-up in local Python DirectRunner execution

Sounds awesome, congratulations and thanks for making this happen! On Thu, Feb 8, 2018 at 10:07 AM Raghu Angadi wrote: > This is terrific news! Thanks Charles. > > On Wed, Feb 7, 2018 at 5:55 PM, Charles Chen wrote: > >> Local execution of Beam pipelines on

Re: A 15x speed-up in local Python DirectRunner execution

This is terrific news! Thanks Charles. On Wed, Feb 7, 2018 at 5:55 PM, Charles Chen wrote: > Local execution of Beam pipelines on the Python DirectRunner currently > suffers from performance issues, which makes it hard for pipeline authors > to iterate, especially on medium to

Re: A 15x speed-up in local Python DirectRunner execution

Awesome! Well done, Charles. On Thu, Feb 8, 2018 at 9:10 AM, Ismaël Mejía wrote: > Sounds impressive, and with the extra portability stuff, great ! > Worth the switch just for he user experience improvement. > > On Thu, Feb 8, 2018 at 5:52 PM, Robert Bradshaw

Re: A 15x speed-up in local Python DirectRunner execution

Sounds impressive, and with the extra portability stuff, great ! Worth the switch just for he user experience improvement. On Thu, Feb 8, 2018 at 5:52 PM, Robert Bradshaw wrote: > This is going to be a great improvement for our users! I'll take a > look at the pull request.

Re: A 15x speed-up in local Python DirectRunner execution

This is going to be a great improvement for our users! I'll take a look at the pull request. On Wed, Feb 7, 2018 at 7:03 PM, Kenneth Knowles wrote: > Nice! > > On Wed, Feb 7, 2018 at 6:45 PM, Charles Chen wrote: >> >> The existing DirectRunner will be needed

Re: A 15x speed-up in local Python DirectRunner execution

Nice! On Wed, Feb 7, 2018 at 6:45 PM, Charles Chen wrote: > The existing DirectRunner will be needed for the foreseeable future since > it is currently the only local runner that supports streaming execution. > > On Wed, Feb 7, 2018, 6:39 PM Pablo Estrada

Re: A 15x speed-up in local Python DirectRunner execution

The existing DirectRunner will be needed for the foreseeable future since it is currently the only local runner that supports streaming execution. On Wed, Feb 7, 2018, 6:39 PM Pablo Estrada wrote: > Very cool Charles! Have you considered whether you'll want to remove the >

Re: A 15x speed-up in local Python DirectRunner execution

Very cool Charles! Have you considered whether you'll want to remove the direct runner code afterwards? Best -P. On Wed, Feb 7, 2018, 6:25 PM Lukasz Cwik wrote: > That is pretty awesome. > > On Wed, Feb 7, 2018 at 5:55 PM, Charles Chen wrote: > >> Local

Re: A 15x speed-up in local Python DirectRunner execution

That is pretty awesome. On Wed, Feb 7, 2018 at 5:55 PM, Charles Chen wrote: > Local execution of Beam pipelines on the Python DirectRunner currently > suffers from performance issues, which makes it hard for pipeline authors > to iterate, especially on medium to large size

A 15x speed-up in local Python DirectRunner execution

Local execution of Beam pipelines on the Python DirectRunner currently suffers from performance issues, which makes it hard for pipeline authors to iterate, especially on medium to large size datasets. We would like to optimize and make this a better experience for Beam users. The FnApiRunner