Re: Java > 8 support

[Romain Manni-Bucau]

@Reuven: bytebuddy by itself no but the way beam tries to inject the proxy
class is. There are other strategies you can use in bytebuddy which work.

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book



Le sam. 6 oct. 2018 à 17:51, Reuven Lax  a écrit :

> Romain, do you have any more details on the ByteBuddy incompatibility? Is
> ByteBuddy incompatible with the Java 11 JRE, or just with new language
> features?
>
> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau 
> wrote:
>
>> Hi Arif,
>>
>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it
>> means your pipeline is very very simple since it does not have a dofn ;))
>> if your engine supports it. Also note that the modules not being named you
>> can have to use some weird import names or even unstable ones if you want
>> to use modules (but there is no real reason to do that yet in java).
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>>
>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim  a écrit :
>>
>>> Hello,
>>> What's the status of java version > 8 support for beam? Thanks.
>>>
>>> -Arif.
>>>
>>

Re: Java > 8 support

[Jean-Baptiste Onofré]

Hi,

I mean that some IO dependency, especially for the tests should be
updated for the build with J9.

The runtime should work with J9, but not the build.

Regards
JB

On 06/10/2018 17:45, Romain Manni-Bucau wrote:
> I dont think so JB - until you want to use jlink but it is out of scope
> of beam - and since j9 didnt break unsafe it should work.
> 
> Le sam. 6 oct. 2018 17:23, Jean-Baptiste Onofré  > a écrit :
> 
> Same thing, it requires some change in the dependency, especially for
> the Jigsaw modules.
> 
> Regards
> JB
> 
> On 06/10/2018 15:01, Arif Kasim wrote:
> > Thanks all. What about Java 9 support?
> >
> >
> >       
> >       
> >       
> >       
> >
> >       
> >
> >     *  •  **Arif Kasim**
> >     **  • * Strategic Cloud Engineer*
> >     **  •  *Google, Inc.*
> >     *  •  arifka...@google.com 
> >  
> >
> >
> >
> >
> > On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau
> mailto:rmannibu...@gmail.com>
> > >> wrote:
> >
> >     Hi Arif,
> >
> >     AFAIK bytebuddy code is not java 11 friendly otherwise it runs
> (but
> >     it means your pipeline is very very simple since it does not
> have a
> >     dofn ;)) if your engine supports it. Also note that the
> modules not
> >     being named you can have to use some weird import names or even
> >     unstable ones if you want to use modules (but there is no real
> >     reason to do that yet in java).
> >
> >     Romain Manni-Bucau
> >     @rmannibucau  |  Blog
> >      | Old Blog
> >      | Github
> >      | LinkedIn
> >      | Book
> >   
>  
> 
> >
> >
> >     Le ven. 5 oct. 2018 à 19:10, Arif Kasim  
> >     >> a
> écrit :
> >
> >         Hello,
> >         What's the status of java version > 8 support for beam?
> Thanks.
> >
> >         -Arif.
> >
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org 
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Java > 8 support

[Reuven Lax]

Romain, do you have any more details on the ByteBuddy incompatibility? Is
ByteBuddy incompatible with the Java 11 JRE, or just with new language
features?

On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau 
wrote:

> Hi Arif,
>
> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it
> means your pipeline is very very simple since it does not have a dofn ;))
> if your engine supports it. Also note that the modules not being named you
> can have to use some weird import names or even unstable ones if you want
> to use modules (but there is no real reason to do that yet in java).
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
>
> Le ven. 5 oct. 2018 à 19:10, Arif Kasim  a écrit :
>
>> Hello,
>> What's the status of java version > 8 support for beam? Thanks.
>>
>> -Arif.
>>
>

Re: Java > 8 support

[Romain Manni-Bucau]

I dont think so JB - until you want to use jlink but it is out of scope of
beam - and since j9 didnt break unsafe it should work.

Le sam. 6 oct. 2018 17:23, Jean-Baptiste Onofré  a écrit :

> Same thing, it requires some change in the dependency, especially for
> the Jigsaw modules.
>
> Regards
> JB
>
> On 06/10/2018 15:01, Arif Kasim wrote:
> > Thanks all. What about Java 9 support?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *  •  **Arif Kasim**
> > **  • * Strategic Cloud Engineer*
> > **  •  *Google, Inc.*
> > *  •  arifka...@google.com 
> >
> >
> >
> >
> > On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau  > > wrote:
> >
> > Hi Arif,
> >
> > AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but
> > it means your pipeline is very very simple since it does not have a
> > dofn ;)) if your engine supports it. Also note that the modules not
> > being named you can have to use some weird import names or even
> > unstable ones if you want to use modules (but there is no real
> > reason to do that yet in java).
> >
> > Romain Manni-Bucau
> > @rmannibucau  |  Blog
> >  | Old Blog
> >  | Github
> >  | LinkedIn
> >  | Book
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >
> >
> > Le ven. 5 oct. 2018 à 19:10, Arif Kasim  > > a écrit :
> >
> > Hello,
> > What's the status of java version > 8 support for beam? Thanks.
> >
> > -Arif.
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Java > 8 support

[Jean-Baptiste Onofré]

Same thing, it requires some change in the dependency, especially for
the Jigsaw modules.

Regards
JB

On 06/10/2018 15:01, Arif Kasim wrote:
> Thanks all. What about Java 9 support?
> 
> 
>   
>   
>   
>   
> 
>   
> 
> *  •  **Arif Kasim**
> **  • * Strategic Cloud Engineer*
> **  •  *Google, Inc.*
> *  •  arifka...@google.com   
> 
> 
> 
> 
> On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau  > wrote:
> 
> Hi Arif,
> 
> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but
> it means your pipeline is very very simple since it does not have a
> dofn ;)) if your engine supports it. Also note that the modules not
> being named you can have to use some weird import names or even
> unstable ones if you want to use modules (but there is no real
> reason to do that yet in java).
> 
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
> 
> 
> 
> Le ven. 5 oct. 2018 à 19:10, Arif Kasim  > a écrit :
> 
> Hello,
> What's the status of java version > 8 support for beam? Thanks.
> 
> -Arif.
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Is there any way to ask the runner to call finalizeCheckpoint() method before it closed the Reader?

[flyisland]

Got it, thanks, will double check the PubsubUnboundedSource.

btw, I'd like to learn more about the lifecycle of IO connector(for
example, when will the runner call the reader's close() method?), could you
recommend some documents, thanks!

On Fri, Oct 5, 2018 at 11:01 PM Maximilian Michels  wrote:

> Restoring from a checkpoint is something different. You asked about
> acking pending CheckpointMarks.
>
> If you look in the PubsubUnboundedSource, it doesn't close the client
> connection if the there are still unacked checkpoints. a) Closing the
> reader and b) closing the client connection, these are two different
> actions which do not have to depend on each other.
>
> On 05.10.18 16:06, flyisland wrote:
> > I've checked the PubsubUnboundedSource, it just throws an exception if
> > it's a "restored checkpoint".
> >
> > Actually, for most MQ system, it's no way to ack a message if the
> > Reader(connection) is closed. So it makes no sense to call the
> > finalizeCheckpoint() method after closed the Reader.
> >
> > On Fri, Oct 5, 2018 at 9:01 PM Maximilian Michels  > > wrote:
> >
> > Hi,
> >
> > Not sure whether I'm a guru but I'll try to answer your question ;)
> >
> >  > Is there any way to ask the runner to call finalizeCheckpoint()
> > method before it closed the Reader?
> > Not that I'm aware of.
> >
> > The point the comment is trying to make is that CheckpointMark should
> > not depend on the life cycle of the Reader. So you should find a way
> to
> > encode all necessary information in your CheckpointMark to
> acknowledge
> > even after `close()` has been called on the Reader.
> >
> > Perhaps you want to check out out PubsubUnboundedSource for an
> example
> > of how to do that?
> >
> > Cheers,
> > Max
> >
> > On 05.10.18 14:47, flyisland wrote:
> >  > Hi Gurus,
> >  >
> >  > I'm building a new IO connector now, and I try to ack messages in
> > the
> >  > "UnboundedSource.CheckpointMark.finalizeCheckpoint()" method as
> > MqttIO
> >  > and JmsIO did, but I found in the javadoc it said
> >  >
> >  >  >  It is NOT safe to assume the UnboundedSource.UnboundedReader
> > from
> >  > which this checkpoint was created still exists at the time this
> > method
> >  > is called.
> >  >
> >  > I do encounter this situation in my testing with the Direct
> > Runner, the
> >  > "msg.ack()" method failed when the finalizeCheckpoint() method is
> > called
> >  > since the related reader has already been closed!
> >  >
> >  > Is there any way to ask the runner to call finalizeCheckpoint()
> > method
> >  > before it closed the Reader?
> >
>

Re: Java > 8 support

[Arif Kasim]

Thanks all. What about Java 9 support?




*  •  **Arif Kasim*
*  • * Strategic Cloud Engineer
*  •  *Google, Inc.
  •  arifka...@google.com




On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau 
wrote:

> Hi Arif,
>
> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it
> means your pipeline is very very simple since it does not have a dofn ;))
> if your engine supports it. Also note that the modules not being named you
> can have to use some weird import names or even unstable ones if you want
> to use modules (but there is no real reason to do that yet in java).
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
>
> Le ven. 5 oct. 2018 à 19:10, Arif Kasim  a écrit :
>
>> Hello,
>> What's the status of java version > 8 support for beam? Thanks.
>>
>> -Arif.
>>
>

Re: Beam website sources migrated to apache/beam

[Alexey Romanenko]

This is great news! Many thanks to all who contributed to this!

I believe that as the process of website update will be clearer and simpler, as 
our documentation will be up-to-date.

> On 6 Oct 2018, at 06:38, Jean-Baptiste Onofré  wrote:
> 
> Awesome,
> 
> thanks for the move Scott !
> 
> Regards
> JB
> 
> On 06/10/2018 01:30, Scott Wegner wrote:
>> I'm excited to announce that source code for the Beam website has been
>> successfully migrated to the apache/beam repository [1]. This makes
>> website contributions first-class and consolidates our tooling and
>> processes [2]. Website validation and HTML generation are integrated
>> into our Gradle build (./gradlew :beam-website:check) and orchestrated
>> via our standard Jenkins infrastructure. We expect this will greatly
>> improve PR reliability. For more
>> background: https://s.apache.org/beam-site-automation 
>> 
>> New website contributions should be made in the apache/beam repository.
>> We'll work on draining off and migrating outstanding PR's from the old
>> repository before we disconnect Mergebot and fully retire the previous
>> branch.
>> 
>> We hope the new contribution experience will be seamless and make your
>> website contributions the best part of your day. If you find any rough
>> edges or areas for improvement, please add them to the fit-and-finish
>> list here: https://issues.apache.org/jira/browse/BEAM-5671
>> 
>> Big thanks to everyone who helped out with this project: Jason Kuster
>> and Melissa Pashniak for their mentorship, Thomas Weise and Robert
>> Bradshaw for design feedback, and Alan Myrvold and Udi Meiri for
>> implementation and documentation.
>> 
>> 
>> [1] https://github.com/apache/beam/tree/master/website
>> [2] https://beam.apache.org/contribute/#contributing-to-the-website
>> 
>> -- 
>> 
>> Got feedback? tinyurl.com/swegner-feedback
>> 
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com

Re: python post-commit failures

[Maximilian Michels]

My changes to the Python option parsing broke the PostCommit. PreCommit 
passed, as well as the Portable Runner tests. Sorry about that.


+1 It would be great to have some more basic integration tests in the 
PreCommit. That will give us more confidence before merge without always 
running the PostCommit.



The same goes for Flink: we should be running wordcount and wordcount_streaming 
integration tests as part of pre-commit tests.


Can look into that.

Thanks,
Max

On 05.10.18 23:08, Ahmet Altay wrote:



On Fri, Oct 5, 2018 at 1:51 PM, Udi Meiri > wrote:


I was sure that we ran some basic Dataflow integration tests in
pre-commit, and that they should have caught this issue.
But then I remembered that we only have those in Java SDK.
I opened this bug to add end-to-end tests to Python pre-commits as
well: https://issues.apache.org/jira/browse/BEAM-5058



+1


The same goes for Flink: we should be running wordcount and
wordcount_streaming integration tests as part of pre-commit tests.

On Fri, Oct 5, 2018 at 1:37 PM Thomas Weise mailto:t...@apache.org>> wrote:

Fixed. Can someone please take a look at the usage of
the --beam_plugins flag in the Dataflow runner so that we can
address the root cause?

We can probably do more to avoid Friday Python post commit
excitement. In this case, extra checking was done pre-merge by
running the Python VR tests for Flink, but the failure occurred
with the Dataflow runner.


It would be good to have a list of what post commit test are available, 
what do they test and what is the keyword to trigger them from a PR.



The changes were pipeline options related, so (pre-existing)
test coverage should have been better.

But beyond that, we can probably make it easier for contributors
and reviewers to know what extra checks are available and
possibly appropriate to run pre-commit. Should we add some
pointers to
https://beam.apache.org/contribute/testing/#pre-commit
 or is
there a better place?

Thanks



On Fri, Oct 5, 2018 at 10:38 AM Udi Meiri mailto:eh...@google.com>> wrote:

More details in
https://issues.apache.org/jira/browse/BEAM-5442


On Fri, Oct 5, 2018 at 10:26 AM Udi Meiri mailto:eh...@google.com>> wrote:

I'm seeing these errors at least in one test:
"Python sdk harness failed:
Traceback (most recent call last):
   File

"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
line 133, in main

sdk_pipeline_options.get_all_options(drop_default=True))

   File

"/usr/local/lib/python2.7/dist-packages/apache_beam/options/pipeline_options.py",
line 224, in get_all_options
     parser.add_argument(arg.split('=', 1)[0], nargs='?')
   File "/usr/lib/python2.7/argparse.py", line 1308, in
add_argument
     return self._add_action(action)
   File "/usr/lib/python2.7/argparse.py", line 1682, in
_add_action
     self._optionals._add_action(action)
   File "/usr/lib/python2.7/argparse.py", line 1509, in
_add_action
     action = super(_ArgumentGroup,
self)._add_action(action)
   File "/usr/lib/python2.7/argparse.py", line 1322, in
_add_action
     self._check_conflict(action)
   File "/usr/lib/python2.7/argparse.py", line 1460, in
_check_conflict
     conflict_handler(action, confl_optionals)
   File "/usr/lib/python2.7/argparse.py", line 1467, in
_handle_conflict_error
     raise ArgumentError(action, message % conflict_string)
ArgumentError: argument --beam_plugins: conflicting
option string(s): --beam_plugins"

This looks like https://github.com/apache/beam/pull/6557


On Fri, Oct 5, 2018 at 9:41 AM Boyuan Zhang
mailto:boyu...@google.com>> wrote:

Seems like tests failed:
test_leader_board_it

(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
-> Bigquery table not found
test_game_stats_it

(apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT)