Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

[Reuven Lax]

Do you set ENABLE_AUTO_COMMIT_CONFIG?

On Sun, Jan 7, 2018 at 10:57 PM, Sushil Ks  wrote:

> HI Aljoscha,
>The issue is let's say I consumed 100 elements in 5
> mins Fixed Window with *GroupByKey* and later I applied *ParDO* for all
> those elements. If there is an issue while processing element 70 in
> *ParDo *and the pipeline restarts with *UserCodeException *it's skipping
> the rest 30 elements. Wanted to know if this is expected? In case if you
> still having doubt let me know will share a code snippet.
>
> Regards,
> Sushil Ks
>

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

[Sushil Ks]

HI Aljoscha,
   The issue is let's say I consumed 100 elements in 5 mins
Fixed Window with *GroupByKey* and later I applied *ParDO* for all those
elements. If there is an issue while processing element 70 in *ParDo *and
the pipeline restarts with *UserCodeException *it's skipping the rest 30
elements. Wanted to know if this is expected? In case if you still having
doubt let me know will share a code snippet.

Regards,
Sushil Ks

Re: Switching to Java 8

[Jean-Baptiste Onofré]

Yes, that's the plan: build first, example "merge" after.

Regards
JB

On 01/08/2018 07:43 AM, Eugene Kirpichov wrote:
Sounds great, thanks! Probably best done as 2 separate steps, because after 
updating the build scripts, everything else can begin in parallel?


On Sun, Jan 7, 2018 at 10:38 PM Jean-Baptiste Onofré > wrote:


Hi Eugene,

I'm taking the build update: Maven/Gradle with enforcer + merge of the 
examples
all together.

Regards
JB

On 01/08/2018 07:34 AM, Eugene Kirpichov wrote:
 > The vote on user@ about switching to Java 8 has concluded, affirmatively.
 >
 > What needs to be done to complete the switch? I can see at least the
following:
 > - Change maven and gradle scripts to use 1.8 source and target version
 > - Fix resulting compilation/test errors (Java8 has slightly different 
type
 > checking, more minor issues may arise)
 > - Remove all special-casing of java8 in build scripts
 > - Merge all modules like "java8 examples" and "java8 tests" into 
respective
 > non-"java8" modules
 > - Organize an effort to modernize code to use Java 8 constructs where
 > appropriate. Especially important to modernize examples. To a large
extent this
 > can probably be automated with an IDE.
 >
 > Anything else?
 >

--
Jean-Baptiste Onofré
jbono...@apache.org 
http://blog.nanthrax.net
Talend - http://www.talend.com



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Switching to Java 8

[Eugene Kirpichov]

Sounds great, thanks! Probably best done as 2 separate steps, because after
updating the build scripts, everything else can begin in parallel?

On Sun, Jan 7, 2018 at 10:38 PM Jean-Baptiste Onofré 
wrote:

> Hi Eugene,
>
> I'm taking the build update: Maven/Gradle with enforcer + merge of the
> examples
> all together.
>
> Regards
> JB
>
> On 01/08/2018 07:34 AM, Eugene Kirpichov wrote:
> > The vote on user@ about switching to Java 8 has concluded,
> affirmatively.
> >
> > What needs to be done to complete the switch? I can see at least the
> following:
> > - Change maven and gradle scripts to use 1.8 source and target version
> > - Fix resulting compilation/test errors (Java8 has slightly different
> type
> > checking, more minor issues may arise)
> > - Remove all special-casing of java8 in build scripts
> > - Merge all modules like "java8 examples" and "java8 tests" into
> respective
> > non-"java8" modules
> > - Organize an effort to modernize code to use Java 8 constructs where
> > appropriate. Especially important to modernize examples. To a large
> extent this
> > can probably be automated with an IDE.
> >
> > Anything else?
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Switching to Java 8

[Jean-Baptiste Onofré]

Hi Eugene,

I'm taking the build update: Maven/Gradle with enforcer + merge of the examples 
all together.


Regards
JB

On 01/08/2018 07:34 AM, Eugene Kirpichov wrote:

The vote on user@ about switching to Java 8 has concluded, affirmatively.

What needs to be done to complete the switch? I can see at least the following:
- Change maven and gradle scripts to use 1.8 source and target version
- Fix resulting compilation/test errors (Java8 has slightly different type 
checking, more minor issues may arise)

- Remove all special-casing of java8 in build scripts
- Merge all modules like "java8 examples" and "java8 tests" into respective 
non-"java8" modules
- Organize an effort to modernize code to use Java 8 constructs where 
appropriate. Especially important to modernize examples. To a large extent this 
can probably be automated with an IDE.


Anything else?



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Switching to Java 8

[Eugene Kirpichov]

The vote on user@ about switching to Java 8 has concluded, affirmatively.

What needs to be done to complete the switch? I can see at least the
following:
- Change maven and gradle scripts to use 1.8 source and target version
- Fix resulting compilation/test errors (Java8 has slightly different type
checking, more minor issues may arise)
- Remove all special-casing of java8 in build scripts
- Merge all modules like "java8 examples" and "java8 tests" into respective
non-"java8" modules
- Organize an effort to modernize code to use Java 8 constructs where
appropriate. Especially important to modernize examples. To a large extent
this can probably be automated with an IDE.

Anything else?

Re: jackson to parse options?

[Romain Manni-Bucau]

Le 7 janv. 2018 12:53, "Jean-Baptiste Onofré"  a écrit :

Hi Romain,

I guess you are assuming that pipeline options are flat command line like
argument, right ?

Actually, theoretically,  pipeline options can be represented as json,
that's why we use jackson.
The pipeline options can be serialized/deserialized as json.


Yes but if users (or dev ;)) start to use that then it is trivial to break
the cli handling and fromArgs parsing or, if not, break the user
experience. So at the end it is a kind of "yes but no", right?

PS: already see some advanced users having a headache trying to pass
pipeline options in json so using the plain command line syntax can be more
friendly too.


The purpose is to remove the jackson dependencies ?


Later yes, seeing core dep tree I identify a lot of dep which can conflict
in some env and are not really needed or bring much being in the core -
like avro as mentionned in another thread. Can need a sanitizing round.
Short term it was really a "why is it that complicated" ;).



Regards
JB


On 01/07/2018 11:38 AM, Romain Manni-Bucau wrote:

> Hi guys,
>
> not sure i fully get why jackson is used to parse pipeline options in the
> testing integration
>
> why not using a token parser like [1] which allows to map 1-1 with the
> user interface (command line) the options?
>
> [1] https://github.com/Talend/component-runtime/blob/master/comp
> onent-server/src/main/java/org/talend/sdk/component/serve
> r/lang/StringPropertiesTokenizer.java
>
> Romain Manni-Bucau
> @rmannibucau  | Blog <
> https://rmannibucau.metawerx.net/> | Old Blog <
> http://rmannibucau.wordpress.com> | Github  |
> LinkedIn 
>

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: jackson to parse options?

[Jean-Baptiste Onofré]

Hi Romain,

I guess you are assuming that pipeline options are flat command line like 
argument, right ?


Actually, theoretically,  pipeline options can be represented as json, that's 
why we use jackson.

The pipeline options can be serialized/deserialized as json.

The purpose is to remove the jackson dependencies ?

Regards
JB

On 01/07/2018 11:38 AM, Romain Manni-Bucau wrote:

Hi guys,

not sure i fully get why jackson is used to parse pipeline options in the 
testing integration


why not using a token parser like [1] which allows to map 1-1 with the user 
interface (command line) the options?


[1] 
https://github.com/Talend/component-runtime/blob/master/component-server/src/main/java/org/talend/sdk/component/server/lang/StringPropertiesTokenizer.java


Romain Manni-Bucau
@rmannibucau  | Blog 
 | Old Blog 
 | Github  | 
LinkedIn 


--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

jackson to parse options?

[Romain Manni-Bucau]

Hi guys,

not sure i fully get why jackson is used to parse pipeline options in the
testing integration

why not using a token parser like [1] which allows to map 1-1 with the user
interface (command line) the options?

[1]
https://github.com/Talend/component-runtime/blob/master/component-server/src/main/java/org/talend/sdk/component/server/lang/StringPropertiesTokenizer.java

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn 

Jenkins build is still unstable: beam_Release_NightlySnapshot #644

[Apache Jenkins Server]

See