[DISCUSS] Include import statements in documentation code examples

2015-11-18 Thread Robert Metzger
Hi, I helped somebody yesterday on SO [1] who had issues with the Scala API because he was importing the classes from the Java API. Somebody else complained about this issue as well in the comments below the documentation [2], and I think both users are right: Its an unnecessary obstacle when lear

Re: [DISCUSS] Include import statements in documentation code examples

2015-11-18 Thread Till Rohrmann
I agree that it would be nice to be able to simply copy paste examples from Flink's website. But it would also be nice if one could hide the imports because they usually take a lot of space. Cheers, Till On Wed, Nov 18, 2015 at 12:09 PM, Robert Metzger wrote: > Hi, > > I helped somebody yesterd

Re: [DISCUSS] Include import statements in documentation code examples

2015-11-18 Thread Maximilian Michels
Hi Robert. Good suggestion. Generally, it would be nice to have complete code examples available in the documentation. Even better, a way to only show excerpts of the complete example with the option of copying the complete working example. For instance: public Example { public static void ma

[DISCUSS] Release Flink 0.10.1 soon

2015-11-18 Thread Robert Metzger
Hi, I was wondering whether we should release Flink 0.10.1 soon, as there are some issues we've identified: (pending PRs) - FLINK-3032: Flink does not start on Hadoop 2.7.1 (HDP), due to class conflict - FLINK-3011, 3019, 3028 Cancel jobs in RESTARTING state - FLINK-3021 Fix class loading issue f

Re: [DISCUSS] Release Flink 0.10.1 soon

2015-11-18 Thread Vasiliki Kalavri
Hey, I would also add FLINK-3012 and FLINK-3036 (both pending PRs). Thanks! -Vasia. On 18 November 2015 at 12:24, Robert Metzger wrote: > Hi, > > I was wondering whether we should release Flink 0.10.1 soon, as there are > some issues we've identified: > > (pending PRs) > - FLINK-3032: Flink do

Re: [DISCUSS] Release Flink 0.10.1 soon

2015-11-18 Thread Stephan Ewen
+1 for a timely 0.10.1 release I would like to add FLINK-2974 - periodic kafka offset committer for case where checkpointing is deactivated On Wed, Nov 18, 2015 at 12:34 PM, Vasiliki Kalavri < vasilikikala...@gmail.com> wrote: > Hey, > > I would also add FLINK-3012 and FLINK-3036 (both pending P

Re: [DISCUSS] Release Flink 0.10.1 soon

2015-11-18 Thread Till Rohrmann
@Vasia, do you mean FLINK-3013 or FLINK-3012? Will merge PRs for FLINK-3036 and FLINK-3013 this afternoon. On Wed, Nov 18, 2015 at 1:47 PM, Stephan Ewen wrote: > +1 for a timely 0.10.1 release > > I would like to add FLINK-2974 - periodic kafka offset committer for case > where checkpointing is

Re: [DISCUSS] Release Flink 0.10.1 soon

2015-11-18 Thread Ufuk Celebi
Big +1 I think the mentioned issues cover all important fixes. – Ufuk > On 18 Nov 2015, at 13:49, Till Rohrmann wrote: > > @Vasia, do you mean FLINK-3013 or FLINK-3012? > > Will merge PRs for FLINK-3036 and FLINK-3013 this afternoon. > > On Wed, Nov 18, 2015 at 1:47 PM, Stephan Ewen wrote:

Re: [DISCUSS] Release Flink 0.10.1 soon

2015-11-18 Thread Vasiliki Kalavri
*FLINK-3013 you're right :) On 18 November 2015 at 13:52, Ufuk Celebi wrote: > Big +1 > > I think the mentioned issues cover all important fixes. > > – Ufuk > > > On 18 Nov 2015, at 13:49, Till Rohrmann wrote: > > > > @Vasia, do you mean FLINK-3013 or FLINK-3012? > > > > Will merge PRs for FLIN

Re: [DISCUSS] Release Flink 0.10.1 soon

2015-11-18 Thread Suneel Marthi
Would be nice to have Flink-2949 in the mix, but I won't be able to get to it until early next week. On Wed, Nov 18, 2015 at 7:49 AM, Till Rohrmann wrote: > @Vasia, do you mean FLINK-3013 or FLINK-3012? > > Will merge PRs for FLINK-3036 and FLINK-3013 this afternoon. > > On Wed, Nov 18, 2015 at

Re: [DISCUSS] Release Flink 0.10.1 soon

2015-11-18 Thread Ufuk Celebi
@Suneel: I think that's OK for the next major release :) On Wed, Nov 18, 2015 at 2:17 PM, Suneel Marthi wrote: > Would be nice to have Flink-2949 in the mix, but I won't be able to get to > it until early next week. > > On Wed, Nov 18, 2015 at 7:49 AM, Till Rohrmann > wrote: > > > @Vasia, do yo

[jira] [Created] (FLINK-3037) Make the behavior of the Kafka consumer configurable if the offsets to restore from are not available

2015-11-18 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-3037: - Summary: Make the behavior of the Kafka consumer configurable if the offsets to restore from are not available Key: FLINK-3037 URL: https://issues.apache.org/jira/browse/FLINK-3

Re: Fixing the ExecutionConfig

2015-11-18 Thread Stephan Ewen
I had pretty much in mind what Aljoscha suggested. On Thu, Nov 12, 2015 at 11:37 AM, Aljoscha Krettek wrote: > IMHO it’s not possible to have streaming/batch specific ExecutionConfig > since the user functions share a common interface, i.e. > getRuntimeContext().getExecutionConfig() simply retur

Re: Streaming statefull operator with hashmap

2015-11-18 Thread Stephan Ewen
For initializing the Map manually, I meant making "null" the default value and writing the code like HashMap map = state.value() if (map == null) { map = new HashMap<>(); } rather than expecting the state to always clone you a new empty map On Thu, Nov 12, 2015 at 11:29 AM, Aljoscha Krettek w

[jira] [Created] (FLINK-3038) Add maven plugin to ensure our modules define all used dependencies

2015-11-18 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-3038: - Summary: Add maven plugin to ensure our modules define all used dependencies Key: FLINK-3038 URL: https://issues.apache.org/jira/browse/FLINK-3038 Project: Flink

Re: Fixing the ExecutionConfig

2015-11-18 Thread Ufuk Celebi
I like this idea. +1 > On 18 Nov 2015, at 15:25, Stephan Ewen wrote: > > I had pretty much in mind what Aljoscha suggested. > > On Thu, Nov 12, 2015 at 11:37 AM, Aljoscha Krettek > wrote: > >> IMHO it’s not possible to have streaming/batch specific ExecutionConfig >> since the user functions

Re: [DISCUSSION] Type hints versus TypeInfoParser

2015-11-18 Thread Stephan Ewen
If we want to drop any of the possible ".returns(...)" methods (like class or string) we should do that now. The "class" method is probably redundant anyways, as the hints are only necessary in generics are involved (where class does not help anyways). @Timo: What can the String do what the TypeH

Re: [DISCUSS] Include import statements in documentation code examples

2015-11-18 Thread Nick Dimiduk
In HBase we keep an hbase-examples module with working code. Snippets from that module are pasted into docs and referenced. Yes, we do see divergence, especially when refactor tools are involved. I once looked into a doc tool for automatically extracting snippets from source code, but that turned i

[jira] [Created] (FLINK-3039) Trigger KeyValueState cannot be Scala Int

2015-11-18 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-3039: Summary: Trigger KeyValueState cannot be Scala Int Key: FLINK-3039 URL: https://issues.apache.org/jira/browse/FLINK-3039 Project: Flink Issue Type: Improveme

Add BigDecimal and BigInteger as types

2015-11-18 Thread Timo Walther
Hey everyone, I'm not sure if we already had a discussion about it but as we are currently adding new types like the Either type, I would like to discuss it again. I think especially for business or scientific applications it makes sense to support the BigInteger and BigDecimal types natively.

Re: [DISCUSSION] Type hints versus TypeInfoParser

2015-11-18 Thread Timo Walther
If the TypeExtractor is not able to handle the fields of a Pojo correctly, the String parser is quite useful to say "org.my.Pojo>". Doing this with TypeInformation classes is terrible. But I think this a corner case we can ignore. I would also remove the ".returns(String)" in favor of TypeHint

Re: Add BigDecimal and BigInteger as types

2015-11-18 Thread Stephan Ewen
I agree that they are important. They are currently generic types and handled by Kryo, which has (AFAIK) proper serializers for them. Are there more benefits of native support (other than more compact serialization) that you are thinking of? On Wed, Nov 18, 2015 at 5:55 PM, Timo Walther wrote:

[jira] [Created] (FLINK-3040) Add docs describing how to configure State Backends

2015-11-18 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-3040: --- Summary: Add docs describing how to configure State Backends Key: FLINK-3040 URL: https://issues.apache.org/jira/browse/FLINK-3040 Project: Flink Issue Type: I

Re: [DISCUSSION] Type hints versus TypeInfoParser

2015-11-18 Thread Stephan Ewen
I think the TypeHints case can cover this: public class MyPojo { public T field1; public R field2; } If you say '.returns(new TypeHint>() {})' this creates an anonymous subclass of the TypeHint, which has the types that T and R bind to, which allows one to construct the POJO type info pro

[jira] [Created] (FLINK-3041) Twitter Streaming Description section of Streaming Programming guide refers to an incorrect example 'TwitterLocal'

2015-11-18 Thread Suneel Marthi (JIRA)
Suneel Marthi created FLINK-3041: Summary: Twitter Streaming Description section of Streaming Programming guide refers to an incorrect example 'TwitterLocal' Key: FLINK-3041 URL: https://issues.apache.org/jira/bro

[jira] [Created] (FLINK-3042) Define a way to let types create their own TypeInformation

2015-11-18 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-3042: --- Summary: Define a way to let types create their own TypeInformation Key: FLINK-3042 URL: https://issues.apache.org/jira/browse/FLINK-3042 Project: Flink Issue

[jira] [Created] (FLINK-3043) Kafka Connector description in Streaming API guide is wrong/outdated

2015-11-18 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-3043: --- Summary: Kafka Connector description in Streaming API guide is wrong/outdated Key: FLINK-3043 URL: https://issues.apache.org/jira/browse/FLINK-3043 Project: Flink

[jira] [Created] (FLINK-3044) In YARN mode, configure FsStateBackend by default.

2015-11-18 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-3044: --- Summary: In YARN mode, configure FsStateBackend by default. Key: FLINK-3044 URL: https://issues.apache.org/jira/browse/FLINK-3044 Project: Flink Issue Type: Im