Re: [RESULT] [VOTE] Release 1.12.2, release candidate #2

[Henry Saputra]

Apparently I suck at math, yes it should be 4 binding Votes. Thanks, Dawid

- Henry

On Tue, Mar 2, 2021 at 12:59 PM Roman Khachatryan  wrote:

> Yes, you are right.
>
> There are 7 approving votes, 4 of which are binding:
> * Kurt Young
> * Piotr Nowojski
> * Yu Li
> * Zhu Zhu
>
> There are no disapproving votes.
>
> Thanks everyone!
>
> Regards,
> Roman
>
>
> On Tue, Mar 2, 2021 at 9:29 PM Dawid Wysakowicz 
> wrote:
>
> > Yes, Henry is right. Binding votes on Apache releases must come from PMC
> > members.
> >
> > Out of the votes 4 are binding:
> >
> > * Kurt Young
> > * Piotr Nowojski
> > * Yu Li
> > * Zhu Zhu
> >
> > Best,
> > Dawid
> >
> > On 02/03/2021 21:17, Henry Saputra wrote:
> > > Roman, binding Votes only come from Apache Flink PMC members.
> > >
> > > From the list you had seems like only 3 binding Votes, right?
> > >
> > >
> > >
> > > On Tue, Mar 2, 2021, 7:08 AM Roman Khachatryan 
> wrote:
> > >
> > >> I'm happy to announce that we have unanimously approved this release.
> > >>
> > >> There are 7 approving votes, 5 of which are binding:
> > >> * Kurt Young
> > >> * Piotr Nowojski
> > >> * Roman Khachatryan
> > >> * Yu Li
> > >> * Zhu Zhu
> > >>
> > >> There are no disapproving votes.
> > >>
> > >> Thanks everyone!
> > >>
> > >> Regards,
> > >> Roman
> > >>
> >
> >
>

Re: [RESULT] [VOTE] Release 1.12.2, release candidate #2

[Henry Saputra]

Roman, binding Votes only come from Apache Flink PMC members.

>From the list you had seems like only 3 binding Votes, right?



On Tue, Mar 2, 2021, 7:08 AM Roman Khachatryan  wrote:

> I'm happy to announce that we have unanimously approved this release.
>
> There are 7 approving votes, 5 of which are binding:
> * Kurt Young
> * Piotr Nowojski
> * Roman Khachatryan
> * Yu Li
> * Zhu Zhu
>
> There are no disapproving votes.
>
> Thanks everyone!
>
> Regards,
> Roman
>

Re: [DISCUSS] FLIP-166: Flink/Pinot Connector

[Henry Saputra]

I think this is good contribution to Apache Bahir:

https://github.com/apache/bahir-flink

as proper place to additional Flink connectors.

- Henry

On Tue, Mar 2, 2021, 11:35 AM Yupeng Fu  wrote:

> Hi all,
>
> I'd like to propose a Flink/Pinot connector to solve the problems of
> streaming/batch unification for the real-time analytical infrastructure on
> top of Flink and Pinot. Apache Pinot  is a
> real-time distributed OLAP datastore with an inbuilt lambda architecture.
> With the Flink/Pinot connector, Pinot's streaming/batch ingestion pipelines
> can be unified using Flink only. Please read more in the FLIP proposal
> <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177045634
> >,
> and share comments.
>
> Thanks,
>
> --
> --Yupeng
>

Re: [DISCUSS] Splitting User support mailing list

[Henry Saputra]

-1 for splitting user list to areas in Flink.

As Robert and others have chimed in, we could have separate user list for
sub projects in Flink, like statefun

- Henry

On Tue, Mar 2, 2021, 11:27 AM Roman Khachatryan  wrote:

> Thanks Robert,
>
> That's a good idea, let's revisit it later.
>
> Regards,
> Roman
>
>
> On Tue, Mar 2, 2021 at 3:40 PM Robert Metzger  wrote:
>
> > Thanks a lot for bringing up this idea Roman!
> >
> > After reading the initial proposal, I quite liked the idea, because it
> > makes our life easier: We can only monitor lists relevant for the topics
> we
> > are working on (I have to admit that I usually skip all questions that
> seem
> > to be related to SQL or Statefun).
> > There are a few Apache projects which have followed a similar approach
> [1],
> > most notable maybe the Hadoop project, which has a user@, as well as
> > hdfs-user@, mapreduce-user@, ozone-user@ etc. There, it seems that
> > sub-projects have separate lists. This would support the idea of
> splitting
> > out statefun into a separate list.
> >
> > But the majority of people who have commented so far seem to have
> concerns
> > regarding the proposal, which seem reasonable.
> > I propose to revisit this proposal at a later point.
> >
> >
> >
> > [1] http://mail-archives.apache.org/mod_mbox/
> >
>

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

[Henry Saputra]

Hi Alexander,

I had to moderate and accept your email to dev@ list. Could you subscribe
to dev@ list for Apache Flink [1] to continue getting updates from your
discussion thread?

Thanks,

Henry

[1] https://flink.apache.org/community.html#mailing-lists

On Mon, Mar 1, 2021 at 3:42 PM Alexander Fedulov 
wrote:

> Hi All,
>
> I would like to start a discussion for FLIP-165: Operator's Flame Graphs
> [1]
>
> A Flame Graph [2] is a visualization that is very effective for providing
> answers to the questions like:
> - Which methods are currently consuming CPU resources?
> - How CPU utilization by one method compares to the others?
> - Which series of calls on the stack led to executing a particular method?
>
> I have already opened a PR [3] that represents the implementation approach
> proposed in the FLIP. It supports both on-CPU and off-CPU [4] Flame Graphs.
>
> Looking forward to your feedback.
>
> P.S: I would like to give kudos to David Moravek for his prototyping work
> [5] on this feature. Although the proposed implementation significantly
> diverges from his prototype on the Flink side, the work done on connecting
> the d3-flame-graph library to the right data structure retrieved from Flink
> was instrumental for enabling this feature.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-165%3A+Operator%27s+Flame+Graphs
> [2] http://www.brendangregg.com/flamegraphs.html
> [3] https://github.com/apache/flink/pull/15054
> [4] http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
> [5]
>
> https://issues.apache.org/jira/browse/FLINK-13550?focusedCommentId=17083026&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17083026
>
>
> Best,
> --
>
> Alexander Fedulov | Solutions Architect
>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward - The Apache Flink Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
>
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
> Wehner
>
>
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>

Re: MODERATE for dev@flink.apache.org

[Henry Saputra]

Hi,

Looks like you have not subscribe to Apache Flink dev@ mailing list.

Please subscribe by following instruction here to continue with the
discussions and receive followups:
https://flink.apache.org/community.html#mailing-lists


Thanks,

Henry Saputra
On behalf of Apache Flink PMC


>
>
> -- Forwarded message --
> From: Teng Fei Liao 
> To: dev@flink.apache.org
> Cc:
> Bcc:
> Date: Tue, 2 Jun 2020 03:04:38 -0400
> Subject: Common HA setups
>
> Hi Flink devs!
>
> After reading through documentation and reading posts others have made
> online for their setups, it seems like there’s similarities in achieving HA
> with just a single job manager. For example, the yarn setup
> <https://apollo.palantircloud.com/aries/logDetails/v2/%7B%22columns%22%3A%5B%7B%22propertyKey%22%3A%22time%22%7D%2C%7B%22propertyKey%22%3A%22traceId%22%7D%2C%7B%22propertyKey%22%3A%22level%22%7D%2C%7B%22propertyKey%22%3A%22message%22%7D%2C%7B%22propertyKey%22%3A%22params%22%7D%2C%7B%22propertyKey%22%3A%22origin%22%7D%2C%7B%22propertyKey%22%3A%22stacktrace%22%7D%2C%7B%22propertyKey%22%3A%22params%22%2C%22path%22%3A%22clusterId%22%7D%5D%2C%22queryStrings%22%3A%5B%22origin%3A%5C%22com.palantir.flink.runtime.ha.FoundryHaServicesFactory%5C%22%22%2C%22((params.key%3AclusterId)%20AND%20(params.value%3A%5C%226a5e3805%5C%5C-aa20%5C%5C-4bd9%5C%5C-969d%5C%5C-08f0acee67ec%5C%22))%22%5D%2C%22startTime%22%3A%222020-06-01T03%3A04%3A05.989%22%2C%22serviceLocators%22%3A%5B%7B%22environmentId%22%3A%22oregano-rubix-staging-lowtrust%22%2C%22logType%22%3A%22SERVICE_LOG%22%2C%22blueGreenGroupId%22%3A%22production%7Cflink-job-manager%22%7D%5D%7D>
> specifies only a single job manager is necessary and will be restarted on
> failures. This Kubernetes post
> <https://jobs.zalando.com/en/tech/blog/running-apache-flink-on-kubernetes/?gh_src=22377bdd1us>
> has a similar single job manager setup. It has a fill-in for zookeeper but
> I think abstractly, the two have these same features in common:
>
> 1. Persistent storage (the high-availability.storageDir config value)
>
> 2. Low latency job manager restart times.
>
> For our setup, we're actually experimenting with a variation of the
> kubernetes set up that removes zookeeper altogether by implementing a file
> based HighAvailabilityServices and trivial leader election services. Given
> the relative simplicity of the setup and code, I was wondering how
> recommended and supported this variant is. Potentially, this could be made
> available by default to help other users simplify their setups. Curious
> what your thoughts are.
>
> Thanks,
>
> Teng.
>

Re: MODERATE for dev@flink.apache.org

[Henry Saputra]

Hi,

Looks like you are trying to send email to Apache Flink dev@ mailing list
without subscribing yet.

Please subscribe to the dev mailing list [1] to be able to see the reply
and follow up with your question.

Thanks,

Henry
On behalf of Apache Flink PMC

[1]
https://flink.apache.org/community.html#how-to-subscribe-to-a-mailing-list


>
>
>
>
> -- Forwarded message --
> From: vishalovercome 
> To: dev@flink.apache.org
> Cc:
> Bcc:
> Date: Tue, 19 May 2020 22:13:39 -0700 (MST)
> Subject: Issues using StreamingFileSink (Cannot locate configuration:
> tried hadoop-metrics2-s3a-file-system.properties,
> hadoop-metrics2.properties)
> I've written a Flink job to stream updates to S3 using StreamingFileSink. I
> used the presto plugin for checkpointing and the hadoop plugin for writing
> part files to S3. I was going through my application logs and found these
> messages which I couldn't understand and would like to know if there's some
> step I might have missed.
> Even if these messages are harmless, I would like to understand what these
> mean.
> 2020-05-19 19:23:33,277 INFO
> org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
>
> - Error when creating PropertyDescriptor for public final void
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
> Ignoring this property.
> *2020-05-19 19:23:33,310 WARN
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsConfig
>
> - Cannot locate configuration: tried
> hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties*
> 2020-05-19 19:23:33,397 INFO
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsSystemImpl
>
> - Scheduled Metric snapshot period at 10 second(s).
> 2020-05-19 19:23:33,397 INFO
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsSystemImpl
>
> - s3a-file-system metrics system started
> 2020-05-19 19:23:34,833 INFO
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.deprecation
>
> - fs.s3a.server-side-encryption-key is deprecated. Instead, use
> fs.s3a.server-side-encryption.key
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>
>

Re: [ANNOUNCE] Yu Li became a Flink committer

[Henry Saputra]

Belated congrats to Yu Li

- Henry

On Thu, Jan 23, 2020 at 12:59 AM Stephan Ewen  wrote:

> Hi all!
>
> We are announcing that Yu Li has joined the rank of Flink committers.
>
> Yu joined already in late December, but the announcement got lost because
> of the Christmas and New Years season, so here is a belated proper
> announcement.
>
> Yu is one of the main contributors to the state backend components in the
> recent year, working on various improvements, for example the RocksDB
> memory management for 1.10.
> He has also been one of the release managers for the big 1.10 release.
>
> Congrats for joining us, Yu!
>
> Best,
> Stephan
>

Re: [VOTE] Flink Project Bylaws

[Henry Saputra]

Oh yeah,  +1 LGTM

Thanks for working on this.

- Henry

On Tue, Aug 20, 2019 at 2:17 AM Becket Qin  wrote:

> Thanks for sharing your thoughts, Thomas, Henry and Stephan. I also think
> the committers are supposed to be mature enough to know when a review on
> their own patch is needed.
>
> @Henry, just want to confirm, are you +1 on the proposed bylaws?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Tue, Aug 20, 2019 at 10:54 AM Stephan Ewen  wrote:
>
> > I see it somewhat similar to Henry.
> >
> > Generally, all committers should go for a review by another committer,
> > unless it is a trivial comment or style fix. I personally do that, even
> > though being one of the committers that have been with the project
> longest.
> >
> > For now, I was hoping though that we have a mature enough community that
> > this "soft rule" is enough. Whenever possible, working based on trust
> with
> > soft processes beats working with hard processes. We can still revisit
> this
> > in case we see that it does not work out.
> >
> >
> > On Mon, Aug 19, 2019 at 10:21 PM Henry Saputra 
> > wrote:
> >
> > > One of the perks of being committers is be able to commit code without
> > > asking from another committer. Having said that, I think we rely on
> > > maturity of the committers to know when to ask for reviews and when to
> > > commit directly.
> > >
> > > For example, if someone just change typos on comments or simple rename
> of
> > > internal variables, I think we could trust the committer to safely
> commit
> > > the changes. When the changes will have effect of changing or introduce
> > new
> > > flows of the code, that's when reviews are needed and strongly
> > encouraged.
> > > I think the balance is needed for this.
> > >
> > > PMCs have the ability and right to revert changes in source repo as
> > > necessary.
> > >
> > > - Henry
> > >
> > > On Sun, Aug 18, 2019 at 9:23 PM Thomas Weise  wrote:
> > >
> > > > +0 (binding)
> > > >
> > > > I don't think committers should be allowed to approve their own
> > changes.
> > > I
> > > > would prefer if non-committer contributors can approve committer PRs
> as
> > > > that would encourage more participation in code review and ability to
> > > > contribute.
> > > >
> > > >
> > > > On Fri, Aug 16, 2019 at 9:02 PM Shaoxuan Wang 
> > > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > On Fri, Aug 16, 2019 at 7:48 PM Chesnay Schepler <
> ches...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Although I think it would be a good idea to always cc
> > > > > > priv...@flink.apache.org when modifying bylaws, if anything to
> > speed
> > > > up
> > > > > > the voting process.
> > > > > >
> > > > > > On 16/08/2019 11:26, Ufuk Celebi wrote:
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > – Ufuk
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 14, 2019 at 4:50 AM Biao Liu 
> > > wrote:
> > > > > > >
> > > > > > >> +1 (non-binding)
> > > > > > >>
> > > > > > >> Thanks for pushing this!
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Biao /'bɪ.aʊ/
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, 14 Aug 2019 at 09:37, Jark Wu 
> wrote:
> > > > > > >>
> > > > > > >>> +1 (non-binding)
> > > > > > >>>
> > > > > > >>> Best,
> > > > > > >>> Jark
> > > > > > >>>
> > > > > > >>> On Wed, 14 Aug 2019 at 09:22, Kurt Young 
> > > wrote:
> > > > > > >>>
> > > > > > >>>> +1 (binding)
> > > > > > >>>>
> > > > > > >>>> Best,
> > > > > > >>>> Kurt
> > > > > > >>

Re: [VOTE] Flink Project Bylaws

[Henry Saputra]

One of the perks of being committers is be able to commit code without
asking from another committer. Having said that, I think we rely on
maturity of the committers to know when to ask for reviews and when to
commit directly.

For example, if someone just change typos on comments or simple rename of
internal variables, I think we could trust the committer to safely commit
the changes. When the changes will have effect of changing or introduce new
flows of the code, that's when reviews are needed and strongly encouraged.
I think the balance is needed for this.

PMCs have the ability and right to revert changes in source repo as
necessary.

- Henry

On Sun, Aug 18, 2019 at 9:23 PM Thomas Weise  wrote:

> +0 (binding)
>
> I don't think committers should be allowed to approve their own changes. I
> would prefer if non-committer contributors can approve committer PRs as
> that would encourage more participation in code review and ability to
> contribute.
>
>
> On Fri, Aug 16, 2019 at 9:02 PM Shaoxuan Wang  wrote:
>
> > +1 (binding)
> >
> > On Fri, Aug 16, 2019 at 7:48 PM Chesnay Schepler 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Although I think it would be a good idea to always cc
> > > priv...@flink.apache.org when modifying bylaws, if anything to speed
> up
> > > the voting process.
> > >
> > > On 16/08/2019 11:26, Ufuk Celebi wrote:
> > > > +1 (binding)
> > > >
> > > > – Ufuk
> > > >
> > > >
> > > > On Wed, Aug 14, 2019 at 4:50 AM Biao Liu  wrote:
> > > >
> > > >> +1 (non-binding)
> > > >>
> > > >> Thanks for pushing this!
> > > >>
> > > >> Thanks,
> > > >> Biao /'bɪ.aʊ/
> > > >>
> > > >>
> > > >>
> > > >> On Wed, 14 Aug 2019 at 09:37, Jark Wu  wrote:
> > > >>
> > > >>> +1 (non-binding)
> > > >>>
> > > >>> Best,
> > > >>> Jark
> > > >>>
> > > >>> On Wed, 14 Aug 2019 at 09:22, Kurt Young  wrote:
> > > >>>
> > >  +1 (binding)
> > > 
> > >  Best,
> > >  Kurt
> > > 
> > > 
> > >  On Wed, Aug 14, 2019 at 1:34 AM Yun Tang 
> wrote:
> > > 
> > > > +1 (non-binding)
> > > >
> > > > But I have a minor question about "code change" action, for those
> > > > "[hotfix]" github pull requests [1], the dev mailing list would
> not
> > > >> be
> > > > notified currently. I think we should change the description of
> > this
> > >  action.
> > > >
> > > > [1]
> > > >
> > > >>
> > >
> >
> https://flink.apache.org/contributing/contribute-code.html#code-contribution-process
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: JingsongLee 
> > > > Sent: Tuesday, August 13, 2019 23:56
> > > > To: dev 
> > > > Subject: Re: [VOTE] Flink Project Bylaws
> > > >
> > > > +1 (non-binding)
> > > > Thanks Becket.
> > > > I've learned a lot from current bylaws.
> > > >
> > > > Best,
> > > > Jingsong Lee
> > > >
> > > >
> > > >
> --
> > > > From:Yu Li 
> > > > Send Time:2019年8月13日(星期二) 17:48
> > > > To:dev 
> > > > Subject:Re: [VOTE] Flink Project Bylaws
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks for the efforts Becket!
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Tue, 13 Aug 2019 at 16:09, Xintong Song <
> tonysong...@gmail.com>
> > >  wrote:
> > > >> +1 (non-binding)
> > > >>
> > > >> Thank you~
> > > >>
> > > >> Xintong Song
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Aug 13, 2019 at 1:48 PM Robert Metzger <
> > > >> rmetz...@apache.org>
> > > >> wrote:
> > > >>
> > > >>> +1 (binding)
> > > >>>
> > > >>> On Tue, Aug 13, 2019 at 1:47 PM Becket Qin <
> becket@gmail.com
> > > > wrote:
> > >  Thanks everyone for voting.
> > > 
> > >  For those who have already voted, just want to bring this up
> to
> > >  your
> > >  attention that there is a minor clarification to the bylaws
> > > >> wiki
> > >  this
> > >  morning. The change is in bold format below:
> > > 
> > >  one +1 from a committer followed by a Lazy approval (not
> > > >> counting
> > >  the
> > > >>> vote
> > > > of the contributor), moving to lazy majority if a -1 is
> > > >>> received.
> > > 
> > >  Note that this implies that committers can +1 their own
> commits
> > > >>> and
> > > >> merge
> > > > right away. *However, the committe**rs should use their best
> > > >> judgement
> > > >>> to
> > > > respect the components expertise and ongoing development
> > > >> plan.*
> > > 
> > >  This addition does not really change anything the bylaws meant
> > > >> to
> > > > set.
> > > >> It
> > >  is simply a clarification. If anyone who have casted the vote
> > > > objects,
> > >  please feel free to withdraw the vote.
> > > 

Re: [Discussion] Flink Pulsar Connector

[Henry Saputra]

Here is the link to Apache Flink JIRA issue for this:

https://issues.apache.org/jira/browse/FLINK-9168

- Henry

On Fri, Apr 20, 2018 at 12:08 AM, Sijie Guo  wrote:

> Hi Flinkers,
>
> As discussed with @tzulitai at apache/flink#5845
> , I am starting a discussion
> thread about contributing flink pulsar connectors (including both source
> and sink connectors) from pulsar community to flink project. We'd like to
> see what are people's thoughts about this and how we can proceed for this.
>
> For people who doesn't know about Apache Pulsar, here are some background:
>
> ---
>
> Apache Pulsar (incubating)  is a
> distributed pub/sub messaging system, which provides very flexible
> messaging model - unifying traditional queuing (e.g. SQS, rabbitmq) and
> high-performance streaming (e.g. Kinesis, Kafka) into one pub/sub messaging
> model + api. It is backed by a scalable segment/log storage Apache
> BookKeeper, which provide unbounded stream storage for Pulsar. Because of
> its segment-centric architecture design, Pulsar provides compelling
> unbounded streaming data storage. It is good for both streaming and batch
> processing, which I believe it fits very well into Flink's data processing
> model. Besides that, pulsar has a lot of advanced features going on its
> upcoming 2.0 release, including built-in schema registry, topic compaction,
> regex subscription, and tiered storage
>  Tiered-storage-for-Pulsar-topics>
>  ...
>
> Pulsar was developed by Yahoo since 2012-ish and has been running on
> production for 4+ years, over 10+ data centers and processing/delivering
> billions of messages per day. It was open sourced at 2016. Since it is open
> sourced, it has been adopted by various companies. Nowadays, the pulsar
> slack channel discussion is very active and fast-growing. The community
> currently has about 15 committers.
>
> ---
>
> I happened to work with ZongYang (who is also a pulsar contributor) on
> developing pulsar connectors for flink to satisfy pulsar users requests. We
> would like to contribute the connector work to flink and continue the
> collaboration between flink and pulsar communities. From pulsar community
> perspective, we are also very committed to developing pulsar's ecosystem,
> and willing and dedicated to developing/maintaining flink pulsar
> connectors.
>
> Hope this email thread give you guys enough background of pulsar and clear
> some of the concerns that @tzulitai raised in the jira ticket / pull
> request. Looking forward to any feedback from pulsar community and deep
> collaboration between flink and pulsar community.
>
> Also /cc pulsar dev mailing list (d...@pulsar.incubator.apache.org). If
> there are any questions, pulsar devs can also help to answer.
>
> Thanks,
> Sijie
>

Re: [VOTE] Release 1.3.3, release candidate #2

[Henry Saputra]

Signature file looks good
Hash files look good
LICENSE file exists
NOTICE file exists
Source artifact build and local tests passed

+1



On Wed, Mar 14, 2018 at 3:26 AM, Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on release candidate #2 for Flink 1.3.3, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code branch “release-1.3.3-rc2” [5],
> * website pull request listing the new release [6].
> * A complete list of all new commits in release-1.3.3-rc2, since
> release-1.3.2 [7]
>
> This release candidate contains fixes for only the following issues:
> [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase
> [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint.
> checkpointsMatch()
> [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in
> infinite loop
> [FLINK-7783] Don’t always remove checkpoints in
> ZooKeeperCompletedCheckpointStore#recover()
>
> Since the last candidate was cancelled only due to incorrect binaries in
> the source artifacts, I think we can also have a shorter voting period for
> RC2.
>
> Please test the release and vote for the release candidate before Thursday
> (March 15th), 7pm CET.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Please let me know if you disagree with the shortened voting time.
>
> Thanks,
> Gordon
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12315522&version=12341142
> [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc2/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1151
> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=
> shortlog;h=refs/heads/release-1.3.3-rc2
> [6] https://github.com/apache/flink-web/pull/104
> [7]
> - 90559b5413455d9d0f2b61c389a60e26e5c87800 [hotfix] Properly delete temp
> flink dir in create_source_release.sh
> - 99c0353a34c09af5bedb73f525f691dd7e78fcdd [hotfix] Ensure pristine
> release in tools/releasing/create_source_release.sh
> - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify ZooKeeper
> checkpoint store behaviour with ITCase
> - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare
> checkpoints with order in CompletedCheckpoint.checkpointsMatch()
> - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix
> ZookeeperCompleted checkpoint store can get stuck in infinite loop
> - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always
> remove checkpoints in ZooKeeperCompletedCheckpointStore#recover()
>
>

Re: [ANNOUNCE] New Flink PMC member: Tzu-Li (Gordon) Tai

[Henry Saputra]

Welcome and congrats!

On Mon, Jul 10, 2017 at 9:24 AM Fabian Hueske  wrote:

> Congrats Gordon!
>
> Cheers, Fabian
>
> 2017-07-10 17:03 GMT+02:00 jincheng sun :
>
> > Hi Gordon, Congratulations !!!
> >
> > Cheers,
> > Jincheng
> >
> > 2017-07-10 22:44 GMT+08:00 Robert Metzger :
> >
> > > Hi Everybody,
> > >
> > > On behalf of the Flink PMC, I'm very excited to announce Gordon as the
> > > latest addition to the Flink PMC.
> > >
> > > Gordon is a very active community member, helping with a lot of the
> > release
> > > tasks, project discussions and of course work on the codebase itself.
> > >
> > >
> > > Regards,
> > > Robert
> > >
> >
>

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Henry Saputra]

Congrats and welcome!

On Thu, Jun 22, 2017 at 8:26 AM, Till Rohrmann  wrote:

> Congrats Shaoxuan :-)
>
> Cheers,
> Till
>
> On Thu, Jun 22, 2017 at 3:23 PM, Shaoxuan Wang 
> wrote:
>
> > Thanks everyone.
> > I will do my best, and let's keep Flink forwarding!
> >
> > Regards,
> > Shaoxuan
> >
> > On Thu, Jun 22, 2017 at 11:21 AM, Jark Wu  wrote:
> >
> > > Congratulations Shaoxuan!
> > >
> > > Regards,
> > > Jark
> > >
> > > 2017-06-22 11:09 GMT+08:00 JingsongLee :
> > >
> > > > Congrats!
> > > > Best, Jingsong Lee
> > > > 
> > > --From:Andrew
> > > > Psaltis Time:2017 Jun 22 (Thu)
> 11:06To:dev <
> > > > dev@flink.apache.org>Subject:Re: [ANNOUNCE] New Flink committer
> > Shaoxuan
> > > > Wang
> > > > Congrats Shaoxuan!
> > > >
> > > > On Wed, Jun 21, 2017 at 10:28 PM, Zhuoluo Yang  > > > alibaba-inc.com>
> > > > wrote:
> > > >
> > > > > Congrats!
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Zhuoluo 😀
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 在 2017年6月22日，上午10:14，Haohui Mai  写道：
> > > > >
> > > > > Congrats!
> > > > > On Thu, Jun 22, 2017 at 9:58 AM SHI Xiaogang <
> shixiaoga...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > Congrats, Shaoxuan
> > > > >
> > > > > Regards,
> > > > > Xiaogang
> > > > >
> > > > > 2017-06-22 9:08 GMT+08:00 jincheng sun :
> > > > >
> > > > > Congratulations Shaoxuan.
> > > > >
> > > > >
> > > > > 2017-06-22 8:56 GMT+08:00 Zhangrucong :
> > > > >
> > > > > Congrats Shaoxuan!
> > > > >
> > > > > -邮件原件-
> > > > > 发件人: Fabian Hueske [mailto:fhue...@gmail.com ]
> > > > > 发送时间: 2017年6月22日 4:19
> > > > > 收件人: dev@flink.apache.org
> > > > > 主题: [ANNOUNCE] New Flink committer Shaoxuan Wang
> > > > >
> > > > > Hi everybody,
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang
> > has
> > > > > accepted the invitation of the PMC to become a Flink committer.
> > > > >
> > > > > Shaoxuan has contributed several major features to the Table API /
> > SQL
> > > > >
> > > > > and
> > > > >
> > > > > is very engaged in discussions about the design of new features and
> > the
> > > > > future direction of Flink's relational APIs.
> > > > >
> > > > > Please join in me congratulating Shaoxuan for becoming a Flink
> > > > >
> > > > > committer.
> > > > >
> > > > >
> > > > > Thanks, Fabian
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Andrew
> > > >
> > > > Subscribe to my book: Streaming Data 
> > > > 
> > > > twiiter: @itmdata  user?screen_name=itmdata>
> > > >
> > > >
> > >
> >
>

Re: FlinkML on slack

[Henry Saputra]

Great news, Theodore

+1 =)

On Thu, Jun 22, 2017 at 9:25 AM, Theodore Vasiloudis <
theodoros.vasilou...@gmail.com> wrote:

> Hello all,
>
> We've created an app to automate the invite process, now you can just use
> the following link
> to get an invite to the FlinkML Slack group:
>
> https://flinkml-invites.herokuapp.com/
>
> Regards,
> Theodore
>
> On Tue, Jun 20, 2017 at 8:45 AM, Stavros Kontopoulos <
> st.kontopou...@gmail.com> wrote:
>
> > Sebastian Jark Shaoxuan done.
> > Stavros
> >
> > On Tue, Jun 20, 2017 at 11:09 AM, Sebastian Schelter <
> > ssc.o...@googlemail.com> wrote:
> >
> > > I'd also like to get an invite to this slack, my email is
> s...@apache.org
> > >
> > > Best,
> > > Sebastian
> > >
> > > 2017-06-20 8:37 GMT+02:00 Jark Wu :
> > >
> > > > Hi, Stravros:
> > > > Could you please invite me to the FlinkML slack channel as well? My
> > email
> > > > is: imj...@gmail.com
> > > >
> > > > Thanks,
> > > > Jark
> > > >
> > > > 2017-06-20 13:58 GMT+08:00 Shaoxuan Wang :
> > > >
> > > > > Hi Stavros,
> > > > > Can I get an invitation for the slack channel.
> > > > >
> > > > > Thanks,
> > > > > Shaoxuan
> > > > >
> > > > >
> > > > > On Thu, Jun 8, 2017 at 3:56 AM, Stavros Kontopoulos <
> > > > > st.kontopou...@gmail.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We took the initiative to create the organization for FlinkML on
> > > slack
> > > > > > (thnx Eron).
> > > > > > There is now a channel for model-serving
> > > > > >  1CjWL9aLxPrKytKxUF5c3ohs0ickp0
> > > > > > fdEXPsPYPEywsE/edit#>.
> > > > > > Another is coming for flink-jpmml.
> > > > > > You are invited to join the channels and the efforts. @Gabor
> @Theo
> > > > please
> > > > > > consider adding channels for the other efforts there as well.
> > > > > >
> > > > > > FlinkMS on Slack  (
> > > > > https://flinkml.slack.com/)
> > > > > >
> > > > > > Details for the efforts here: Flink Roadmap doc
> > > > > >  1afQbvZBTV15qF3vobVWUjxQc49h3U
> > > > > > d06MIRhahtJ6dw/edit#>
> > > > > >
> > > > > > Github  (https://github.com/FlinkML)
> > > > > >
> > > > > >
> > > > > > Stavros
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Dawid Wysakowicz

[Henry Saputra]

Congrats and welcome! =)

- Henry

On Mon, Jun 19, 2017 at 6:55 PM, SHI Xiaogang 
wrote:

> Congrats  Dawid.
> Great thanks for your contribution!
>
> Xiaogang
>
> 2017-06-19 18:52 GMT+08:00 Dawid Wysakowicz :
>
> > Thank you all for the warm welcome. I will do my best to be as helpful as
> > possible.
> >
>

Re: FlinkML on slack

[Henry Saputra]

Hi Stavros,

Could you also send me invite to the Slack?

My email is hsapu...@apache.org

Thanks,

Henry


On Thu, Jun 8, 2017 at 2:21 AM, Stavros Kontopoulos <
st.kontopou...@gmail.com> wrote:

> Hi Aljoscha,
>
> Slack is invite only to the best of my knowledge, I just sent you an
> invitation.
>
> Best,
> Stavros
>
>
> On Thu, Jun 8, 2017 at 11:31 AM, Aljoscha Krettek 
> wrote:
>
> > Hi,
> >
> > Is the slack invite based? If yes, could you please send me one?
> >
> > Best,
> > Aljoscha
> >
> > > On 7. Jun 2017, at 21:56, Stavros Kontopoulos <
> st.kontopou...@gmail.com>
> > wrote:
> > >
> > > Hi all,
> > >
> > > We took the initiative to create the organization for FlinkML on slack
> > > (thnx Eron).
> > > There is now a channel for model-serving
> > >  > fdEXPsPYPEywsE/edit#>.
> > > Another is coming for flink-jpmml.
> > > You are invited to join the channels and the efforts. @Gabor @Theo
> please
> > > consider adding channels for the other efforts there as well.
> > >
> > > FlinkMS on Slack  (
> > https://flinkml.slack.com/)
> > >
> > > Details for the efforts here: Flink Roadmap doc
> > >  > d06MIRhahtJ6dw/edit#>
> > >
> > > Github  (https://github.com/FlinkML)
> > >
> > >
> > > Stavros
> >
> >
>

Re: Official Flink Docker images

[Henry Saputra]

We already have a Dockerfile in our source repo as part of simple test:
  https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
but it was never automatically build by our build system AFAIK.

- Henry

On Tue, May 16, 2017 at 8:58 AM, Robert Metzger  wrote:

> How hard would it be to integrate the docker images into the Flink release
> process?
>
> Ideally, we could provide something like a staging directory for the docker
> images, so that we can include them into the vote.
> Once the vote has passed, the images will be made public through docker hub
> and apache.
>
> On Tue, May 16, 2017 at 3:12 PM, Till Rohrmann 
> wrote:
>
> > Great to hear Ismaël. This will make running Flink even easier :-)
> >
> > Cheers,
> > Till
> >
> > On Tue, May 16, 2017 at 1:38 PM, Ismaël Mejía  wrote:
> >
> > > As a follow up for this thread, the docker official images are out
> > > since last week ago so you please guys go ahead and try them. The blog
> > > post that presents them was published today.
> > >
> > > https://flink.apache.org/news/2017/05/16/official-docker-image.html
> > >
> > > On Sun, May 7, 2017 at 3:51 PM, Ismaël Mejía 
> wrote:
> > > > Sure we would do it. I will sync with Patrick to do this quickly.
> > > >
> > > > On May 5, 2017 3:45 PM, "Robert Metzger" 
> wrote:
> > > >>
> > > >> Thanks a lot for your efforts!
> > > >>
> > > >> Maybe we can even put a small blog post on the Flink post once its
> > > >> available on docker hub.
> > > >>
> > > >> On Fri, Apr 28, 2017 at 11:00 AM, Ismaël Mejía 
> > > wrote:
> > > >>
> > > >> > Hello,
> > > >> >
> > > >> > I am absolutely happy to see that this is finally happening!
> > > >> > We have a really neat image right now and it is great that it will
> > be
> > > >> > soon so easy to use.
> > > >> >
> > > >> > One extra thing to mention is that Flink will have now two docker
> > > >> > images, one based on debian and the other one based on Alpine as
> > most
> > > >> > official java-based projects do.
> > > >> >
> > > >> > In the future we expect to improve the documentation on how to use
> > the
> > > >> > image with kubernetes and continue improving the actual
> > documentation
> > > >> > with docker. If anyone wants to join to also document something or
> > add
> > > >> > any improvement/feature you need, you are all welcome.
> > > >> >
> > > >> > Finally, I would also like to thank Maximilian Michels which
> > > >> > contributed and reviewed some of my early changes on the image.
> > > >> >
> > > >> > Regards,
> > > >> > Ismaël
> > > >> >
> > > >> > ps. We will 'announce' when the official images are available on
> > > >> > docker hub, so everyone can start to use them.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, Apr 27, 2017 at 1:38 PM, Patrick Lucas
> > > >> >  wrote:
> > > >> > > I've been informed that images don't make it through the list!
> > > >> > >
> > > >> > > You can see the aforementioned squirrel here
> > > >> > >  > > >> > uploads/8/7/6/3/9/ar12988558393678.JPG>
> > > >> > > .
> > > >> > >
> > > >> > > --
> > > >> > > Patrick Lucas
> > > >> > >
> > > >> > > On Thu, Apr 27, 2017 at 12:21 PM, Patrick Lucas <
> > > >> > patr...@data-artisans.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> As part of an ongoing effort to improve the experience of using
> > > Flink
> > > >> > >> on
> > > >> > >> Docker, some work has been done over the last two months to
> > publish
> > > >> > >> official Flink Docker images to Docker Hub. The goal in the
> short
> > > >> > >> term
> > > >> > is
> > > >> > >> to make running a simple Flink cluster (almost) as easy as
> > running
> > > >> > docker
> > > >> > >> run flink. In the long term, we would like these images to be
> > good
> > > >> > enough
> > > >> > >> to use in production directly, or as base images for use in an
> > > >> > >> existing
> > > >> > >> Docker workflow.
> > > >> > >>
> > > >> > >> Flink 1.2.1 has some fixes over the last few releases that make
> > > >> > >> running
> > > >> > it
> > > >> > >> on Docker nicer—and in some cases, possible. Notably,
> FLINK-2821
> > > >> > >>  allowed
> > > Dockerized
> > > >> > >> Flink to run across multiple hosts, and FLINK-4326
> > > >> > >>  added an
> > > option to
> > > >> > run
> > > >> > >> Flink in the foreground, which is greatly preferred when
> running
> > in
> > > >> > Docker.
> > > >> > >>
> > > >> > >> We (Ismaël Mejía and myself, with some discussion with Stephan
> > > Ewen)
> > > >> > >> decided it made sense to bring the actual Dockerfiles outside
> of
> > > the
> > > >> > Apache
> > > >> > >> Flink git repo, primarily to conform with every other Apache
> > > project
> > > >> > that
> > > >> > >> has official images, but also because these scripts logically
> > exist
> > > >> > >> decoupled from any particular Flink version. They are still
> > > >> > Apache-licensed

Re: [DISCUSS] Feature Freeze

[Henry Saputra]

The FLINK-6364  seems
need an accompanying FLIP [1] to help review.

I dont see for this one in the list of existing proposals.

- Henry

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

On Thu, Apr 27, 2017 at 7:39 AM, Gyula Fóra  wrote:

> Hi Ufuk,
>
> Thanks for starting this discussion!
>
> One feature that immediately comes to my mind is incremental checkpointing
> given it's production impact.
>
> https://issues.apache.org/jira/browse/FLINK-6364
>
> It would be good to get some better understanding how the implementation
> effort is going and whether it is reasonable to expect this to be included
> in 1.3.
>
> Regards,
> Gyula
>
> Ufuk Celebi  ezt írta (időpont: 2017. ápr. 27., Cs,
> 16:24):
>
> > Hey devs! :-)
> >
> > We decided to follow a time-based release model with the upcoming 1.3
> > release and the planned feature freeze is on Monday, May 1st.
> >
> > I wanted to start a discussion to get a quick overview of the current
> > state of things.
> >
> > - Is everyone on track and aware of the feature freeze? ;)
> > - Are there any major features we want in 1.3 that have not been merged
> > yet?
> > - Do we need to extend the feature freeze, because of an important
> feature?
> >
> > Would be great to gather a list of features/PRs that we want in the
> > 1.3 release. This could be a good starting point for the release
> > manager (@Robert?).
> >
> > Best,
> >
> > Ufuk
> >
>

Re: [RESULT] [VOTE] Release Apache Flink 1.2.1 (RC2)

[Henry Saputra]

Thanks for being awesome RE, as always, Robert!

- Henry

On Tue, Apr 25, 2017 at 2:09 AM, Robert Metzger  wrote:

> I've uploaded the artifacts to the apache mirrors and released the maven
> stuff to central.
>
> While the artifacts are syncing, please review the release announcement:
> https://github.com/apache/flink-web/pull/54
>
> On Tue, Apr 25, 2017 at 10:29 AM, Robert Metzger 
> wrote:
>
> > Okay, I'll then put out the release.
> >
> >
> > The vote to release this package as Flink 1.2.1 has passed with:
> >
> > +1 votes:
> > - Andrew (non-binding)
> > - Gyula (binding)
> > - Till (binding)
> > - Greg (binding)
> > - Henry (binding)
> >
> > No 0 or -1 votes.
> >
> >
> > On Mon, Apr 24, 2017 at 2:11 PM, Aljoscha Krettek 
> > wrote:
> >
> >> Agreed, as I said above:
> >>
> >> >>>> I have the fix ready but we can do that in Flink 1.2.2. Very
> quickly,
> >> >>>> though.
> >>
> >> Best,
> >> Aljoscha
> >>
> >> > On 24. Apr 2017, at 13:19, Ufuk Celebi  wrote:
> >> >
> >> > I agree with Till and would NOT cancel this release. It has been
> >> > delayed already quite a bit already and the feature freeze for 1.3.0
> >> > is coming up (i.e. most contributors will be busy and not spend a lot
> >> > of time for 1.2.1).
> >> >
> >> > – Ufuk
> >> >
> >> >
> >> > On Mon, Apr 24, 2017 at 9:31 AM, Till Rohrmann 
> >> wrote:
> >> >> If this bug was already present in 1.2.0, then I guess not many users
> >> have
> >> >> used this feature. Otherwise we would have seen complaints on the
> >> mailing
> >> >> list.
> >> >>
> >> >> From the JIRA issue description, it looks as if we have to fix it for
> >> 1.3.0
> >> >> anyway. What about fixing it this week and then backporting it to the
> >> 1.2.1
> >> >> branch?
> >> >>
> >> >> Cheers,
> >> >> Till
> >> >>
> >> >> On Mon, Apr 24, 2017 at 8:12 AM, Aljoscha Krettek <
> aljos...@apache.org
> >> >
> >> >> wrote:
> >> >>
> >> >>> It means that users cannot restore from 1.2.0 to 1.2.0, 1.2.0 to
> >> 1.2.1, or
> >> >>> 1.2.1 to 1.2.1. However, this only happens when using the
> >> >>> CheckpointedRestoring interface, which you have to do when you want
> to
> >> >>> migrate away form the Checkpointed interface.
> >> >>>
> >> >>> tl;dr It’s not a new bug but one that was present in 1.2.0 already.
> >> >>>
> >> >>>> On 23. Apr 2017, at 21:16, Robert Metzger 
> >> wrote:
> >> >>>>
> >> >>>> @all: I'm sorry for being a bad release manager this time. I'm not
> >> >>> spending
> >> >>>> much time online these days. I hope to increase my dev@ list
> >> activity a
> >> >>>> little bit next week.
> >> >>>>
> >> >>>> @Aljoscha:
> >> >>>> Does this mean that users can not upgrade from 1.2.0 to 1.2.1 ?
> >> >>>>
> >> >>>> Can we make the minor versions easily compatible?
> >> >>>> If so, I would prefer to cancel this release as well and do another
> >> one.
> >> >>>>
> >> >>>>
> >> >>>> On Fri, Apr 21, 2017 at 12:04 PM, Aljoscha Krettek <
> >> aljos...@apache.org>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> There is this (somewhat pesky) issue:
> >> >>>>> - https://issues.apache.org/jira/browse/FLINK-6353: Restoring
> using
> >> >>>>> CheckpointedRestoring does not work from 1.2 to 1.2
> >> >>>>>
> >> >>>>> I have the fix ready but we can do that in Flink 1.2.2. Very
> >> quickly,
> >> >>>>> though.
> >> >>>>>
> >> >>>>>> On 20. Apr 2017, at 17:20, Henry Saputra <
> henry.sapu...@gmail.com>
> >> >>>>> wrote:
> >> >>>>>>
> >> >>>>>> LICENSE file exists
> >> >>>>>> NOTICE file looks good
> &

Re: [DISCUSS] Code style / checkstyle

[Henry Saputra]

Cool! So, it begins =)

- Henry

On Wed, Apr 26, 2017 at 7:30 AM, Aljoscha Krettek 
wrote:

> I merged the stricter checkstyle for flink-streaming-java today. (Sans
> checking for right curly braces)
>
> > On 18. Apr 2017, at 22:17, Chesnay Schepler  wrote:
> >
> > +1 to use earth rotation as the new standard time unit. Maybe more
> importantly, I'm absolutely in favor of merging it.
> >
> > On 18.04.2017 20:39, Aljoscha Krettek wrote:
> >> I rebased the PR [1] on current master. Is there any strong objection
> against merging this (minus the two last commits which introduce
> curly-brace-style checking). If not, I would like to merge this after two
> earth rotations, i.e. after all the time zones have had some time to react.
> >>
> >> The complete set of checks has been listed by Chesnay (via Greg) before
> but the gist of it is that we only add common-sense checks that most people
> should be able to agree upon so that we avoid edit wars (especially when it
> comes to whitespace, import order and Javadoc paragraph styling).
> >>
> >> [1] https://github.com/apache/flink/pull/3567
> >>> On 5. Apr 2017, at 23:54, Chesnay Schepler  wrote:
> >>>
> >>> I agree to just allow both. While I definitely prefer 1) i can see why
> someone might prefer 2).
> >>>
> >>> Wouldn't want to delay this anymore; can't find to add this to
> flink-metrics and flink-python...
> >>>
> >>> On 03.04.2017 18:33, Aljoscha Krettek wrote:
> >>>> I think enough people did already look at the checkstyle rules
> proposed in the PR.
> >>>>
> >>>> On most of the rules reaching consensus is easy so I propose to
> enable all rules except those regarding placement of curly braces and
> control flow formatting. The two styles in the Flink code base are:
> >>>>
> >>>> 1)
> >>>> if () {
> >>>> } else {
> >>>> }
> >>>>
> >>>> try {
> >>>> } catch () {
> >>>> }
> >>>>
> >>>> and
> >>>>
> >>>> 2)
> >>>>
> >>>> if () {
> >>>> }
> >>>> else {
> >>>> }
> >>>>
> >>>> try {
> >>>> }
> >>>> catch () {
> >>>> }
> >>>>
> >>>> I think it’s hard to reach consensus on these so I suggest to keep
> allowing both styles.
> >>>>
> >>>> Any comments very welcome! :-)
> >>>>
> >>>> Best,
> >>>> Aljoscha
> >>>>> On 19. Mar 2017, at 17:09, Aljoscha Krettek 
> wrote:
> >>>>>
> >>>>> I played around with this over the week end and it turns out that
> the required changes in flink-streaming-java are not that big. I created a
> PR with a proposed checkstyle.xml and the required code changes:
> https://github.com/apache/flink/pull/3567 <https://github.com/apache/
> flink/pull/3567>. There’s a longer description of the style in the PR.
> The commits add continuously more invasive changes so we can start with the
> more lightweight changes if we want to.
> >>>>>
> >>>>> If we want to go forward with this I would also encourage other
> people to use this for different modules and see how it turns out.
> >>>>>
> >>>>> Best,
> >>>>> Aljoscha
> >>>>>
> >>>>>> On 18 Mar 2017, at 08:00, Aljoscha Krettek  <mailto:aljos...@apache.org>> wrote:
> >>>>>>
> >>>>>> I added an issue for adding a custom checkstyle.xml for
> flink-streaming-java so that we can gradually add checks:
> https://issues.apache.org/jira/browse/FLINK-6107 <
> https://issues.apache.org/jira/browse/FLINK-6107>. I outlined the
> procedure in the Jira. We can use this as a pilot project and see how it
> goes and then gradually also apply to other modules.
> >>>>>>
> >>>>>> What do you think?
> >>>>>>
> >>>>>>> On 6 Mar 2017, at 12:42, Stephan Ewen  se...@apache.org>> wrote:
> >>>>>>>
> >>>>>>> A singular "all reformat in one instant" will cause immense damage
> to the
> >>>>>>> project, in my opinion.
> >>>>>>>
> >>>>>>> - There are so many pull requests that we are having a hard time
> keeping
> >>&

Re: [VOTE] Release Apache Flink 1.2.1 (RC2)

[Henry Saputra]

LICENSE file exists
NOTICE file looks good
Signature files look good
Hash files look good
No 3rd party exes in source artifact
Source compiled and pass tests
Local run work
Run simple job on YARN

+1

- Henry

On Wed, Apr 12, 2017 at 4:06 PM, Robert Metzger  wrote:

> Dear Flink community,
>
> Please vote on releasing the following candidate as Apache Flink version
> 1.2
> .1.
>
> The commit to be voted on:
> 76eba4e0 
> (*http://git-wip-us.apache.org/repos/asf/flink/commit/76eba4e0
> *)
>
> Branch:
> release-1.2.1-rc2
>
> The release artifacts to be voted on can be found at:
> http://people.apache.org/~rmetzger/flink-1.2.1-rc2/
>
>
> The release artifacts are signed with the key with fingerprint D9839159:
> http://www.apache.org/dist/flink/KEYS
>
> The staging repository for this release can be found at:
> *https://repository.apache.org/content/repositories/orgapacheflink-1117
> *
>
> -
>
>
> The vote ends on Tuesday, 1pm CET.
>
> [ ] +1 Release this package as Apache Flink 1.2.1
> [ ] -1 Do not release this package, because ...
>

Re: [VOTE] Release Apache Flink 1.2.1 (RC2)

[Henry Saputra]

This should be the one: https://github.com/aljoscha/FliRTT

On Wed, Apr 19, 2017 at 7:48 AM, Ted Yu  wrote:

> Till:
> A bit curious: where can I find the Flirrt tool ?
>
> Thanks
>
> On Wed, Apr 19, 2017 at 5:24 AM, Till Rohrmann 
> wrote:
>
> > +1 (binding) for the release
> >
> > - checked checksums and signature
> > - no dependencies added or removed
> > - build Flink with Hadoop 2.7.1 and Scala 2.11 from sources and ran all
> > tests
> > - Ran Flirrt locally, on standalone cluster and Yarn with Hadoop 2.7.1
> and
> > Scala 2.10
> >
> > Cheers,
> > Till
> >
> >
> > On Thu, Apr 13, 2017 at 11:27 AM, Gyula Fóra 
> wrote:
> >
> > > Hi,
> > >
> > > Unfortunately I cannot test run the rc as I am on vacation. But we have
> > > been running pretty much the same build (+1-2 commits) in production
> for
> > > some time now.
> > >
> > > +1 from me
> > >
> > > Gyula
> > >
> > > On Thu, Apr 13, 2017, 08:27 Andrew Psaltis 
> > > wrote:
> > >
> > > > +1 -- checked out all code, built with all tests, ran local cluster,
> > > > deployed example streaming jobs
> > > >
> > > > On Thu, Apr 13, 2017 at 2:26 AM, Andrew Psaltis <
> > > psaltis.and...@gmail.com>
> > > > wrote:
> > > >
> > > > > Ted -- I did not see those errors. My environment is:
> > > > > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > > > > 2015-11-10T11:41:47-05:00)
> > > > > Maven home: /usr/local/Cellar/maven/3.3.9/libexec
> > > > > Java version: 1.8.0_121, vendor: Oracle Corporation
> > > > > Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_
> > > > > 121.jdk/Contents/Home/jre
> > > > > Default locale: en_US, platform encoding: UTF-8
> > > > > OS name: "mac os x", version: "10.12.3", arch: "x86_64", family:
> > "mac"
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Apr 13, 2017 at 12:36 AM, Ted Yu 
> > wrote:
> > > > >
> > > > >> I ran test suite where the following failed:
> > > > >>
> > > > >> Failed tests:
> > > > >>   StreamExecutionEnvironmentTest.testDefaultParallelismIsDefaul
> > t:143
> > > > >> expected:<-1> but was:<24>
> > > > >>
> > > > >> StreamExecutionEnvironmentTest.testMaxParallelismMustBeBigge
> > > > >> rEqualParallelism
> > > > >> Expected test to throw an instance of java.lang.
> > > IllegalArgumentException
> > > > >>
> > > > >> StreamExecutionEnvironmentTest.testParallelismMustBeSmallerE
> > > > >> qualMaxParallelism
> > > > >> Expected test to throw an instance of java.lang.
> > > IllegalArgumentException
> > > > >>
> > > > >> This is what I used:
> > > > >>
> > > > >> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > > > >> 2015-11-10T16:41:47+00:00)
> > > > >> Java version: 1.8.0_101, vendor: Oracle Corporation
> > > > >>
> > > > >> Have anyone else seen the above failures ?
> > > > >>
> > > > >> Cheers
> > > > >>
> > > > >> On Wed, Apr 12, 2017 at 4:06 PM, Robert Metzger <
> > rmetz...@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Dear Flink community,
> > > > >> >
> > > > >> > Please vote on releasing the following candidate as Apache Flink
> > > > version
> > > > >> > 1.2
> > > > >> > .1.
> > > > >> >
> > > > >> > The commit to be voted on:
> > > > >> > 76eba4e0 <
> > > > http://git-wip-us.apache.org/repos/asf/flink/commit/76eba4e0>
> > > > >> > (*http://git-wip-us.apache.org/repos/asf/flink/commit/76eba4e0
> > > > >> >  >*)
> > > > >> >
> > > > >> > Branch:
> > > > >> > release-1.2.1-rc2
> > > > >> >
> > > > >> > The release artifacts to be voted on can be found at:
> > > > >> > http://people.apache.org/~rmetzger/flink-1.2.1-rc2/
> > > > >> >
> > > > >> >
> > > > >> > The release artifacts are signed with the key with fingerprint
> > > > D9839159:
> > > > >> > http://www.apache.org/dist/flink/KEYS
> > > > >> >
> > > > >> > The staging repository for this release can be found at:
> > > > >> > *
> > > > https://repository.apache.org/content/repositories/
> orgapacheflink-1117
> > > > >> > <
> > > > https://repository.apache.org/content/repositories/
> orgapacheflink-1117
> > > > >> >*
> > > > >> >
> > > > >> > -
> > > > >> >
> > > > >> >
> > > > >> > The vote ends on Tuesday, 1pm CET.
> > > > >> >
> > > > >> > [ ] +1 Release this package as Apache Flink 1.2.1
> > > > >> > [ ] -1 Do not release this package, because ...
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > Subscribe to my book: Streaming Data 
> > > > > 
> > > > > twiiter: @itmdata  > user?screen_name=itmdata>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Andrew
> > > >
> > > > Subscribe to my book: Streaming Data 
> > > > 
> > > > twiiter: @itmdata  user?screen_name=itmdat

Re: [DISCUSS] Code style / checkstyle

[Henry Saputra]

It is actually Databricks Scala guide to help contributions to Apache Spark
so not really official Spark Scala guide.
The style guide feels bit more like asking people to write Scala in Java
mode so I am -1 to follow the style for Apache Flink Scala if that what you
are recommending.

If the "unification" means ONE style guide for both Java and Scala I would
vote -1 to it because both languages have different semantics and styles to
make them readable and understandable.

We could start with improving the Scala maven style guide to follow more
Scala official style guide [1] and add IntelliJ Idea or Eclipse plugin
style to follow suit.

Java side has bit more strict style check but we could make it tighter but
embracing more Google Java guide closely with minor exceptions

- Henry

[1] http://docs.scala-lang.org/style/

On Mon, Feb 27, 2017 at 11:54 AM, Stavros Kontopoulos <
st.kontopou...@gmail.com> wrote:

> +1 to provide and enforcing a unified code style for both java and scala.
> Unification should apply when it makes sense like comments though.
>
> Eventually code base should be re-factored. I would vote for the one at a
> time module fix apporoach.
> Style guide should be part of any PR review.
>
> We could also have a look at the spark style guide:
> https://github.com/databricks/scala-style-guide
>
> The style code and general guidelines help keep code more readable and keep
> things simple
> with many contributors and different styles of code writing + language
> features.
>
>
> On Mon, Feb 27, 2017 at 8:01 PM, Stephan Ewen  wrote:
>
> > I agree, reformatting 90% of the code base is tough.
> >
> > There are two main issues:
> >   (1) Incompatible merges. This is hard, especially for the folks that
> have
> > to merge the pull requests ;-)
> >
> >   (2) Author history: This is less of an issue, I think. "git log
> > " and "git show  -- " will still work and
> one
> > may have to go one commit back to find out why something was changed
> >
> >
> > What I could image is to do this incrementally. Define the code style in
> > "flink-parent" but do not activate it.
> > Then start with some projects (new projects, plus some others):
> > merge/reject PRs, reformat, activate code style.
> >
> > Piece by piece. This is realistically going to take a long time until it
> is
> > pulled through all components, but that's okay, I guess.
> >
> > Stephan
> >
> >
> > On Mon, Feb 27, 2017 at 1:53 PM, Aljoscha Krettek 
> > wrote:
> >
> > > Just for a bit of context, this is the output of running cloc on the
> > Flink
> > > codebase:
> > > 
> > > ---
> > > Language files  blankcomment
> > > code
> > > 
> > > ---
> > > Java  4609 126825 185428
> > >   519096
> > >
> > > => 704,524 lines of code + comments/javadoc
> > >
> > > When I apply the google style to the Flink code base using
> > > https://github.com/google/google-java-format I get these commit
> > > statistics:
> > >
> > > 4577 files changed, 647645 insertions(+), 622663 deletions(-)
> > >
> > > That is, a change to the Google Code Style would touch roughly over 90%
> > of
> > > all code/comment lines.
> > >
> > > I would like to have a well defined code style, such as the Google Code
> > > style, that has nice tooling and support but I don't think we will ever
> > > convince enough people to do this kind of massive change. Even I think
> > it's
> > > a bit crazy to change 90% of the code base in one commit.
> > >
> > > On Mon, 27 Feb 2017 at 11:10 Till Rohrmann 
> wrote:
> > >
> > > > No, I think that's exactly what people mean when saying "losing the
> > > commit
> > > > history". With the reformatting you would have to go manually through
> > all
> > > > past commits until you find the commit which changed a given line
> > before
> > > > the reformatting.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Sun, Feb 26, 2017 at 6:32 PM, Alexander Alexandrov <
> > > > alexander.s.alexand...@gmail.com> wrote:
> > > >
> > > > > Just to clarify - by "losing the commit history

Re: [DISCUSS] Code style / checkstyle

[Henry Saputra]

Just want to clarify what unify code style here.

Is the intention to have IDE and Maven plugins to have the same check style
rules?

Or are we talking about having ONE code style for both Java and Scala?

- Henry

On Fri, Feb 24, 2017 at 8:08 AM, Greg Hogan  wrote:

> I agree wholeheartedly with Ufuk. We cannot reformat the codebase, cannot
> pause while flushing the PR queue, and won't find a consensus code style.
>
> I think we can create a baseline code style for new and existing
> contributors for which reformatting on changed files will be acceptable for
> PR reviews.
>
> On Fri, Feb 24, 2017 at 5:01 AM, Dawid Wysakowicz <
> wysakowicz.da...@gmail.com> wrote:
>
> > The problem with code style when it is not enforced is that it will be a
> > matter of luck to what parts of files / new files will it be applied.
> When
> > the code style is not applied to whole file, it is pretty much useless
> > anyway. You would need to manually select just the fragments one is
> > changing. The benefits of having code style and enforcing it I see are:
> >  - being able to apply autoformatter, which speeds up writing code
> >  - it would make reviewing PRs easier as e.g. there would be line length
> > limit applied etc. which will make line breaking more reader friendly.
> >
> > Though I think if a consensus is not reachable it would be good to once
> and
> > forever decide that we don't want a code style and checkstyle.
> >
> > 2017-02-24 10:51 GMT+01:00 Ufuk Celebi :
> >
> > > On Fri, Feb 24, 2017 at 10:46 AM, Fabian Hueske 
> > wrote:
> > > > I agree with Till that encouraging a code style without enforcing it
> > does
> > > > not make a lot of sense.
> > > > If we enforce it, we need to touch all files and PRs.
> > >
> > > I think it makes sense for new contributors to have a starting point
> > > without enforcing anything (I do agree that we are past the point to
> > > reach consensus on a style and enforcement ;-)).
> > >
> >
>

Re: [ANNOUNCE] Welcome Stefan Richter as a new committer

[Henry Saputra]

Congrats and welcome!

On Fri, Feb 10, 2017 at 7:45 AM, Tzu-Li (Gordon) Tai 
wrote:

> Great news! Welcome Stefan :-D
>
>
> On February 10, 2017 at 11:36:14 PM, Aljoscha Krettek (aljos...@apache.org)
> wrote:
>
> Welcome! :-)
>
> On Fri, 10 Feb 2017 at 16:10 Till Rohrmann  wrote:
>
> > Great to have you on board as a committer Stefan :-)
> >
> > On Fri, Feb 10, 2017 at 3:32 PM, Greg Hogan  wrote:
> >
> > > Welcome, Stefan, and thank you for your contributions!
> > >
> > > On Fri, Feb 10, 2017 at 5:00 AM, Ufuk Celebi  wrote:
> > >
> > > > Hey everyone,
> > > >
> > > > I'm very happy to announce that the Flink PMC has accepted Stefan
> > > > Richter to become a committer of the Apache Flink project.
> > > >
> > > > Stefan is part of the community for almost a year now and worked on
> > > > major features of the latest 1.2 release, most notably rescaling and
> > > > backwards compatibility of program state.
> > > >
> > > > Please join me in welcoming Stefan. :-)
> > > >
> > > > – Ufuk
> > > >
> > >
> >
>

Re: [ANNOUNCE] Welcome Jark Wu and Kostas Kloudas as committers

[Henry Saputra]

Awesome! Congrats and Welcome, Jark and Kostas K.

- Henry

On Tue, Feb 7, 2017 at 12:16 PM, Fabian Hueske  wrote:

> Hi everybody,
>
> I'm very happy to announce that Jark Wu and Kostas Kloudas accepted the
> invitation of the Flink PMC to become committers of the Apache Flink
> project.
>
> Jark and Kostas are longtime members of the Flink community.
> Both are actively driving Flink's development and contributing to its
> community in many ways.
>
> Please join me in welcoming Kostas and Jark as committers.
>
> Fabian
>

Re: Travis CI

[Henry Saputra]

Thanks for the insight, Robert.

I have new MacbookPro so I think should have enough CPU to build.

Seemed like the long time taken mostly on shading and pulling the world
(dependencies)

Do you have anything in your settings.xml to set the updatePolicy for
central repo?

- Henry

On Wed, Dec 14, 2016 at 2:34 AM, Robert Metzger  wrote:

> On my machine, I need ~10 minutes to do a clean install without tests. Not
> sure what is causing the slow builds in your environment.
> I think our builds are both IO (shading) and CPU (Scala compiler)
> intensive. If one is not well equipped you'll have these build times.
>
> I contacted the Travis support to hear if there's anything we can do
> regarding the time limit.
>
> On Wed, Dec 14, 2016 at 3:52 AM, Henry Saputra 
> wrote:
>
> > Building Flink from root now takes long time due to more and more modules
> > that we have =(
> > For me if takes about 50 minutes or more when doing "mvn clean install
> > -DskipTests"
> >
> > Should we start adding Maven profiles to help exclude some Maven modules,
> > like connectors and flink-libraries by default ?
> >
> > Do you guys see the same behavior?
> >
> > - Henry
> >
> > On Thu, Nov 10, 2016 at 1:35 PM, Ufuk Celebi  wrote:
> >
> > > I also just noticed it today. We used to work with the 120 minutes
> limit
> > > and the builds took way longer as you said. I don't know what's going
> on
> > > here...
> > >
> > > It might be related to some issues they reported a few hours ago (
> > > https://twitter.com/traviscistatus), but I can't tell.
> > >
> > > I really hope that the new env is temporary (although the reduced build
> > > time is indeed nice ;)). Let's monitor this...
> > >
> > > – Ufuk
> > >
> > > On 10 November 2016 at 22:15:51, Greg Hogan (c...@greghogan.com)
> wrote:
> > > > We're getting the dreaded "The job exceeded the maximum time limit
> for
> > > > jobs, and has been terminated." error for some recent Travis-CI
> builds.
> > > > https://travis-ci.org/apache/flink/builds/174615801
> > > >
> > > > The docs state that termination will occur when "A job takes longer
> > than
> > > 50
> > > > minutes on travis-ci.org", which applies to Flink as well as user
> > GitHub
> > > > accounts.
> > > > https://docs.travis-ci.com/user/customizing-the-build/#
> Build-Timeouts
> > > >
> > > > The jobs are running much faster now, as our builds have consistently
> > > taken
> > > > over an hour and up to an hour and a half. The following recently
> > > > successful build looks to be a mix of fast and slow jobs.
> > > > https://travis-ci.org/apache/flink/builds/174596630
> > > >
> > > > Greg
> > > >
> > >
> > >
> >
>

Re: Travis CI

[Henry Saputra]

Building Flink from root now takes long time due to more and more modules
that we have =(
For me if takes about 50 minutes or more when doing "mvn clean install
-DskipTests"

Should we start adding Maven profiles to help exclude some Maven modules,
like connectors and flink-libraries by default ?

Do you guys see the same behavior?

- Henry

On Thu, Nov 10, 2016 at 1:35 PM, Ufuk Celebi  wrote:

> I also just noticed it today. We used to work with the 120 minutes limit
> and the builds took way longer as you said. I don't know what's going on
> here...
>
> It might be related to some issues they reported a few hours ago (
> https://twitter.com/traviscistatus), but I can't tell.
>
> I really hope that the new env is temporary (although the reduced build
> time is indeed nice ;)). Let's monitor this...
>
> – Ufuk
>
> On 10 November 2016 at 22:15:51, Greg Hogan (c...@greghogan.com) wrote:
> > We're getting the dreaded "The job exceeded the maximum time limit for
> > jobs, and has been terminated." error for some recent Travis-CI builds.
> > https://travis-ci.org/apache/flink/builds/174615801
> >
> > The docs state that termination will occur when "A job takes longer than
> 50
> > minutes on travis-ci.org", which applies to Flink as well as user GitHub
> > accounts.
> > https://docs.travis-ci.com/user/customizing-the-build/#Build-Timeouts
> >
> > The jobs are running much faster now, as our builds have consistently
> taken
> > over an hour and up to an hour and a half. The following recently
> > successful build looks to be a mix of fast and slow jobs.
> > https://travis-ci.org/apache/flink/builds/174596630
> >
> > Greg
> >
>
>

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

[Henry Saputra]

HI David,

Thanks so much for the interest to contribute to Apache Flink.

To help review and DISCUSS for new feature in Flink, please do submit FLIP
[1] proposal.

It will help the PMCs managing the new feature proposals and keep resources
with ASF realm.


Thanks,

Henry

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

On Sun, Sep 11, 2016 at 7:56 PM, Jeffery David  wrote:

> Hi ALL,
>
> Recently, we have designed and implemented a new proposal in FLINK to
> support Asynchronous Operation while streaming. The main feature of this
> proposal is to introduce async i/o operation in FLINK to boost the TPS of
> streaming job without delaying the checkpoint, and provide an easy way for
> the FLINK users to implement their async i/o codes in FLINK job.
>
> Here is the link to Google Doc:
> https://docs.google.com/document/d/1Lr9UYXEz6s6R_
> 3PWg3bZQLF3upGaNEkc0rQCFSzaYDI/edit
>
> Any feedback is appreciated.
>
> Thanks,
> David
>

Re: Posting Flink Cluster questions

[Henry Saputra]

HI,

Please send email to dev-subscr...@flink.apache.org and follow the
instructions from the auto reply to subscribe to Apache Flink dev@ list to
participate in the discussions.

Thanks,

Henry

On Mon, Sep 12, 2016 at 2:25 PM, amir bahmanyari 
wrote:

>
> Hi,
> Whats the link to post a FlinkCluster load-balancing question pls?
> Thanks+regards,
> Amir-
>

Re: Reducing the JIRA PR message verbosity

[Henry Saputra]

Sorry for late reply, but +1 too

This should still provide tracking of discussion from Github PR and reduce
the noise at the same time.

Thanks for working on it, Max.

- Henry

On Fri, Sep 2, 2016 at 8:42 AM, Till Rohrmann  wrote:

> +1
>
> On Fri, Sep 2, 2016 at 4:20 PM, Fabian Hueske  wrote:
>
> > +1
> >
> > Thanks Max!
> >
> > 2016-09-02 15:20 GMT+02:00 Stephan Ewen :
> >
> > > Sounds good to me!
> > >
> > > On Fri, Sep 2, 2016 at 3:08 PM, Maximilian Michels 
> > wrote:
> > >
> > > > If there are no objections, I will contact Infra to change the GitHub
> > > > JIRA notifications as follows:
> > > >
> > > > Jira comments section
> > > >   - initial PR description
> > > >   - comments of the main GitHub thread
> > > >
> > > > Jira Work Log
> > > >   - all diff comments
> > > >
> > > >
> > > > On Mon, Aug 29, 2016 at 6:58 PM, Maximilian Michels 
> > > > wrote:
> > > > >> Will work log updates still lead to notifications on the mailing
> > list?
> > > > >
> > > > > I don't know but it is most likely configurable. I would prefer if
> we
> > > > > didn't get notifications for work log items because these would
> again
> > > > > be duplicates of the GitHub notifications.
> > > > >
> > > > > On Mon, Aug 29, 2016 at 5:39 PM, Neelesh Salian <
> > nsal...@cloudera.com>
> > > > wrote:
> > > > >> Thanks Max for coordinating the effort.
> > > > >> +1 as well.
> > > > >>
> > > > >>
> > > > >> On Mon, Aug 29, 2016 at 6:57 AM, Robert Metzger <
> > rmetz...@apache.org>
> > > > wrote:
> > > > >>
> > > > >>> +1
> > > > >>>
> > > > >>> I didn't know that we can put the comments into the work log.
> > > > >>> Will work log updates still lead to notifications on the mailing
> > > list?
> > > > >>>
> > > > >>> On Mon, Aug 29, 2016 at 11:52 AM, Maximilian Michels <
> > m...@apache.org
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > From what I understand so far, the message mirroring can be
> > > adjusted
> > > > >>> > in the follow parts:
> > > > >>> >
> > > > >>> > 1) GitHub PR description
> > > > >>> > 2) GitHub Diff comments
> > > > >>> > 3) GitHub Main thread comments
> > > > >>> >
> > > > >>> > We can choose to put these as
> > > > >>> >
> > > > >>> > 1) The full GitHub comment
> > > > >>> > 2) Only supply a link to the GitHub comment
> > > > >>> >
> > > > >>> > Next, we can store all posts to JIRA
> > > > >>> >
> > > > >>> > 1) In the JIRA main comments
> > > > >>> > 2) In the Work Log
> > > > >>> >
> > > > >>> >
> > > > >>> > I think it would be a nice setup to have the GitHub PR
> > description
> > > > and
> > > > >>> > comments directly in the JIRA comments. Diff comments should go
> > in
> > > > the
> > > > >>> > Work Log.
> > > > >>> >
> > > > >>> > On Fri, Aug 19, 2016 at 2:56 PM, Maximilian Michels <
> > > m...@apache.org>
> > > > >>> > wrote:
> > > > >>> > > Sure, I've filed a JIRA: https://issues.apache.org/
> > > > >>> > jira/browse/INFRA-12456
> > > > >>> > >
> > > > >>> > > On Thu, Aug 18, 2016 at 10:57 AM, Stephan Ewen <
> > se...@apache.org
> > > >
> > > > >>> wrote:
> > > > >>> > >> @max - can you contact infra about that?
> > > > >>> > >>
> > > > >>> > >> On Thu, Aug 18, 2016 at 10:25 AM, Maximilian Michels <
> > > > m...@apache.org>
> > > > >>> > wrote:
> > > > >>> > >>
> > > > >>> > >>> Actually that is a good suggestion. I know from other
> Apache
> > > > projects
> > > > >>> > >>> that they only mirror the initial description of the pull
> > > > request but
> > > > >>> > >>> not the discussion. I agree with you that it's very hard to
> > > have
> > > > >>> > >>> meaningful discussion in JIRA if it is overlapped with
> GitHub
> > > > >>> > >>> comments.
> > > > >>> > >>>
> > > > >>> > >>> Cheers,
> > > > >>> > >>> Max
> > > > >>> > >>>
> > > > >>> > >>> On Wed, Aug 17, 2016 at 10:52 PM, Neelesh Salian <
> > > > >>> nsal...@cloudera.com
> > > > >>> > >
> > > > >>> > >>> wrote:
> > > > >>> > >>> > Hello,
> > > > >>> > >>> >
> > > > >>> > >>> > I have noticed there is a high verbosity coming from the
> PR
> > > > into
> > > > >>> the
> > > > >>> > JIRA
> > > > >>> > >>> > which makes it hard to focus on the message or the
> content
> > of
> > > > the
> > > > >>> PR
> > > > >>> > >>> > through the JIRA itself.
> > > > >>> > >>> > Has there been a discussion over minimizing the details
> > > > displayed
> > > > >>> > back on
> > > > >>> > >>> > the PR's JIRA?
> > > > >>> > >>> >
> > > > >>> > >>> > Maybe just add the PR link and the description of the
> > changes
> > > > >>> rather
> > > > >>> > than
> > > > >>> > >>> > the entire template?
> > > > >>> > >>> >
> > > > >>> > >>> > Any thoughts?
> > > > >>> > >>> >
> > > > >>> > >>> > Regards,
> > > > >>> > >>> >
> > > > >>> > >>> > --
> > > > >>> > >>> > Neelesh Srinivas Salian
> > > > >>> > >>>
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Neelesh Srinivas Salian
> > > > >> Customer Operations Engineer
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Flink 1.1.1 (RC1)

[Henry Saputra]

Official vote
+1 (binding)

On Tuesday, August 9, 2016, Gyula Fóra  wrote:

> +1 from me, this is a very important fix.
>
> Gyula
>
> Vasiliki Kalavri > ezt írta
> (időpont: 2016. aug.
> 9., K, 19:15):
>
> > On 9 August 2016 at 18:27, Ufuk Celebi >
> wrote:
> >
> > > Dear Flink community,
> > >
> > > Please vote on releasing the following candidate as Apache Flink
> version
> > > 1.1.1.
> > >
> > > *Important*: I would like to reduce the voting time to 24 hours (with
> > > a majority of at least three +1 PMC votes as usual). We discovered
> > > that the Maven artifacts published with version 1.1.0 have dependency
> > > issues, which will affect users running on Hadoop 2 infrastructure
> > > (like HDFS). Since Maven artifacts are immutable, we cannot override
> > > them and we have to publish a new version to fix this.
> > >
> > > The release script contained a bug, which resulted in no deployment of
> > > a Hadoop 1 specific version (1.1.0-hadoop1) and regular 1.1.0
> > > artifacts having a dependency on Hadoop 1 instead of Hadoop 2. I've
> > > updated the release announcement accordingly with a warning (see
> > > http://flink.apache.org/news/2016/08/08/release-1.1.0.html).
> > >
> > > Please indicate whether you are OK with the reduced voting time.
> > >
> >
> > ​+1 fine with me​
> >
> >
> >
> > >
> > > The commit to be voted on:
> > > 61bfb36 (http://git-wip-us.apache.org/repos/asf/flink/commit/61bfb36)
> > >
> > > Branch:
> > > release-1.1.1-rc1
> > > (https://git1-us-west.apache.org/repos/asf/flink/repo?p=
> > > flink.git;a=shortlog;h=refs/heads/release-1.1.1-rc1)
> > >
> > > The release artifacts to be voted on can be found at:
> > > http://people.apache.org/~uce/flink-1.1.1-rc1/
> > >
> > > The release artifacts are signed with the key with fingerprint
> 9D403309:
> > > http://www.apache.org/dist/flink/KEYS
> > >
> > > The staging repository for this release can be found at:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1101
> > >
> > > -
> > >
> > > The vote is open for the next 24 hours (see above for explanation) and
> > > passes if a majority of at least three +1 PMC votes are cast.
> > >
> > > The vote ends on Wednesday, August 10th, 2016.
> > >
> > > [ ] +1 Release this package as Apache Flink 1.1.1
> > > [ ] -1 Do not release this package, because ...
> > >
> >
>

Re: [VOTE] Release Apache Flink 1.1.1 (RC1)

[Henry Saputra]

Yes, We can do this with reduced time for VOTE

On Tuesday, August 9, 2016, Ufuk Celebi  wrote:

> PS: Let me add that no changes other than the release 1.1.1 commit
> have been added on top of what we had in 1.1.0.
>
> On Tue, Aug 9, 2016 at 6:27 PM, Ufuk Celebi >
> wrote:
> > Dear Flink community,
> >
> > Please vote on releasing the following candidate as Apache Flink version
> 1.1.1.
> >
> > *Important*: I would like to reduce the voting time to 24 hours (with
> > a majority of at least three +1 PMC votes as usual). We discovered
> > that the Maven artifacts published with version 1.1.0 have dependency
> > issues, which will affect users running on Hadoop 2 infrastructure
> > (like HDFS). Since Maven artifacts are immutable, we cannot override
> > them and we have to publish a new version to fix this.
> >
> > The release script contained a bug, which resulted in no deployment of
> > a Hadoop 1 specific version (1.1.0-hadoop1) and regular 1.1.0
> > artifacts having a dependency on Hadoop 1 instead of Hadoop 2. I've
> > updated the release announcement accordingly with a warning (see
> > http://flink.apache.org/news/2016/08/08/release-1.1.0.html).
> >
> > Please indicate whether you are OK with the reduced voting time.
> >
> > The commit to be voted on:
> > 61bfb36 (http://git-wip-us.apache.org/repos/asf/flink/commit/61bfb36)
> >
> > Branch:
> > release-1.1.1-rc1
> > (https://git1-us-west.apache.org/repos/asf/flink/repo?p=
> flink.git;a=shortlog;h=refs/heads/release-1.1.1-rc1)
> >
> > The release artifacts to be voted on can be found at:
> > http://people.apache.org/~uce/flink-1.1.1-rc1/
> >
> > The release artifacts are signed with the key with fingerprint 9D403309:
> > http://www.apache.org/dist/flink/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapacheflink-1101
> >
> > -
> >
> > The vote is open for the next 24 hours (see above for explanation) and
> > passes if a majority of at least three +1 PMC votes are cast.
> >
> > The vote ends on Wednesday, August 10th, 2016.
> >
> > [ ] +1 Release this package as Apache Flink 1.1.1
> > [ ] -1 Do not release this package, because ...
>

Re: [ANNOUNCE] Flink 1.1.0 Released

[Henry Saputra]

Great work all. Great Thanks to Ufuk as RE :)

On Monday, August 8, 2016, Stephan Ewen  wrote:

> Great work indeed, and big thanks, Ufuk!
>
> On Mon, Aug 8, 2016 at 6:55 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com >
> wrote:
>
> > yoo-hoo finally announced 🎉
> > Thanks for managing the release Ufuk!
> >
> > On 8 August 2016 at 18:36, Ufuk Celebi >
> wrote:
> >
> > > The Flink PMC is pleased to announce the availability of Flink 1.1.0.
> > >
> > > On behalf of the PMC, I would like to thank everybody who contributed
> > > to the release.
> > >
> > > The release announcement:
> > > http://flink.apache.org/news/2016/08/08/release-1.1.0.html
> > >
> > > Release binaries:
> > > http://apache.openmirror.de/flink/flink-1.1.0/
> > >
> > > Please update your Maven dependencies to the new 1.1.0 version and
> > > update your binaries.
> > >
> > > – Ufuk
> > >
> >
>

Re: Introduction

[Henry Saputra]

Welcome! :)

On Sunday, July 31, 2016, Neelesh Salian  wrote:

> Hello folks,
>
> I am Neelesh Salian; I recently joined the Flink community and I wanted to
> take this opportunity to formally introduce myself.
>
> I have been working with the Hadoop and Spark ecosystems over the past two
> years and found Flink really interesting in the Streaming use case.
>
>
> Excited to start working and help build the community. :)
> Thank you.
> Have a great Sunday.
>
>
> --
> Neelesh Srinivas Salian
> Customer Operations Engineer
>

Re: [DISCUSS] Putting Flink user names / logos on the homepage

[Henry Saputra]

With lots of discussion about branding issues with Apahe projects, eg
Apache Spark, I would recommend us to move cautiously.

We need to make sure companies listed in the homepage do not represent
contributions or control over Apche Flink project.

We got some concerns about our Shepard initiative so let's make sure we
don't get another concern bc of this. What's being done in other projects
may not actually acceptable or the right thing to do.

- Henry


On Monday, July 4, 2016, Stephan Ewen  wrote:

> We have the "Powered By" page already.
> My naive assumption was that companies that have clearance to put their
> name there are also okay with a logo.
> After all, it is only displaying the same information in a more prominent
> place.
>
> On Mon, Jul 4, 2016 at 5:06 PM, Márton Balassi  >
> wrote:
>
> > I do like the idea, that seems to be the trend now - the Bigtop community
> > had a similar initiative recently. [1]
> > Helps dealing with the "Is it mature enough?" question. :)
> >
> > [1] http://kaiyzen.github.io/bigtop/
> >
> > On Mon, Jul 4, 2016 at 5:00 PM, Ufuk Celebi  > wrote:
> >
> > > I would like that! +1
> > >
> > > On Mon, Jul 4, 2016 at 4:59 PM, Aljoscha Krettek  >
> > > wrote:
> > > > Hi,
> > > > If we have some high-profile users that a worthwhile putting there
> and
> > > that
> > > > are OK with us putting up their logos then this would be great.
> > > >
> > > > Cheers,
> > > > Aljoscha
> > > >
> > > > On Mon, 4 Jul 2016 at 16:58 Stephan Ewen  > wrote:
> > > >
> > > >> Hi all!
> > > >>
> > > >> I was wondering if we want to put some names / logos of Flink users
> on
> > > the
> > > >> page.
> > > >> A bunch of Apache projects do that, for example, have a look at the
> > > Storm
> > > >> home page http://storm.apache.org
> > > >>
> > > >> From that section of users/logos, we would link to the "Powered By"
> > > section
> > > >> in the wiki.
> > > >>
> > > >> That would help showing the great adoption that the community has
> > > achieved
> > > >> by now.
> > > >>
> > > >> Let me know what you think!
> > > >>
> > > >> Greetings,
> > > >> Stephan
> > > >>
> > >
> >
>

Re: [DISCUSS] Putting Flink user names / logos on the homepage

[Henry Saputra]

Good discussion and I think we could bring this to dev list instead.

One reminder we should reduce cross posting to private& and dev@ lists to
avoid accidental exposure to internal PMC business.

- Henry

On Monday, July 4, 2016, Stephan Ewen  wrote:

> Hi all!
>
> I was wondering if we want to put some names / logos of Flink users on the
> page.
> A bunch of Apache projects do that, for example, have a look at the Storm
> home page http://storm.apache.org
>
> From that section of users/logos, we would link to the "Powered By" section
> in the wiki.
>
> That would help showing the great adoption that the community has achieved
> by now.
>
> Let me know what you think!
>
> Greetings,
> Stephan
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Henry Saputra]

Thanks for the reply, @Max. I was not aware it was about dynamic scaling,
which I think also asked for YARN support.
I agree to list all related half merge JIRA for the ResourceManager.

Looking forward for the Apache Mesos integration design for sure =)

- Henry

On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels  wrote:

> Hi Robert, hi Henry,
>
> +1 for a 1.1.0 release soon! We have enough new features that justify
> a major release.
>
> @Henry We have plans to extend the ResourceManager to interact with
> the Scheduler which will be a prerequisite for dynamic scaling. I
> think this is out of scope for 1.1.0. The upcoming Mesos integration
> won't require additional refactoring of the ResourceManager. Instead,
> we will create a new "Dispatcher" component that takes care of
> bootstrapping the initial node with the JobManager/ResourceManager.
> From there on, everything will be handled by the Mesos
> ResourceManager. I recently discussed this with Eron (CC) who came up
> with this design and he plans to publish it to the mailing list soon.
>
> How about listing relevant JIRA issues here? "Half Merged" is kind of
> hard to get for people who are not involved in the different
> components.
>
> The Cassandra adapter seems like a pretty important thing to have for
> the next release. In addition, I would like to merge FLINK-3667 and
> FLINK-3937. Robert is doing a review at the moment :) Those are a)
> refactoring of the command-line and client classes b) adding
> capability to resume cluster programmatically.
>
> Then we should also have a look at any other critical/major bugs listed in
> JIRA.
>
> Cheers,
> Max
>
> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra 
> wrote:
> > Hi Robert,
> >
> > Thanks for staying the discussion.
> >
> > Do you know if there any open tasks for the Resource Manager left?
> >
> > That is probably needed for Mesos integration?
> >
> > - Henry
> >
> > On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger 
> > wrote:
> >
> >> Hi,
> >>
> >> Flink 1.0.0 was released early March, so three months have passed and I
> >> think we should start discussing the scope of the next major release
> >> (1.1.0).
> >>
> >> From a high level point of view, we've added the following new features:
> >> in master:
> >> - Table API Refactoring, SQL, StreamSQL
> >> - The metrics system
> >> - Kinesis Connector
> >> - Persistent file sources for streaming
> >>
> >> Half merged:
> >> - Resource manager refactoring
> >>
> >> Unmerged features:
> >> - Cassandra connector
> >> - Key groups ("rescaling from savepoints")
> >> - Queryable state
> >>
> >> I'm pretty sure I forgot many other features / pull requests, please
> post
> >> them to this thread. I'll collect them and create a Wiki page out of it.
> >>
> >> Some immediate TODOs for us:
> >> - Which of the unmerged features are we going to add to the release?
> >> - Which blockers do we need to address before releasing?
> >> - Are there any volunteers for the release manager?
> >>
> >>
> >> Regards,
> >> Robert
> >>
>

Re: [DISCUSS] Releasing Flink 1.1.0

[Henry Saputra]

Hi Robert,

Thanks for staying the discussion.

Do you know if there any open tasks for the Resource Manager left?

That is probably needed for Mesos integration?

- Henry

On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger 
wrote:

> Hi,
>
> Flink 1.0.0 was released early March, so three months have passed and I
> think we should start discussing the scope of the next major release
> (1.1.0).
>
> From a high level point of view, we've added the following new features:
> in master:
> - Table API Refactoring, SQL, StreamSQL
> - The metrics system
> - Kinesis Connector
> - Persistent file sources for streaming
>
> Half merged:
> - Resource manager refactoring
>
> Unmerged features:
> - Cassandra connector
> - Key groups ("rescaling from savepoints")
> - Queryable state
>
> I'm pretty sure I forgot many other features / pull requests, please post
> them to this thread. I'll collect them and create a Wiki page out of it.
>
> Some immediate TODOs for us:
> - Which of the unmerged features are we going to add to the release?
> - Which blockers do we need to address before releasing?
> - Are there any volunteers for the release manager?
>
>
> Regards,
> Robert
>

Re: [PROPOSAL] Structure the Flink Open Source Development

[Henry Saputra]

+1 for shepherd

I would prefer using that term rather than maintainer. It is being used in
Incubator PMC to help them keeping healthy development in podlings.

The term "maintainer" kind of being scrutinized in ASF communities, in
recent episodes happening in Spark community.

- Henry

On Wed, Jun 1, 2016 at 12:00 PM, Stephan Ewen  wrote:

> I like the name "shepherd". It implies a non-authorative role, and implies
> guidance, which is very fitting.
>
> I also thing there is no problem with having a "component shepherd" and a
> "pull request shepherd".
>
> Stephan
>
>
> On Wed, Jun 1, 2016 at 7:11 PM, Fabian Hueske  wrote:
>
> > I think calling the role maintainer is not a good idea.
> > The Spark community had a maintainer process which they just voted to
> > remove. From my understanding, a maintainer in Spark had a more active
> role
> > than the role we are currently discussing.
> >
> > I would prefer to not call the role "maintainer" to make clear that the
> > responsibilities are different (less active) and mainly observing.
> >
> > 2016-06-01 13:14 GMT+02:00 Ufuk Celebi :
> >
> > > Thanks! I like the idea of renaming it.  I'm fine with shepherd and I
> > > also like Vasia's suggestion "champion".
> > >
> > > I would like to add "Distributed checkpoints" as a separate component
> > > to track development for check- and savepoints.
> > >
> > >
> > >
> > > On Wed, Jun 1, 2016 at 10:59 AM, Aljoscha Krettek  >
> > > wrote:
> > > > Btw, in Jira, if we clean up our components we can also set a
> component
> > > > Lead that would get notified of issues for that component.
> > > >
> > > > On Wed, 1 Jun 2016 at 10:43 Chesnay Schepler 
> > wrote:
> > > >
> > > >> I'd also go with maintainer.
> > > >>
> > > >> On 01.06.2016 10:32, Aljoscha Krettek wrote:
> > > >> > Hi,
> > > >> > I think maintainer is also fine if we clearly specify that they
> are
> > > not
> > > >> > meant as dictators or gatekeepers of the component that they are
> > > >> > responsible for.
> > > >> >
> > > >> > -Aljoscha
> > > >> >
> > > >> > On Wed, 1 Jun 2016 at 09:48 Vasiliki Kalavri <
> > > vasilikikala...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Hi,
> > > >> >>
> > > >> >> we could go for something like "sponsor" or "champion" :)
> > > >> >> I'm fine with the proposal. Good to see more than 1 person for
> both
> > > >> Gelly
> > > >> >> and Table API.
> > > >> >>
> > > >> >> cheers,
> > > >> >> -V.
> > > >> >>
> > > >> >> On 1 June 2016 at 05:46, Tzu-Li (Gordon) Tai  >
> > > >> wrote:
> > > >> >>
> > > >> >>> I'd like to be added to the Streaming Connectors component
> > (already
> > > >> >> edited
> > > >> >>> Wiki) :)
> > > >> >>>
> > > >> >>> Ah, naming, one of the hardest problems in programming :P Some
> > > >> comments:
> > > >> >>> I agree with Robert that the name "maintainers" will be somewhat
> > > >> >> misleading
> > > >> >>> regarding the authoritative difference with committers / PMCs,
> > > >> especially
> > > >> >>> for future newcomers to the community who don't come across the
> > > >> original
> > > >> >>> discussion on this thread.
> > > >> >>>
> > > >> >>> Simone's suggestion of Overseer seems good. The name naturally
> > > matches
> > > >> >> its
> > > >> >>> role -
> > > >> >>> - A group of "Overseers" for components, who keeps an eye on
> > related
> > > >> mail
> > > >> >>> threads, known limitations and issues, JIRAs, open PRs,
> requested
> > > >> >> features,
> > > >> >>> and potential new overseers and committers, etc, for the
> component
> > > >> >>> (original
> > > >> >>> maintainer role).
> > > >> >>> - A "Shepherd" for individual PRs, assigned from the overseers
> of
> > > the
> > > >> >>> component with the aim to guide the submitting contributor.
> > > Overseers
> > > >> >>> typically pick up new PRs to shepherd themselves, or the leading
> > > >> overseer
> > > >> >>> allocates an overseer to shepherd a PR which hasn't been picked
> up
> > > yet
> > > >> >>> after
> > > >> >>> a certain period of time.
> > > >> >>>
> > > >> >>> Or perhaps we can also simply go for "Shepherds" for components
> > and
> > > >> >>> "Assigned Shepherd" for individual PRs?
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>> --
> > > >> >>> View this message in context:
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/PROPOSAL-Structure-the-Flink-Open-Source-Development-tp11598p11932.html
> > > >> >>> Sent from the Apache Flink Mailing List archive. mailing list
> > > archive
> > > >> at
> > > >> >>> Nabble.com.
> > > >> >>>
> > > >>
> > > >>
> > >
> >
>

Re: Blogpost on Flink's SQL support

[Henry Saputra]

Awesome!
Thanks for the great post, Fabian

- Henry

On Tue, May 24, 2016 at 2:36 AM, Fabian Hueske  wrote:

> Thanks everybody for the feedback and comments!
>
> I moved the Google doc into Markdown and opened a PR:
> https://github.com/apache/flink-web/pull/22
>
> Will merge this PR and publish the post later today.
>
> Thanks, Fabian
>
> 2016-05-24 10:01 GMT+02:00 Kostas Tzoumas :
>
> > +1, great post
> >
> > On Sun, May 22, 2016 at 4:15 PM, Matthias J. Sax 
> wrote:
> >
> > > Will be a nice post!
> > >
> > > On 05/21/2016 10:40 PM, Henry Saputra wrote:
> > > > I agree with Ufuk, that this is more internal posts which perfect for
> > > blog.
> > > >
> > > > For high level and use cases I think would be better to be added to
> > > Apache
> > > > Flink release docs bc that is where most users will try to find info
> on
> > > how
> > > > to use it.
> > > >
> > > > - Henry
> > > >
> > > > On Saturday, May 21, 2016, Ufuk Celebi  wrote:
> > > >
> > > >> Hey Fabian,
> > > >>
> > > >> thank you for this blog post. I added some minor comments in the
> > > >> document. Great read and great work by you and the others who have
> > > >> contributed to SQL! :-)
> > > >>
> > > >> In general, I think that the post is very much an "Internals" post
> > > >> like the "bits and bytes" one. This is definitely nice, but I think
> > > >> that we should definitely follow up with a high-level/use case
> driven
> > > >> post after the release.
> > > >>
> > > >> – Ufuk
> > > >>
> > > >>
> > > >> On Sat, May 21, 2016 at 5:07 PM, Aljoscha Krettek <
> > aljos...@apache.org
> > > >> > wrote:
> > > >>> A great post!
> > > >>>
> > > >>> I had some small comments on the doc.
> > > >>>
> > > >>> On Sat, 21 May 2016 at 16:52 Robert Metzger  > > >> > wrote:
> > > >>>
> > > >>>> Thanks a lot for the great blog post!
> > > >>>>
> > > >>>> +1 for publishing it on the Flink blog.
> > > >>>>
> > > >>>> On Fri, May 20, 2016 at 5:12 PM, Fabian Hueske  > > >> > wrote:
> > > >>>>
> > > >>>>> Hi everybody,
> > > >>>>>
> > > >>>>> I wrote a blog post about the SQL efforts of the Flink community
> > and
> > > >>>> would
> > > >>>>> like to get your feedback.
> > > >>>>>
> > > >>>>> You can read and comment the Google doc:
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1xy6d9w6Gjm8Bsh9SJbGuTZsulBJxmdIXhESJ4dV2jEY
> > > >>>>>
> > > >>>>> I am planning to publish the post around next Tuesday.
> > > >>>>>
> > > >>>>> Looking forward to your comments and have a nice weekend,
> > > >>>>> Fabian
> > > >>>>>
> > > >>>>
> > > >>
> > > >
> > >
> > >
> >
>

Re: XGBoost on DataFlow and Flink

[Henry Saputra]

Hi Tianqi,

I am also impressed with XGBoost and looking forward to having integration
with Apache Flink.

As for resource requirements, Apache Flink is using abstraction of slots to
express parallel execution in the TaskManager [1]

I suppose you were looking for some DSL or specification configuration to
set resources such as CPU, memory, and GPU to be set per execution of a
task in the TaskManager?

- Henry

[1]
https://ci.apache.org/projects/flink/flink-docs-master/concepts/concepts.html#distributed-execution

On Mon, Mar 14, 2016 at 2:39 PM, Tianqi Chen 
wrote:

> Thanks! I am not aware of SrcOperator before. Then yes things can be done.
>
> About multi-threading issue, I am looking for more principled API to
> specify the resources requirement, e.g. the slots in this stage needs 4 GPU
> cores and 1 GPU. So the resource allocator can be aware of that.
>
> We have published the release announcement
>
> http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html
>
> And you can try the Flink version out:)
>
>
>
>
> On Mon, Mar 14, 2016 at 10:00 AM, Till Rohrmann 
> wrote:
>
> > Hi Tianqi,
> >
> > dmlc looks really cool and it would be great to integrate it with Flink.
> As
> > far as I understood your requirements, I think that you can already
> > implement most of it on Flink.
> >
> > For example, starting a special container which does not receive any
> input
> > could be a specialized SourceOperator. In this SourceOperator you could
> > then start a parameter server which receives the partial updates of the
> > processing tasks.
> >
> > You're right that each task is executed by its own thread. However, you
> > always can spawn new threads from within your user function. This should
> > allow you to run the ML code multi-threaded. But then it might be
> advisable
> > to lower the number of slots a bit so that you have more cores available
> > than slots defined.
> >
> > The integration of the dmlc API with the existing ML pipelines should not
> > be too hard. As long as one has access to the resulting data set it
> should
> > be easy to plug it into another predictor/estimator instance. I guess we
> > would mainly need some tooling around it.
> >
> > Looking forward running xgboost and mxnet with Flink :-)
> >
> > Cheers,
> > Till
> >
> > On Sat, Mar 12, 2016 at 7:17 PM, Simone Robutti <
> > simone.robu...@radicalbit.io> wrote:
> >
> > > Thanks for the insight, what you're doing is really interesting. I will
> > > definitely spend some time looking at DMLC and MXNet.
> > >
> > > 2016-03-12 18:35 GMT+01:00 Tianqi Chen :
> > >
> > > > Thanks for the reply.  I am writing a long email to give the answers
> to
> > > > Simone and clarifies what we do
> > > >
> > > > I want to mention that *you can use the library already in Flink*.
> See
> > > > Flink example here:
> > > >
> https://github.com/dmlc/xgboost/tree/master/jvm-packages#xgboost-flink
> > > >
> > > > I have not run pressure test on top of Flink, but we did the pressure
> > > test
> > > > thing on Spark and it is consistent with our standalone version on a
> > 100M
> > > > example dataset, gives around 10x over mllib's version. I assume same
> > > thing
> > > > holds for Flink as well.
> > > > So if you are interested, please try it out. I imagine this can be
> very
> > > > useful to have a Flink demo that can directly give competitive result
> > on
> > > > say a kaggle competition, and can attract more users to Flink
> community
> > > as
> > > > well.
> > > >
> > > > *The Internal Details*
> > > >Here at dmlc.ml , we are building libraries that we dive deep and
> > aim
> > > > for the best performance and flexibility. We build our own
> abstractions
> > > > when needed, for example, XGBoost relies on Allreduce, and MXNet,
> > another
> > > > well known deep learning project, relies on parameter server
> > abstraction.
> > > > We tried to make these abstractions portable, so they are not
> > stand-alone
> > > > C++ programs, but can be used as library in other languages, e.g.
> > scala.
> > > >So essentially, here is what is needed:
> > > > - Start a tracker on driver side to connect the workers together to
> use
> > > our
> > > > version of communication library (this can be swapped depending on
> > level
> > > of
> > > > integration, if communication API is provided natively by the
> > platform).
> > > > - An API to start concurrent jobs(containers), that can execute a
> > > function
> > > > (either worker or server).
> > > > - Gettng accessed to partition of data in each worker.
> > > >
> > > > Take XGBoost for example, what we do in Flink is to as a MapPartition
> > > > stage, and treat each slot as an worker.  Each worker then
> > > collaboratively
> > > > solve the machine learning problem, and return the trained model.
> > > >
> > > > *What is needed from DataFlow *
> > > > Dataflow is a nice abstraction for data processing. As you can see,
> the
> > > > approach we take is somewhat more low lev

Re: Blogpost on Flink's SQL support

[Henry Saputra]

I agree with Ufuk, that this is more internal posts which perfect for blog.

For high level and use cases I think would be better to be added to Apache
Flink release docs bc that is where most users will try to find info on how
to use it.

- Henry

On Saturday, May 21, 2016, Ufuk Celebi  wrote:

> Hey Fabian,
>
> thank you for this blog post. I added some minor comments in the
> document. Great read and great work by you and the others who have
> contributed to SQL! :-)
>
> In general, I think that the post is very much an "Internals" post
> like the "bits and bytes" one. This is definitely nice, but I think
> that we should definitely follow up with a high-level/use case driven
> post after the release.
>
> – Ufuk
>
>
> On Sat, May 21, 2016 at 5:07 PM, Aljoscha Krettek  > wrote:
> > A great post!
> >
> > I had some small comments on the doc.
> >
> > On Sat, 21 May 2016 at 16:52 Robert Metzger  > wrote:
> >
> >> Thanks a lot for the great blog post!
> >>
> >> +1 for publishing it on the Flink blog.
> >>
> >> On Fri, May 20, 2016 at 5:12 PM, Fabian Hueske  > wrote:
> >>
> >> > Hi everybody,
> >> >
> >> > I wrote a blog post about the SQL efforts of the Flink community and
> >> would
> >> > like to get your feedback.
> >> >
> >> > You can read and comment the Google doc:
> >> >
> >> >
> >> >
> >>
> https://docs.google.com/document/d/1xy6d9w6Gjm8Bsh9SJbGuTZsulBJxmdIXhESJ4dV2jEY
> >> >
> >> > I am planning to publish the post around next Tuesday.
> >> >
> >> > Looking forward to your comments and have a nice weekend,
> >> > Fabian
> >> >
> >>
>

Re: [DISCUSS] Secure Flink clusters

[Henry Saputra]

Eron,

Could you please do also loop me in in the early discussions since we are
interested on deploying Flink as standalone to access secure data via
Kerberized access.

I also was talking to Owen from HDFS at the Apache Big Data and there could
be some work we can ask to be done in the Hadoop common or HDFS side.


- Henry

On Tue, May 17, 2016 at 11:10 AM, Wright, Eron  wrote:

> Thanks to all who reviewed the document.It appears we have a good plan
> and I'm filing JIRA issues accordingly.
>
> Robert, I'm in touch with Max, Stephan, and Stefano.I’ll update the
> thread when we have a better sense of the timing.   The work will clearly
> span a couple of releases.
>
> Eron
>
>
> > On May 17, 2016, at 8:35 AM, Robert Metzger  wrote:
> >
> > Hi Eron,
> >
> > thanks a lot for putting so much effort into the design document. You've
> > probably spend a lot of time to come up with it!
> > I have to admit that I'm not that familiar with the topic, so I probably
> > need to re-read it again to digest it completely.
> >
> > What are your plans for implementing the proposed changes? (time-wise and
> > people-wise?) I'm asking to get an idea of when we can expect the changes
> > in the master, in releases, ...
> >
> > I think Stefano Baghino also had some discussions about improving Flink's
> > security on the mailing list recently. Maybe you guys can sync your
> efforts
> > and collaborate on this.
> >
> > Regards,
> > Robert
> >
> >
> > On Fri, May 13, 2016 at 12:47 PM, Maximilian Michels 
> wrote:
> >
> >> Hi Eron,
> >>
> >> Thank you for this comprehensive design document. Really great read.
> >> I've left some minor comments.
> >>
> >> +1 for breaking down the tasks into many JIRA issues; we have quite
> >> some ambitious plans now :) It would be great to get some more people
> >> from the community involved as well.
> >>
> >> Best,
> >> Max
> >>
> >> On Wed, May 11, 2016 at 9:09 AM, Wright, Eron  wrote:
> >>> Hello!
> >>>
> >>> There’s been a few discussions lately on how to improve the Kerberos
> >> support in Flink.  I’ve drafted a design document that lays out a plan
> to
> >> support keytab-based authentication for HDFS, Kafka, and ZooKeeper.  In
> >> addition, the plan contemplates secure, TLS-based communication between
> >> cluster components.
> >>>
> >>> The main goals are secure data access for Kerberized connectors and
> >> cluster authentication to prevent unauthorized access to cluster
> secrets.
> >>>
> >>> Here is the document:
> >>>
> >>
> https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing
> >>>
> >>> I anticipate filing a multitude of JIRAs following a design discussion.
> >>  It is a big task and there will be opportunities for others in the
> >> community to help.
> >>>
> >>> Thanks,
> >>> Eron Wright
> >>> EMC
> >>
>
>

Re: [PROPOSAL] Structure the Flink Open Source Development

[Henry Saputra]

The maintainers concept is good idea to make sure PRs are moved smoothly.

But, we need to make sure that this is not additional hierarchy on top of
Flink PMCs.
This will keep us in spirit of ASF community over code.

Please do add me as cluster management maintainer member.

- Henry

On Tuesday, May 10, 2016, Stephan Ewen  wrote:

> Hi everyone!
>
> We propose to establish some lightweight structures in the Flink open
> source community and development process,
> to help us better handle the increased interest in Flink (mailing list and
> pull requests), while not overwhelming the
> committers, and giving users and contributors a good experience.
>
> This proposal is triggered by the observation that we are reaching the
> limits of where the current community can support
> users and guide new contributors. The below proposal is based on
> observations and ideas from Till, Robert, and me.
>
> 
> Goals
> 
>
> We try to achieve the following
>
>   - Pull requests get handled in a timely fashion
>   - New contributors are better integrated into the community
>   - The community feels empowered on the mailing list.
> But questions that need the attention of someone that has deep
> knowledge of a certain part of Flink get their attention.
>   - At the same time, the committers that are knowledgeable about many core
> parts do not get completely overwhelmed.
>   - We don't overlook threads that report critical issues.
>   - We always have a pretty good overview of what the status of certain
> parts of the system are.
>   -> What are often encountered known issues
>   -> What are the most frequently requested features
>
>
> 
> Problems
> 
>
> Looking into the process, there are two big issues:
>
> (1) Up to now, we have been relying on the fact that everything just
> "organizes itself", driven by best effort. That assumes
> that everyone feels equally responsible for every part, question, and
> contribution. At the current state, this is impossible
> to maintain, it overwhelms the committers and contributors.
>
> Example: Pull requests are picked up by whoever wants to pick them up. Pull
> requests that are a lot of work, have little
> chance of getting in, or relate to less active components are sometimes not
> picked up. When contributors are pretty
> loaded already, it may happen that no one eventually feels responsible to
> pick up a pull request, and it falls through the cracks.
>
> (2) There is no good overview of what are known shortcomings, efforts, and
> requested features for different parts of the system.
> This information exists in various peoples' heads, but is not easily
> accessible for new people. The Flink JIRA is not well
> maintained, it is not easy to draw insights from that.
>
>
> ===
> The Proposal
> ===
>
> Since we are building a parallel system, the natural solution seems to be:
> partition the workload ;-)
>
> We propose to define a set of components for Flink. Each component is
> maintained or tracked by one or more
> people - let's call them maintainers. It is important to note that we don't
> suggest the maintainers as an authoritative role, but
> simply as committers or contributors that visibly step up for a certain
> component, and mainly track and drive the efforts
> pertaining to that component.
>
> It is also important to realize that we do not want to suggest that people
> get less involved with certain parts and components, because
> they are not the maintainers. We simply want to make sure that each pull
> request or question or contribution has in the end
> one person (or a small set of people) responsible for catching and tracking
> it, if it was not worked on by the pro-active
> community.
>
> For some components, having multiple maintainers will be helpful. In that
> case, one maintainer should be the "chair" or "lead"
> and make sure that no issue of that component gets lost between the
> multiple maintainers.
>
>
> A maintainers' role is:
> -
>
>   - Have an overview of which of the open pull requests relate to their
> component
>   - Drive the pull requests relating to the component to resolution
>   => Moderate the decision whether the feature should be merged
>   => Make sure the pull request gets a shepherd.
>In many cases, the maintainers would shepherd themselves.
>   => In case the shepherd becomes inactive, the maintainers need to
> find a new shepherd.
>
>   - Have an overview of what are the known issues of their component
>   - Have an overview of what are the frequently requested features of their
> component
>
>   - Have an overview of which contributors are doing very good work in
> their component,
> would be candidates for committers, and should be mentored towards
> that.
>
>   - Resolve email threads that have been brought to their attention,
> because deeper
> component knowledge is required for that thread.
>
> A maintainers' role 

Re: Intellij code style

[Henry Saputra]

We could actually put this in the tools directory of the source and repo
and refer it from contribution guide.

@Dawid want to try to send Pull request for it?

On Thursday, April 28, 2016, Theodore Vasiloudis <
theodoros.vasilou...@gmail.com> wrote:

> Do we plan to include something like this in the contribution guide as
> well?
>
> On Thu, Apr 28, 2016 at 3:16 PM, Stefano Baghino <
> stefano.bagh...@radicalbit.io > wrote:
>
> > Awesome Dawid! Thanks for taking the time to do this. :)
> >
> > On Thu, Apr 28, 2016 at 1:45 PM, Dawid Wysakowicz <
> > wysakowicz.da...@gmail.com > wrote:
> >
> > > Hi,
> > >
> > > I tried to create a code style that would follow Flink code-style. It
> may
> > > be not "production" ready, but I think it can be a good start.
> > > Hope it will be useful for someone. Also I will be glad for any
> comments
> > > on that.
> > >
> > > 2016-04-10 13:59 GMT+02:00 Stephan Ewen  >:
> > >
> > >> I don't know how close Phoenix' code style is to Flink's de-facto code
> > >> style.
> > >> I would create one that reflects Flink's de-facto code style, so that
> > the
> > >> formatter does not change everything...
> > >>
> > >> On Sun, Apr 10, 2016 at 4:40 AM, Naveen Madhire <
> vmadh...@umail.iu.edu >
> > >> wrote:
> > >>
> > >> > Apache Phoenix has one code template which contributors use. Do you
> > >> think
> > >> > onc can use the same for Flink or may be with some more
> modifications?
> > >> >
> > >> >
> > >> >
> > >>
> >
> https://github.com/apache/phoenix/blob/master/dev/PhoenixCodeTemplate.xml
> > >> >
> > >> > On Sat, Apr 9, 2016 at 11:00 AM, Stephan Ewen  >
> > wrote:
> > >> >
> > >> > > Actually, It would be amazing to create a code style profile for
> > >> > download,
> > >> > > so that all contributors would use that.
> > >> > >
> > >> > > Same thing actually for IntelliJ inspections: A set of inspections
> > we
> > >> > want
> > >> > > to have active and where we strive for zero warnings.
> > >> > >
> > >> > > On Sat, Apr 9, 2016 at 10:00 AM, Robert Metzger <
> > rmetz...@apache.org >
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Dawid,
> > >> > > >
> > >> > > > we don't have an automated formatter for intelliJ. However, you
> > can
> > >> use
> > >> > > the
> > >> > > > "Checkstyle" plugin of IntelliJ to mark checkstyle violations in
> > the
> > >> > IDE.
> > >> > > >
> > >> > > > On Fri, Apr 8, 2016 at 12:30 PM, Dawid Wysakowicz <
> > >> > > > wysakowicz.da...@gmail.com > wrote:
> > >> > > >
> > >> > > > > Hi all,
> > >> > > > >
> > >> > > > > I am currently working on some issues and been wondering if
> you
> > >> have
> > >> > > > > settings for Intellij code style that would follow your coding
> > >> > > guidelines
> > >> > > > > available (I tried to look on wikis but could not find it). If
> > not
> > >> > > could
> > >> > > > > someone share its own? I would be grateful.
> > >> > > > >
> > >> > > > > Regards
> > >> > > > > Dawid Wysakowicz
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
> >
> > --
> > BR,
> > Stefano Baghino
> >
> > Software Engineer @ Radicalbit
> >
>

Re: Powered by Flink

[Henry Saputra]

Thanks, Slim. I have just updated the wiki page with this entries.

On Tue, Apr 5, 2016 at 10:20 AM, Slim Baltagi  wrote:

> Hi
>
> The following are missing in the ‘Powered by Flink’ list:
>
>- *king.com  *
>
> https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces88
>- *Otto Group  *
>http://data-artisans.com/how-we-selected-apache-flink-at-otto-group/
>- *Eura Nova *https://research.euranova.eu/flink-forward-2015-talk/
>- *Big Data Europe *http://www.big-data-europe.eu
>
> Thanks
>
> Slim Baltagi
>
>
> On Apr 5, 2016, at 10:08 AM, Robert Metzger  wrote:
>
> Hi everyone,
>
> I would like to bring the "Powered by Flink" wiki page [1] to the
> attention of Flink user's who recently joined the Flink community. The list
> tracks which organizations are using Flink.
> If your company / university / research institute / ... is using Flink but
> the name is not yet listed there, let me know and I'll add the name.
>
> Regards,
> Robert
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink
>
>
> On Mon, Oct 19, 2015 at 4:10 PM, Matthias J. Sax  wrote:
>
>> +1
>>
>> On 10/19/2015 04:05 PM, Maximilian Michels wrote:
>> > +1 Let's collect in the Wiki for now. At some point in time, we might
>> > want to have a dedicated page on the Flink homepage.
>> >
>> > On Mon, Oct 19, 2015 at 3:31 PM, Timo Walther 
>> wrote:
>> >> Ah ok, sorry. I think linking to the wiki is also ok.
>> >>
>> >>
>> >> On 19.10.2015 15:18, Fabian Hueske wrote:
>> >>>
>> >>> @Timo: The proposal was to keep the list in the wiki (can be easily
>> >>> extended) but link from the main website to the wiki page.
>> >>>
>> >>> 2015-10-19 15:16 GMT+02:00 Timo Walther :
>> >>>
>>  +1 for adding it to the website instead of wiki.
>>  "Who is using Flink?" is always a question difficult to answer to
>>  interested users.
>> 
>> 
>>  On 19.10.2015 15:08, Suneel Marthi wrote:
>> 
>>  +1 to this.
>> 
>>  On Mon, Oct 19, 2015 at 3:00 PM, Fabian Hueske 
>> wrote:
>> 
>> > Sounds good +1
>> >
>> > 2015-10-19 14:57 GMT+02:00 Márton Balassi < <
>> balassi.mar...@gmail.com>
>> > balassi.mar...@gmail.com>:
>> >
>> >> Thanks for starting and big +1 for making it more prominent.
>> >>
>> >> On Mon, Oct 19, 2015 at 2:53 PM, Fabian Hueske < <
>> fhue...@gmail.com>
>> >
>> > fhue...@gmail.com> wrote:
>> >>>
>> >>> Thanks for starting this Kostas.
>> >>>
>> >>> I think the list is quite hidden in the wiki. Should we link from
>> >>> flink.apache.org to that page?
>> >>>
>> >>> Cheers, Fabian
>> >>>
>> >>> 2015-10-19 14:50 GMT+02:00 Kostas Tzoumas < 
>> >
>> > ktzou...@apache.org>:
>> 
>>  Hi everyone,
>> 
>>  I started a "Powered by Flink" wiki page, listing some of the
>>  organizations that are using Flink:
>> 
>> 
>> https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink
>> 
>>  If you would like to be added to the list, just send me a short
>> email
>>  with your organization's name and a description and I will add
>> you to
>> >
>> > the
>> 
>>  wiki page.
>> 
>>  Best,
>>  Kostas
>> 
>> >>>
>> 
>> 
>> >>
>>
>>
>
>

[jira] [Created] (FLINK-3690) Create documentation on the new ResourceManager component

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-3690:


 Summary: Create documentation on the new ResourceManager component
 Key: FLINK-3690
 URL: https://issues.apache.org/jira/browse/FLINK-3690
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, ResourceManager
Affects Versions: 1.1.0
Reporter: Henry Saputra
Assignee: Maximilian Michels


Need proper documentation for the new ResourceManager and how it will impact 
deployment in different supported modes.

Also, we have been very good adding new docs for our internal in the wiki [1] 
so would like that to happen for people evaluating Flink.

[1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Extending and improving our "How to contribute" page

[Henry Saputra]

+1 for slight addition from Max.

Travis exist for helping to run all tests not in your local dev.

On Friday, February 19, 2016, Fabian Hueske  wrote:

> Max, you're right.
> Not necessarily a local build, but a least "some" build to verify that the
> code compiles and most tests pass.
> I said local builds because this is the easiest way to check for somebody
> not familiar with our setup.
>
> I think it is a good idea to explicitly state the command to run: "mvn
> clean verify".
> This will help everybody not familiar with the setup and all others (e.g.,
> Travis users) know how to run a build anyway.
>
> Best, Fabian
>
> 2016-02-19 13:47 GMT+01:00 Maximilian Michels  >:
>
> > +1 for the documentation check box.
> >
> > Are we requiring local builds? Travis builds are fine, right? So what
> > about "Builds locally or on Travis"?
> >
> > Could we add more subpoints from the How to Contribute guide?
> >
> > [X] General
> >   - JIRA issue associated
> >   - Single PR per change
> >   - Meaningful commit message
> >
> > [X] CodeStyle
> >   - No unnecessary style changes
> >   - Check Style passes
> >
> > [X] Documentation
> >   - New documentation added
> >   - Old documentation updated
> >   - Javadocs for public methods
> >
> > [X] Tests
> >- Tests added or adapted
> >- Executed mvn verify or built on Travis
> >
> >
> > Martin, do you want to move this discussion to a new thread and
> > propose a template?
> >
> > Cheers,
> > Max
> >
> > On Fri, Feb 19, 2016 at 10:39 AM, Fabian Hueske  > wrote:
> > > Thanks Martin!
> > >
> > > can you add two more fields?
> > >
> > > - Builds locally (mvn clean verify)
> > > - Documentation updated or not updates necessary
> > >
> > > Best, Fabian
> > >
> > > 2016-02-19 9:35 GMT+01:00 Martin Liesenberg <
> martin.liesenb...@gmail.com 
> > >:
> > >
> > >> Cool, if no one objects, I'll create a JIRA ticket and open a
> > corresponding
> > >> PR during the weekend.
> > >>
> > >> Best regards
> > >> Martin
> > >>
> > >> On Thu, 18 Feb 2016, 17:36 Maximilian Michels  > wrote:
> > >>
> > >> > Hi Martin,
> > >> >
> > >> > Sounds like a good idea to me to create a checklist like this. It
> > >> > would be a nice reminder for people who didn't read the
> > >> > how-to-contribute section of the README :)
> > >> >
> > >> > Cheers,
> > >> > Max
> > >> >
> > >> > On Thu, Feb 18, 2016 at 9:31 AM, Martin Liesenberg
> > >> > > wrote:
> > >> > > Hi,
> > >> > >
> > >> > > GitHub just introduced a way to supply PR templates. [1]
> > >> > >
> > >> > > To support the changes discussed here, we could add a simple
> > template
> > >> > with
> > >> > > check boxes like:
> > >> > > [ ] did you add tests
> > >> > > [ ] did you check against the coding guidelines
> > >> > > [ ] is there a jira supporting the PR
> > >> > >
> > >> > > Let me know what you think. The language/tone probably needs a bit
> > of
> > >> > > refinement.
> > >> > >
> > >> > > best regards
> > >> > > martin
> > >> > >
> > >> > > [1] https://github.com/blog/2111-issue-and-pull-request-templates
> > >> > >
> > >> > > Till Rohrmann > schrieb am
> Do., 15. Okt. 2015
> > um
> > >> > > 11:58 Uhr:
> > >> > >
> > >> > >> Thanks for leading the effort Fabian!
> > >> > >>
> > >> > >> On Fri, Oct 9, 2015 at 10:07 AM, Maximilian Michels <
> > m...@apache.org >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Very nice work, Fabian. I think we'll have to send around a
> > reminder
> > >> > >> > from time to time and, perhaps, evaluate the new guidelines
> after
> > >> some
> > >> > >> > period of time. It's great to have these documents now as a
> > >> reference.
> > >> > >> >
> > >> > >

Re: [ANNOUNCE] Flink 0.10.2 Released

[Henry Saputra]

Yep, great work, Ufuk

On Friday, February 12, 2016, Kostas Kloudas 
wrote:

> Yes thanks a lot Ufuk!
>
> > On Feb 12, 2016, at 3:09 PM, Till Rohrmann  > wrote:
> >
> > Thanks for being our release manager Ufuk :-) Great work!
> >
> > On Fri, Feb 12, 2016 at 2:15 PM, Robert Metzger  > wrote:
> >
> >> Thank you for doing a release Ufuk!
> >>
> >> I just tweeted about it:
> >> https://twitter.com/ApacheFlink/status/698130110709428224
> >>
> >>
> >> On Fri, Feb 12, 2016 at 2:13 PM, Maximilian Michels  >
> >> wrote:
> >>
> >>> Bravo! Thank you Ufuk for managing the release!
> >>>
> >>> On Fri, Feb 12, 2016 at 2:02 PM, Fabian Hueske  >
> >> wrote:
>  Thanks Ufuk!
> 
>  2016-02-12 12:57 GMT+01:00 Ufuk Celebi 
> >:
> 
> > The Flink PMC is pleased to announce the availability of Flink
> 0.10.2.
> >
> > On behalf of the Flink PMC, I would like to thank everybody who
> >>> contributed
> > to the release.
> >
> > The official release announcement:
> > http://flink.apache.org/news/2016/02/11/release-0.10.2.html
> >
> > Release binaries:
> > http://apache.openmirror.de/flink/flink-0.10.2/
> >
> > Please update your Maven dependencies to the new 0.10.2 version and
> >>> update
> > your binaries.
> >
> >>>
> >>
>
>

Re: [VOTE] Release Apache Flink 0.10.2 (RC2)

[Henry Saputra]

+1 (binding)

Signature file on source looks good
Hash files on source look good
LICENSE file looks good
NOTICE file looks good
Source compiled
Tests passed
Standalone works
Test on YARN session works

- Henry


On Mon, Feb 8, 2016 at 9:37 AM, Ufuk Celebi  wrote:

> Dear Flink community,
>
> Please vote on releasing the following candidate as Apache Flink version
> 0.10.2.
>
> Please note that this vote has a slightly shorter voting period of 48
> hours. Only a single change has been made since the last release
> candidate. Since the community has already done extensive testing of the
> previous release candidate, I'm assuming 48 hours will suffice to vote on
> this one.
>
> The commit to be voted on:
> e525eb2f1413df238e994d01c909d2b90f1b7709
>
> Branch:
> release-0.10.2-rc2 (see
>
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-0.10.2-rc2
> )
>
> The release artifacts to be voted on can be found at:
> http://people.apache.org/~uce/flink-0.10.2-rc2/
>
> The release artifacts are signed with the key with fingerprint 9D403309:
> http://www.apache.org/dist/flink/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapacheflink-1061
>
> -
>
> The vote is open for the next 48 hours and passes if a majority of at least
> three +1 PMC votes are cast.
>
> The vote ends on Wednesday February 10, 2016, 18:45 CET.
>
> [ ] +1 Release this package as Apache Flink 0.10.2
> [ ] -1 Do not release this package because ...
>
> ===
>
> The following commits have been added since the 0.10.2 RC 1:
>
> * 2cd0618 - [tools] Properly update all POM versions in release script
> (3 hours ago) 
>

Re: [VOTE] Release Apache Flink 0.10.2 (RC2)

[Henry Saputra]

Hi Nick,

Thanks for bringing up the issue. I believe some people (including me) will
try to run Apache Flink on YARN =)

If more 0.10.x releases needed before 1.0 definitely would love to include
this in.

- Henry

On Tue, Feb 9, 2016 at 9:33 AM, Nick Dimiduk  wrote:

> Agree it's not a blocker. I can manage my own build with this patch if
> necessary. Just thought I might not be the only one running streaming on
> yarn.
>
> On Tuesday, February 9, 2016, Henry Saputra 
> wrote:
>
> > Hi Ufuk,
> >
> > This is nice to have but not a blocker.
> >
> > So unless we find blocker for the current RC I prefer to continue
> evaluate
> > and VOTE current RC.
> >
> > - Henry
> >
> > On Tuesday, February 9, 2016, Ufuk Celebi  >
> > wrote:
> >
> > > Hey Nick,
> > >
> > > I agree that this can be problematic when running multiple jobs on
> > > YARN. Since there is a chance that this might be the last release 0.10
> > > release, I would be OK to cancel the vote for your fix.
> > >
> > > Still, let's hear the opinion of others before doing this. What do you
> > > think?
> > >
> > > – Ufuk
> > >
> > >
> > > On Mon, Feb 8, 2016 at 8:05 PM, Nick Dimiduk  > 
> > > > wrote:
> > > > Perhaps too late for the RC, but I've backported FLINK-3293 to this
> > > branch
> > > > via FLINK-3372. Would be nice for those wanting to monitory yarn
> > > > application submissions.
> > > >
> > > > On Mon, Feb 8, 2016 at 9:37 AM, Ufuk Celebi  > 
> > > > wrote:
> > > >
> > > >> Dear Flink community,
> > > >>
> > > >> Please vote on releasing the following candidate as Apache Flink
> > version
> > > >> 0.10.2.
> > > >>
> > > >> Please note that this vote has a slightly shorter voting period of
> 48
> > > >> hours. Only a single change has been made since the last release
> > > >> candidate. Since the community has already done extensive testing of
> > the
> > > >> previous release candidate, I'm assuming 48 hours will suffice to
> vote
> > > on
> > > >> this one.
> > > >>
> > > >> The commit to be voted on:
> > > >> e525eb2f1413df238e994d01c909d2b90f1b7709
> > > >>
> > > >> Branch:
> > > >> release-0.10.2-rc2 (see
> > > >>
> > > >>
> > >
> >
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-0.10.2-rc2
> > > >> )
> > > >>
> > > >> The release artifacts to be voted on can be found at:
> > > >> http://people.apache.org/~uce/flink-0.10.2-rc2/
> > > >>
> > > >> The release artifacts are signed with the key with fingerprint
> > 9D403309:
> > > >> http://www.apache.org/dist/flink/KEYS
> > > >>
> > > >> The staging repository for this release can be found at:
> > > >>
> > https://repository.apache.org/content/repositories/orgapacheflink-1061
> > > >>
> > > >> -
> > > >>
> > > >> The vote is open for the next 48 hours and passes if a majority of
> at
> > > least
> > > >> three +1 PMC votes are cast.
> > > >>
> > > >> The vote ends on Wednesday February 10, 2016, 18:45 CET.
> > > >>
> > > >> [ ] +1 Release this package as Apache Flink 0.10.2
> > > >> [ ] -1 Do not release this package because ...
> > > >>
> > > >> ===
> > > >>
> > > >> The following commits have been added since the 0.10.2 RC 1:
> > > >>
> > > >> * 2cd0618 - [tools] Properly update all POM versions in release
> script
> > > >> (3 hours ago) 
> > > >>
> > >
> >
>

Re: [VOTE] Release Apache Flink 0.10.2 (RC2)

[Henry Saputra]

Hi Ufuk,

This is nice to have but not a blocker.

So unless we find blocker for the current RC I prefer to continue evaluate
and VOTE current RC.

- Henry

On Tuesday, February 9, 2016, Ufuk Celebi  wrote:

> Hey Nick,
>
> I agree that this can be problematic when running multiple jobs on
> YARN. Since there is a chance that this might be the last release 0.10
> release, I would be OK to cancel the vote for your fix.
>
> Still, let's hear the opinion of others before doing this. What do you
> think?
>
> – Ufuk
>
>
> On Mon, Feb 8, 2016 at 8:05 PM, Nick Dimiduk  > wrote:
> > Perhaps too late for the RC, but I've backported FLINK-3293 to this
> branch
> > via FLINK-3372. Would be nice for those wanting to monitory yarn
> > application submissions.
> >
> > On Mon, Feb 8, 2016 at 9:37 AM, Ufuk Celebi  > wrote:
> >
> >> Dear Flink community,
> >>
> >> Please vote on releasing the following candidate as Apache Flink version
> >> 0.10.2.
> >>
> >> Please note that this vote has a slightly shorter voting period of 48
> >> hours. Only a single change has been made since the last release
> >> candidate. Since the community has already done extensive testing of the
> >> previous release candidate, I'm assuming 48 hours will suffice to vote
> on
> >> this one.
> >>
> >> The commit to be voted on:
> >> e525eb2f1413df238e994d01c909d2b90f1b7709
> >>
> >> Branch:
> >> release-0.10.2-rc2 (see
> >>
> >>
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-0.10.2-rc2
> >> )
> >>
> >> The release artifacts to be voted on can be found at:
> >> http://people.apache.org/~uce/flink-0.10.2-rc2/
> >>
> >> The release artifacts are signed with the key with fingerprint 9D403309:
> >> http://www.apache.org/dist/flink/KEYS
> >>
> >> The staging repository for this release can be found at:
> >> https://repository.apache.org/content/repositories/orgapacheflink-1061
> >>
> >> -
> >>
> >> The vote is open for the next 48 hours and passes if a majority of at
> least
> >> three +1 PMC votes are cast.
> >>
> >> The vote ends on Wednesday February 10, 2016, 18:45 CET.
> >>
> >> [ ] +1 Release this package as Apache Flink 0.10.2
> >> [ ] -1 Do not release this package because ...
> >>
> >> ===
> >>
> >> The following commits have been added since the 0.10.2 RC 1:
> >>
> >> * 2cd0618 - [tools] Properly update all POM versions in release script
> >> (3 hours ago) 
> >>
>

Re: [DISCUSS] Remove Combinable Annotation from DataSet API

[Henry Saputra]

+1 for approach #1



- Henry

On Thu, Jan 14, 2016 at 5:35 AM, Chiwan Park  wrote:

> I’m also for approach #1. Now is nice time to apply some API-breaking
> changes.
>
> > On Jan 14, 2016, at 1:19 AM, Aljoscha Krettek 
> wrote:
> >
> > I’m also for Approach #1. I like simplifying things.
> >> On 13 Jan 2016, at 14:25, Vasiliki Kalavri 
> wrote:
> >>
> >> Hi,
> >>
> >> ​+1 for removing the Combinable annotation​. Approach 1 sounds like the
> >> best option to me.
> >>
> >>
> >> On 13 January 2016 at 14:11, Till Rohrmann 
> wrote:
> >>
> >>> Hi Fabian,
> >>>
> >>> thanks for bringing this issue up. I agree with you that it would be
> nice
> >>> to remove the Combinable annotation if it is not really needed. Now if
> >>> people are not aware of the Combinable interface then they might think
> that
> >>> they are actually using combiners by simply implementing
> CombineFunction.
> >>>
> >>>
> >> ​has happened to me :S​
> >>
> >>
> >>
> >>> I would also be in favour of approach 1 combined with a migration guide
> >>> where we highlight possible problems with a faulty combine
> implementation.
> >>>
> >>>
> >> Migration guide is a ​good idea​!
> >>
> >>
> >>
> >>> Cheers,
> >>> Till
> >>> ​
> >>>
> >>>
> >> ​-Vasia.​
> >>
> >>
> >>
> >>> On Wed, Jan 13, 2016 at 1:53 PM, Fabian Hueske 
> wrote:
> >>>
>  Hi everybody,
> 
> 
> 
>  As part of the upcoming 1.0 release we are stabilizing Flink's APIs.
> 
>  I would like to start a discussion about removing the Combinable
> >>> annotation
>  from the DataSet API.
> 
> 
> 
>  # The Current State:
> 
>  The DataSet API offers two interface for combine functions,
> >>> CombineFunction
>  and GroupCombineFunction, which can be added to a GroupReduceFunctions
>  (GRF).
> 
> 
>  However, implementing a combine interface does not make a GRF
> combinable.
>  In addition, the GRF class needs to be annotated with the Combinable
>  annotation. The RichGroupReduceFunction has a default implementation
> of
>  combine, which forwards the combine parameters to the reduce method.
> This
>  default implementation is not used, unless the class that extends RGRF
> >>> has
>  the Combinable annotation.
> 
> 
> 
>  In addition to the Combinable annotation, the DataSet API
>  GroupReduceOperator features the setCombinable(boolean) method to
> >>> override
>  a missing or existing Combinable annotation.
> 
> 
> 
>  # My Proposal:
> 
>  I propose to remove the Combinable annotation because it is not
> required
>  and complicates the definition of combiners. Instead, the combine
> method
> >>> of
>  a GroupReduceFunction should be executed if it implements one of the
>  combine function interfaces. This would require to remove the default
>  combine implementation of the RichGroupReduceFunction as well.
> 
>  This would be an API breaking change and has a few implications.
> 
> 
> 
>  # Possible Implementation:
> 
>  There are a few ways to implement this change.
> 
>  - Remove Combinable annotation or mark it deprecated (and keep effect)
> 
>  - Remove default combine method from RichGroupReduceFunction or
> deprecate
>  it
> 
> 
> 
>  Approach 1:
>  - Remove Combinable annotation
>  - Remove default combine() method from RichGroupReduceFunction
>  - Effect:
>    - All RichGroupReduceFunctions that do either have the Combinable
>  annotation or implement the combine method do not compile anymore
>    - GroupReduceFunctions that have the Combinable annotation do not
>  compile anymore
>    - GroupReduceFunctions that implement a combine interface without
>  having the annotation (and not being set to combinable during program
>  construction) will execute the previously not executed combine method.
> >>> This
>  might change the behavior of the program. In worst case, the program
> >>> might
>  silently produce wrong results (or crash) if the combiner
> implementation
>  was faulty. In best case, the program executes faster.
> 
> 
> 
>  Approach 2:
>  - As Approach 1
>  - In addition extend both combine interfaces with a deprecated marker
>  method. This will ensure that all functions that implement a
> combinable
>  interface do not compile anymore and need to be fixed. This could
> prevent
>  the silent failure as in Approach 1, but would also cause an
> additional
> >>> API
>  breaking change once the deprecated marker method is removed again.
> 
> 
> 
>  Approach 3:
>  - Mark Combinable annotation deprecated
>  - Mark combine() method in RichGroupReduceFunction as deprecated
>  - Effect:
>    - There'll be a couple of deprecation warnings.
>    - We face the same problem with silent failures as in Approach 1.
>    - We have to check i

Re: Dripping the Flink-on-Tez code for Flink 1.0

[Henry Saputra]

+1

I am always for simplifying our code base when possible.



On Sunday, January 10, 2016, Stephan Ewen  wrote:

> I was typing with fat fingers again. Meant "dropping code" of course, not
> "dripping" ;-)
>
> On Sun, Jan 10, 2016 at 11:55 AM, Fabian Hueske  > wrote:
>
> > +1
> >
> > 2016-01-08 17:35 GMT+01:00 Till Rohrmann  >:
> >
> > > +1 since it increase maintainability of the code base if it is not
> really
> > > used and thus removed.
> > >
> > > On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi  > wrote:
> > >
> > > > +1
> > > >
> > > > I wanted to make a similar proposal.
> > > >
> > > > – Ufuk
> > > >
> > > > > On 08 Jan 2016, at 17:03, Kostas Tzoumas  >
> > wrote:
> > > > >
> > > > > for clarification, I was talking about dropping the code, I am
> unsure
> > > > about
> > > > > the consequences of dripping code :-)
> > > > >
> > > > > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <
> ktzou...@apache.org >
> > > > wrote:
> > > > >
> > > > >> +1 from my side
> > > > >>
> > > > >> Flink on Tez never got a lot of user traction. It served well as a
> > > > >> prototype of "this is possible", but since the core functionality
> > will
> > > > be
> > > > >> subsumed by making Flink on YARN resource elastic, I don't see any
> > > > reason
> > > > >> we should have it as part of the Flink codebase.
> > > > >>
> > > > >> Best,
> > > > >> Kostas
> > > > >>
> > > > >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen  >
> > > wrote:
> > > > >>
> > > > >>> Hi all!
> > > > >>>
> > > > >>> Currently, Flink has a module to run batch program code on Tez
> > rather
> > > > than
> > > > >>> Flink's own distributed execution engine.
> > > > >>>
> > > > >>> I would suggest that we drop this code for the next release (1.0)
> > as
> > > > part
> > > > >>> of a code consolidation:
> > > > >>>
> > > > >>>  - There seems little in both the Flink and the Tez community to
> > use
> > > > and
> > > > >>> expand this functionality.
> > > > >>>
> > > > >>>  - The original motivation (better exploit resource elasticity in
> > > YARN)
> > > > >>> will no longer be valid in the near future. I am re-working the
> > YARN
> > > > >>> integration currently to make it more elastic and make it
> possible
> > to
> > > > run
> > > > >>> Flink on Mesos.
> > > > >>>
> > > > >>>  - The Flink-on-Tez code is rather POC status. Large scale
> testing
> > it
> > > > and
> > > > >>> making adding all missing features will take more effort than
> > making
> > > > >>> Flink's own YARN integration resource elastic.
> > > > >>>
> > > > >>>
> > > > >>> Please let me know what you think!
> > > > >>> Especially @Kostas, since you wrote the initial POC, I'd be
> > > interested
> > > > in
> > > > >>> your opinion.
> > > > >>>
> > > > >>> Greetings,
> > > > >>> Stephan
> > > > >>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: Effort to add SQL / StreamSQL to Flink

[Henry Saputra]

Awesome! Thanks for the reply, Fabian.

- Henry

On Sunday, January 10, 2016, Fabian Hueske  wrote:

> Hi Henry,
>
> There is https://issues.apache.org/jira/browse/FLINK-2099 and a few
> subissues.
> I'll reorganize these and add more issues for the tasks described in the
> design document in the next days.
>
> Thanks, Fabian
>
> 2016-01-10 2:45 GMT+01:00 Henry Saputra  >:
>
> > HI Fabian,
> >
> > Have you created JIRA ticket to keep track of this new feature?
> >
> > - Henry
> >
> > On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske  > wrote:
> > > Hi everybody,
> > >
> > > in the last days, Timo and I refined the design document for adding a
> > SQL /
> > > StreamSQL interface on top of Flink that was started by Stephan.
> > >
> > > The document proposes an architecture that is centered around Apache
> > > Calcite. Calcite is an Apache top-level project and includes a SQL
> > parser,
> > > a semantic validator for relational queries, and a rule- and cost-based
> > > relational optimizer. Calcite is used by Apache Hive and Apache Drill
> > > (among other projects). In a nutshell, the plan is to translate Table
> API
> > > and SQL queries into Calcite's relational expression trees, optimize
> > these
> > > trees, and translate them into DataSet and DataStream programs.The
> > document
> > > breaks down the work into several tasks and subtasks.
> > >
> > > Please review the design document and comment.
> > >
> > > -- >
> > >
> >
> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit?usp=sharing
> > >
> > > Unless there are major concerns with the design, Timo and I want to
> start
> > > next week to move the current Table API on top of Apache Calcite (Task
> 1
> > in
> > > the document). The goal of this task is to have the same functionality
> as
> > > currently, but with Calcite in the translation process. This is a
> > blocking
> > > task that we hope to complete soon. Afterwards, we can independently
> work
> > > on different aspects such as extending the Table API, adding a SQL
> > > interface (basically just a parser), integration with external data
> > > sources, better code generation, optimization rules, streaming support
> > for
> > > the Table API, StreamSQL, etc..
> > >
> > > Timo and I plan to work on a WIP branch to implement Task 1 and merge
> it
> > to
> > > the master branch once the task is completed. Of course, everybody is
> > > welcome to contribute to this effort. Please let us know such that we
> can
> > > coordinate our efforts.
> > >
> > > Thanks,
> > > Fabian
> >
>

Re: Add CEP library to Flink

[Henry Saputra]

HI Till,

Have you created JIRA ticket to keep track of this proposed new feature?

We should create one to keep track updates on the effort.

Thanks,

Henry

On Fri, Jan 8, 2016 at 6:54 AM, Till Rohrmann  wrote:
> Hi everybody,
>
> recently we've seen an increased interest in complex event processing (CEP)
> by Flink users. Even though most functionality is already there to solve
> many use cases it would still be helpful for most users to have an easy to
> use library. Having such a library which allows to define complex event
> patterns would increase Flink's user range to the CEP community. Once
> having laid the foundation, I'm optimistic that people will quickly pick it
> up and further extend it.
>
> The major contribution of this library would be to add an efficient
> non-deterministic finite automaton which can detect complex event patterns.
> For everything else, Flink already has most of the functionality in place.
>
> I've drafted a design document for the first version. Please review it and
> comment:
>
> https://docs.google.com/document/d/15iaBCZkNcpqSma_qrF0GUyobKV_JttEDVuhNd0Y1aAU/edit?usp=sharing
>
> Thanks,
> Till

Re: Effort to add SQL / StreamSQL to Flink

[Henry Saputra]

HI Fabian,

Have you created JIRA ticket to keep track of this new feature?

- Henry

On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske  wrote:
> Hi everybody,
>
> in the last days, Timo and I refined the design document for adding a SQL /
> StreamSQL interface on top of Flink that was started by Stephan.
>
> The document proposes an architecture that is centered around Apache
> Calcite. Calcite is an Apache top-level project and includes a SQL parser,
> a semantic validator for relational queries, and a rule- and cost-based
> relational optimizer. Calcite is used by Apache Hive and Apache Drill
> (among other projects). In a nutshell, the plan is to translate Table API
> and SQL queries into Calcite's relational expression trees, optimize these
> trees, and translate them into DataSet and DataStream programs.The document
> breaks down the work into several tasks and subtasks.
>
> Please review the design document and comment.
>
> -- >
> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit?usp=sharing
>
> Unless there are major concerns with the design, Timo and I want to start
> next week to move the current Table API on top of Apache Calcite (Task 1 in
> the document). The goal of this task is to have the same functionality as
> currently, but with Calcite in the translation process. This is a blocking
> task that we hope to complete soon. Afterwards, we can independently work
> on different aspects such as extending the Table API, adding a SQL
> interface (basically just a parser), integration with external data
> sources, better code generation, optimization rules, streaming support for
> the Table API, StreamSQL, etc..
>
> Timo and I plan to work on a WIP branch to implement Task 1 and merge it to
> the master branch once the task is completed. Of course, everybody is
> welcome to contribute to this effort. Please let us know such that we can
> coordinate our efforts.
>
> Thanks,
> Fabian

Re: Effort to add SQL / StreamSQL to Flink

[Henry Saputra]

I am excited and nervous at the same time =)

- Henry

On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske  wrote:
> Hi everybody,
>
> in the last days, Timo and I refined the design document for adding a SQL /
> StreamSQL interface on top of Flink that was started by Stephan.
>
> The document proposes an architecture that is centered around Apache
> Calcite. Calcite is an Apache top-level project and includes a SQL parser,
> a semantic validator for relational queries, and a rule- and cost-based
> relational optimizer. Calcite is used by Apache Hive and Apache Drill
> (among other projects). In a nutshell, the plan is to translate Table API
> and SQL queries into Calcite's relational expression trees, optimize these
> trees, and translate them into DataSet and DataStream programs.The document
> breaks down the work into several tasks and subtasks.
>
> Please review the design document and comment.
>
> -- >
> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit?usp=sharing
>
> Unless there are major concerns with the design, Timo and I want to start
> next week to move the current Table API on top of Apache Calcite (Task 1 in
> the document). The goal of this task is to have the same functionality as
> currently, but with Calcite in the translation process. This is a blocking
> task that we hope to complete soon. Afterwards, we can independently work
> on different aspects such as extending the Table API, adding a SQL
> interface (basically just a parser), integration with external data
> sources, better code generation, optimization rules, streaming support for
> the Table API, StreamSQL, etc..
>
> Timo and I plan to work on a WIP branch to implement Task 1 and merge it to
> the master branch once the task is completed. Of course, everybody is
> welcome to contribute to this effort. Please let us know such that we can
> coordinate our efforts.
>
> Thanks,
> Fabian

2015: A Year in Review for Apache Flink

[Henry Saputra]

Dear All,

It is almost end of 2015 and it has been busy and great year for Apache Flink =)

Robert Metzger had posted great blog summarizing Apache Flink grow for
this year:

  https://flink.apache.org/news/2015/12/18/a-year-in-review.html

Happy New Year everyone and thanks for being part of this great community!


Thanks,

- Henry

Re: Externalizing the Flink connectors

[Henry Saputra]

Yes, that would be the way to go.

We could follow Cask CDAP hydrator plugin repository [1] that support
different plugins to run in their main CDAP hydrator [2]  product

- Henry

[1] https://github.com/caskdata/hydrator-plugins
[2] https://github.com/caskdata/cdap

On Mon, Dec 14, 2015 at 1:49 AM, Maximilian Michels  wrote:
>>
>> Regarding Max suggestion to have version compatible connectors: I'm not
>> sure if we are able to maintain all connectors across different releases.
>>
>
> That was not my suggestion. Whenever we release, existing connectors should
> be compatible with that release. Otherwise, they should be removed from the
> release branch. This doesn't imply every connector version should be
> compatible across all releases.
>
> On Mon, Dec 14, 2015 at 10:39 AM, Robert Metzger 
> wrote:
>> Regarding Max suggestion to have version compatible connectors: I'm not
>> sure if we are able to maintain all connectors across different releases.
> I
>> think its okay to have a document describing the minimum required Flink
>> version for each connector.
>>
>> With the interface stability guarantees from 1.0 on, the number of
> breaking
>> changes will go down.
>>
>> I'm against the name "plugins" because everything (documentation, code,
>> code comments, ...) is called "connectors" and it would be a pretty
>> breaking change. I also think that "connector" describes much better what
>> the whole thing is about.
>>
>>
>>
>> On Mon, Dec 14, 2015 at 10:20 AM, Maximilian Michels 
> wrote:
>>
>>> Yes, absolutely. Setting up another repository for Flink ML would be no
>>> problem.
>>>
>>> On Sat, Dec 12, 2015 at 1:52 AM, Henry Saputra 
>>> wrote:
>>> > I had small chat with Till about how to help manage Flink ML Libraries
>>> > contributions, which use Flink ML as dependencies.
>>> >
>>> > I suppose if this approached is the way to go for Flink connectors,
>>> > could we do the same for Flink ML libraries?
>>> >
>>> >
>>> > - Henry
>>> >
>>> > On Fri, Dec 11, 2015 at 1:33 AM, Maximilian Michels 
>>> wrote:
>>> >> We should have release branches which are in sync with the release
>>> >> branches in the main repository. Connectors should be compatible
>>> >> across minor releases. The versioning could be of the form
>>> >> "flinkversion-connectorversion", e.g. 0.10-connector1.
>>> >>
>>> >>>The pluggable architecture is great! (why Don't we call it Flink
>>> plugins? my 2 cents)
>>> >>
>>> >> We can still change the name. IMHO "Plugins" is a bit broad since this
>>> >> is currently only targeted at the connectors included in Flink.
>>> >>
>>> >>>Would we loose test coverage by putting the connectors into a separate
>>> repository/maven project?
>>> >>
>>> >> Not necessarily. Two possibilities:
>>> >>
>>> >> 1) Run a connectors test jar during the normal Travis tests in the
>>> >> main repository
>>> >> 2) Trigger a Travis test run at the connectors repository upon a
>>> >> commit into the main repository
>>> >>
>>> >> Option 1 seems like the better alternative because we would
>>> >> immediately see if a change breaks the connectors. Of course, if
>>> >> changes are made in the connectors repository, we would also run tests
>>> >> with the main repository.
>>> >>
>>> >> On Thu, Dec 10, 2015 at 11:00 PM, jun aoki  wrote:
>>> >>> The pluggable architecture is great! (why Don't we call it Flink
>>> plugins?
>>> >>> my 2 cents)
>>> >>> It will be nice if we come up with an idea of what directory
> structure
>>> >>> should look like before start dumping connectors (plugins).
>>> >>> Also wonder what to do with versioning.
>>> >>> At some point, for example, Twitter v1 connector could be compatible
>>> with
>>> >>> flink 0.10 but Flume v2 connector could be compatible with trunk,
> etc.
>>> It
>>> >>> should be taken consideration either in the directory structure or
>>> >>> branching strategy.
>>> >>>
>>> >>> On Thu, Dec 10, 2015 at 7:12 AM, Aljoscha Krette

Re: Externalizing the Flink connectors

[Henry Saputra]

I had small chat with Till about how to help manage Flink ML Libraries
contributions, which use Flink ML as dependencies.

I suppose if this approached is the way to go for Flink connectors,
could we do the same for Flink ML libraries?


- Henry

On Fri, Dec 11, 2015 at 1:33 AM, Maximilian Michels  wrote:
> We should have release branches which are in sync with the release
> branches in the main repository. Connectors should be compatible
> across minor releases. The versioning could be of the form
> "flinkversion-connectorversion", e.g. 0.10-connector1.
>
>>The pluggable architecture is great! (why Don't we call it Flink plugins? my 
>>2 cents)
>
> We can still change the name. IMHO "Plugins" is a bit broad since this
> is currently only targeted at the connectors included in Flink.
>
>>Would we loose test coverage by putting the connectors into a separate 
>>repository/maven project?
>
> Not necessarily. Two possibilities:
>
> 1) Run a connectors test jar during the normal Travis tests in the
> main repository
> 2) Trigger a Travis test run at the connectors repository upon a
> commit into the main repository
>
> Option 1 seems like the better alternative because we would
> immediately see if a change breaks the connectors. Of course, if
> changes are made in the connectors repository, we would also run tests
> with the main repository.
>
> On Thu, Dec 10, 2015 at 11:00 PM, jun aoki  wrote:
>> The pluggable architecture is great! (why Don't we call it Flink plugins?
>> my 2 cents)
>> It will be nice if we come up with an idea of what directory structure
>> should look like before start dumping connectors (plugins).
>> Also wonder what to do with versioning.
>> At some point, for example, Twitter v1 connector could be compatible with
>> flink 0.10 but Flume v2 connector could be compatible with trunk, etc. It
>> should be taken consideration either in the directory structure or
>> branching strategy.
>>
>> On Thu, Dec 10, 2015 at 7:12 AM, Aljoscha Krettek 
>> wrote:
>>
>>> We would need to have a stable interface between the connectors and flink
>>> and have very good checks that ensure that we don’t inadvertently break
>>> things.
>>>
>>> > On 10 Dec 2015, at 15:45, Fabian Hueske  wrote:
>>> >
>>> > Sounds like a good idea to me.
>>> >
>>> > +1
>>> >
>>> > Fabian
>>> >
>>> > 2015-12-10 15:31 GMT+01:00 Maximilian Michels :
>>> >
>>> >> Hi squirrels,
>>> >>
>>> >> By this time, we have numerous connectors which let you insert data
>>> >> into Flink or output data from Flink.
>>> >>
>>> >> On the streaming side we have
>>> >>
>>> >> - RollingSink
>>> >> - Flume
>>> >> - Kafka
>>> >> - Nifi
>>> >> - RabbitMQ
>>> >> - Twitter
>>> >>
>>> >> On the batch side we have
>>> >>
>>> >> - Avro
>>> >> - Hadoop compatibility
>>> >> - HBase
>>> >> - HCatalog
>>> >> - JDBC
>>> >>
>>> >>
>>> >> Many times we would have liked to release updates to the connectors or
>>> >> even create new ones in between Flink releases. This is currently not
>>> >> possible because the connectors are part of the main repository.
>>> >>
>>> >> Therefore, I have created a new repository at
>>> >> https://git-wip-us.apache.org/repos/asf/flink-connectors.git. The idea
>>> >> is to externalize the connectors to this repository. We can then
>>> >> update and release them independently of the main Flink repository. I
>>> >> think this will give us more flexibility in the development process.
>>> >>
>>> >> What do you think about this idea?
>>> >>
>>> >> Cheers,
>>> >> Max
>>> >>
>>>
>>>
>>
>>
>> --
>> -jun

Re: Distributed DataFrame - ddf.io

[Henry Saputra]

Internally, DDF uses Flink Table APIs to process the SQL queries.

I would say that DDF would be very useful to provide good
virtualization when building application platform.

- Henry

On Thu, Dec 3, 2015 at 8:48 AM, Kostas Tzoumas  wrote:
> Hi Nam-Luc,
>
> I cc Rohit who implemented the DDF framework.
>
> I would say that the main difference with the Table API is that DDF aims at
> portability (running the same code using Flink, Spark, or a database),
> whereas the Table API is meant to be part of Flink itself.
>
> Best,
> Kostas
>
>
> On Thu, Dec 3, 2015 at 11:23 AM, Nam-Luc Tran 
> wrote:
>
>> Hello Everyone,
>>
>> We came across the Distributed DataFrame project (http://ddf.io) that aims
>> at implementing a dataframe representation targeting Spark and Flink.
>>
>> Has anybody already heard or played with this project? How would you
>> position that with regards to Flink's Tables?
>>
>> Cheers,
>>
>> --
>>
>> *Nam-Luc TRAN*
>>
>> R&D Manager
>>
>> EURA NOVA
>>
>> (M) +32 498 37 36 23
>>
>> *euranova.eu *
>>

Re: [VOTE] Release Apache Flink 0.10.1 (release-0.10.0-rc1)

[Henry Saputra]

+1

LICENSE file looks good in source artifact
NOTICE file looks good in source artifact
Signature file looks good in source artifact
Hash files looks good in source artifact
No 3rd party executables in source artifact
Source compiled
All tests are passed
Run standalone mode test app

- Henry

On Mon, Nov 23, 2015 at 4:45 AM, Robert Metzger  wrote:
> Hi All,
>
> this is the first bugfix release for the 0.10 series of Flink.
> I've CC'ed the user@ list if users are interested in helping to verify the
> release.
>
> It contains fixes for critical issues, in particular:
> - FLINK-3021 Fix class loading issue for streaming sources
> - FLINK-2974 Add periodic offset committer for Kafka
> - FLINK-2977 Using reflection to load HBase Kerberos tokens
> - FLINK-3024 Fix TimestampExtractor.getCurrentWatermark() Behaviour
> - FLINK-2967 Increase timeout for LOCAL_HOST address detection stratey
> - FLINK-3025 [kafka consumer] Bump transitive ZkClient dependency
> - FLINK-2989 job cancel button doesn't work on YARN
> - FLINK-3032: Flink does not start on Hadoop 2.7.1 (HDP), due to class
> conflict
> - FLINK-3011, 3019, 3028 Cancel jobs in RESTARTING state
>
> This is the guide on how to verify a release:
> https://cwiki.apache.org/confluence/display/FLINK/Releasing
>
> During the testing, please focus on trying out Flink on different Hadoop
> platforms: We changed the way how Hadoop's Maven dependencies are packaged,
> so maybe there are issues with different Hadoop distributions.
> The Kafka consumer also changed a bit, would be good to test it on a
> cluster.
>
> -
>
> Please vote on releasing the following candidate as Apache Flink version
> 0.10.1:
>
> The commit to be voted on:
> http://git-wip-us.apache.org/repos/asf/flink/commit/2e9b2316
>
> Branch:
> release-0.10.1-rc1 (see
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
>
> The release artifacts to be voted on can be found at:
> http://people.apache.org/~rmetzger/flink-0.10.1-rc1/
>
> The release artifacts are signed with the key with fingerprint  D9839159:
> http://www.apache.org/dist/flink/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapacheflink-1058
>
> -
>
> The vote is open for the next 72 hours and passes if a majority of at least
> three +1 PMC votes are cast.
>
> The vote ends on Wednesday, November 25.
>
> [ ] +1 Release this package as Apache Flink 0.10.1
> [ ] -1 Do not release this package because ...
>
> ===

Re: could you please add me Contributor list?

[Henry Saputra]

Hi Jun,

What is your JIRA username?

But for now, you can always work on JIRA issue without assigning to
yourself. Just add to comment that you are planning to work on it.

- Henry

On Tue, Nov 24, 2015 at 5:18 PM, jun aoki  wrote:
> So that I can assign myself to Flink jiras?
>
> --
> -jun

Re: [VOTE] [RESULT] Release Apache Flink 0.10.0 (release-0.10.0-rc8)

[Henry Saputra]

Awesome news! Thanks a lot for driving the release, Max! =)

On Thu, Nov 12, 2015 at 3:59 PM, fhueske  wrote:
> Thanks Max! :-)
>
>
> Von: Maximilian Michels
> Gesendet: Freitag, 13. November 2015 00:02
> An: dev@flink.apache.org
> Betreff: [VOTE] [RESULT] Release Apache Flink 0.10.0 (release-0.10.0-rc8)
>
>
> Thanks for voting! The vote passes.
>
> The following votes have been cast:
>
> +1 votes: 7
>
> Stephan
> Aljoscha
> Robert
> Max
> Chiwan*
> Henry
> Fabian
>
> * non-binding
>
> -1 votes: none
>
> I'll upload the release artifacts and release the Maven artifacts.
> Once the changes are effective, the community may announce the
> release.
>
>

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc8)

[Henry Saputra]

LICENSE file looks good
NOTICE file looks good
Hash files looks good for source artifact
Signature file checked for source artifact
No third party executable in source artifact
Source compiled
Tests passed
Run Word Count with local and Apache Hadoop YARN 2.6.0 in session mode.

+1

On Tue, Nov 10, 2015 at 12:41 PM, Maximilian Michels  wrote:
> Please note that this vote has a slightly shorter voting period of 48
> hours. Only very small changes have been made since the last release
> candidate. Since the community has already done extensive testing of the
> previous release candidates, I'm assuming 48 hours will suffice to vote on
> this release candidate.
>
> -
>
> Please vote on releasing the following candidate as Apache Flink version
> 0.10.0:
>
> The commit to be voted on:
> ab2cca4891f58e31bc3ec8d758d253a6cf84bc71
>
> Branch:
> release-0.10.0-rc8 (see
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
>
> The release artifacts to be voted on can be found at:
> http://people.apache.org/~mxm/flink-0.10.0-rc8/
>
> The release artifacts are signed with the key with fingerprint C2909CBF:
> http://www.apache.org/dist/flink/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapacheflink-1055
>
> -
>
> The vote is open for the next 48 hours and passes if a majority of at least
> three +1 PMC votes are cast.
>
> The vote ends on Thursday November 12, 2015.
>
> [ ] +1 Release this package as Apache Flink 0.10.0
> [ ] -1 Do not release this package because ...
>
> ===
>
> The following commits have been added on top of release-0.10.0-rc7:
>
> c0fe305 [FLINK-2992] Remove use of SerializationUtils
> c098377 [hotfix] Check for null in StreamSource.cancel()

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc4)

[Henry Saputra]

+1 to remove binary LICENSE/NOTICE

It should be ok since Apache officially just do source release. Binary
release is just for convenience.
Lets keep reducing complexity on releases.

- Henry

On Mon, Nov 2, 2015 at 1:56 PM, Robert Metzger  wrote:
> +1 for the approach without the full LICENSE/NOTICE for binary releases. As
> long as the source releases are correct there should be no problem
>
> On Mon, Nov 2, 2015 at 7:11 PM, Aljoscha Krettek 
> wrote:
>
>> Sorry for the back-and-forth guys. I updated my PR to completely remove
>> the LICENSE/NOTICE files that where specific to the binary release. Now we
>> just copy over the LICENSE/NOTICE files from the source release. This is
>> also how Hadoop does it, by the way.
>>
>> > On 02 Nov 2015, at 17:51, Aljoscha Krettek  wrote:
>> >
>> > Hi,
>> > I also discovered that basically we would need to provide custom
>> LICENSE/NOTICE files for our released binaries for different hadoop/scala/…
>> versions
>> > because they come with different dependencies (that we also include due
>> to shading).
>> >
>> > For example, this is the dependency tree for flink-shaded-hadoop2 (Flink
>> 0.9.0, Hadoop 2.4.0):
>> > [INFO] +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
>> > [INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.4.0:compile
>> > [INFO] |  +- com.google.guava:guava:jar:11.0.2:compile
>> > [INFO] |  +- commons-cli:commons-cli:jar:1.2:compile
>> > [INFO] |  +- org.apache.commons:commons-math3:jar:3.1.1:compile
>> > [INFO] |  +- xmlenc:xmlenc:jar:0.52:compile
>> > [INFO] |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
>> > [INFO] |  +- commons-codec:commons-codec:jar:1.4:compile
>> > [INFO] |  +- commons-io:commons-io:jar:2.4:compile
>> > [INFO] |  +- commons-net:commons-net:jar:3.1:compile
>> > [INFO] |  +- commons-collections:commons-collections:jar:3.2.1:compile
>> > [INFO] |  +- com.sun.jersey:jersey-core:jar:1.9:compile
>> > [INFO] |  +- com.sun.jersey:jersey-json:jar:1.9:compile
>> > [INFO] |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
>> > [INFO] |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
>> > [INFO] |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
>> > [INFO] |  |  | +- javax.xml.stream:stax-api:jar:1.0-2:compile
>> > [INFO] |  |  | \- javax.activation:activation:jar:1.1:compile
>> > [INFO] |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
>> > [INFO] |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
>> > [INFO] |  +- com.sun.jersey:jersey-server:jar:1.9:compile
>> > [INFO] |  +- commons-el:commons-el:jar:1.0:runtime
>> > [INFO] |  +- commons-logging:commons-logging:jar:1.1.3:compile
>> > [INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile
>> > [INFO] |  |  +- org.apache.httpcomponents:httpclient:jar:4.1.2:compile
>> > [INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.1.2:compile
>> > [INFO] |  |  \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
>> > [INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
>> > [INFO] |  +- commons-configuration:commons-configuration:jar:1.6:compile
>> > [INFO] |  |  +- commons-digester:commons-digester:jar:1.8:compile
>> > [INFO] |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
>> > [INFO] |  |  \-
>> commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
>> > [INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
>> > [INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
>> > [INFO] |  +- org.apache.avro:avro:jar:1.7.6:compile
>> > [INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
>> > [INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.0.5:compile
>> > [INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
>> > [INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.4.0:compile
>> > [INFO] |  +- com.jcraft:jsch:jar:0.1.42:compile
>> > [INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
>> > [INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
>> > [INFO] |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
>> > [INFO] | \- org.tukaani:xz:jar:1.0:compile
>> > [INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.4.0:compile
>> > [INFO] |  +- commons-daemon:commons-daemon:jar:1.0.13:compile
>> > [INFO] |  \- javax.servlet.jsp:jsp-api:jar:2.1:compile
>> > [INFO] +-
>> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.4.0:compile
>> > [INFO] +- org.apache.commons:commons-lang3:jar:3.3.2:compile
>> > [INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile
>> > [INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.7:compile
>> > [INFO] +- log4j:log4j:jar:1.2.17:compile
>> > [INFO] +- junit:junit:jar:4.11:test
>> > [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
>> > [INFO] +- org.mockito:mockito-all:jar:1.9.5:test
>> > [INFO] +- org.powermock:powermock-module-junit4:jar:1.5.5:test
>> > [INFO] |  \- org.powermock:powermock-module-junit4-common:jar:1.5.5:test
>> > [INFO] | +- org.powermock:powermock-core:jar:1.5.5:test
>> > [INFO] | |  \- o

Re: Broken link for master Javadocs

[Henry Saputra]

Wow, good catch. Thanks for the explanation, Max.
Yeah, we should run JavaDoc generation on Travis to hopefully catch
this issue early.

- Henry

On Mon, Nov 2, 2015 at 1:21 AM, Maximilian Michels  wrote:
> Hi Henry,
>
> Note that we use a special Maven profile for aggregating all the java
> docs and scala docs (-Paggregate-scaladoc). This makes the Scala
> classes available in the JavaDoc. We also have extra Scala docs.
>
> There were two issues recently for the Java Docs. The first one was
> with Java 8 which complained about a missing AkkaTestkit dependency
> (see https://issues.apache.org/jira/browse/FLINK-1610).
>
> The one which led to this email thread was a problem with the Scala
> shell adding a sources directory too late (see a39aa52).
>
> Another issue was that we ran Travis tests without JavaDoc generation.
> This saves a minute or so runtime but leaves us unaware of JavaDoc
> related problems. I recently changed that in 0845529.
>
> Cheers,
> Max
>
> On Mon, Nov 2, 2015 at 12:53 AM, Henry Saputra  
> wrote:
>> Hi Max,
>>
>> Thanks for this. Is it issue from Flink side? The Javadoc always built in
>> my local env.
>>
>> - Henry
>>
>> On Wednesday, October 28, 2015, Maximilian Michels  wrote:
>>
>>> The issue with our Java Docs has been resolved. The link works again.
>>>
>>> On Tue, Oct 27, 2015 at 3:57 PM, Henry Saputra >> > wrote:
>>> > Ah thanks Max, sending to commits@ is good
>>> >
>>> > - Henry
>>> >
>>> > On Tue, Oct 27, 2015 at 2:35 AM, Maximilian Michels >> > wrote:
>>> >> Hi Henry,
>>> >>
>>> >> Yes, there is. The Commits@ list actually gets notifications on
>>> failures
>>> >> and recoveries. I figured sending them to dev@ would bother too many
>>> people
>>> >> because sometimes the infrastructure is flaky and it fails for no
>>> >> particular reason.
>>> >>
>>> >> Cheers,
>>> >> Max
>>> >>
>>> >> On Tue, Oct 27, 2015 at 4:18 AM, Henry Saputra >> >
>>> >> wrote:
>>> >>
>>> >>> Hi Max,
>>> >>>
>>> >>> Is there a way that dev@ list gets email notification if the build
>>> fail
>>> >>> for
>>> >>> the build bot?
>>> >>>
>>> >>> - Henry
>>> >>>
>>> >>> On Monday, October 26, 2015, Maximilian Michels >> > wrote:
>>> >>>
>>> >>> > Thanks for reporting, Suneel. On my machine the Java docs build.
>>> >>> >
>>> >>> > Here's the build log:
>>> >>> >
>>> >>> >
>>> >>>
>>> https://ci.apache.org/builders/flink-docs-master/builds/122/steps/Java%20%26%20Scala%20docs/logs/stdio
>>> >>> >
>>> >>> >
>>> >>> > [ERROR]
>>> >>> >
>>> >>>
>>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:35:
>>> >>> > error: not found: type ILoopCompat
>>> >>> > [ERROR]   extends ILoopCompat(in0, out0) {
>>> >>> > [ERROR]   ^
>>> >>> > [ERROR]
>>> >>> >
>>> >>>
>>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:29:
>>> >>> > error: too many arguments for constructor Object: ()Object
>>> >>> > [ERROR] class FlinkILoop(
>>> >>> > [ERROR] ^
>>> >>> > [ERROR]
>>> >>> >
>>> >>>
>>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:118:
>>> >>> > error: value createInterpreter is not a member of AnyRef
>>> >>> > [ERROR] super.createInterpreter()
>>> >>> > [ERROR]   ^
>>> >>> > [ERROR]
>>> >>> >
>>> >>>
>>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:120:
>>> >>> > error: not found: value addThunk
>>> >>> > [ERROR] 

Re: Broken link for master Javadocs

[Henry Saputra]

Hi Max,

Thanks for this. Is it issue from Flink side? The Javadoc always built in
my local env.

- Henry

On Wednesday, October 28, 2015, Maximilian Michels  wrote:

> The issue with our Java Docs has been resolved. The link works again.
>
> On Tue, Oct 27, 2015 at 3:57 PM, Henry Saputra  > wrote:
> > Ah thanks Max, sending to commits@ is good
> >
> > - Henry
> >
> > On Tue, Oct 27, 2015 at 2:35 AM, Maximilian Michels  > wrote:
> >> Hi Henry,
> >>
> >> Yes, there is. The Commits@ list actually gets notifications on
> failures
> >> and recoveries. I figured sending them to dev@ would bother too many
> people
> >> because sometimes the infrastructure is flaky and it fails for no
> >> particular reason.
> >>
> >> Cheers,
> >> Max
> >>
> >> On Tue, Oct 27, 2015 at 4:18 AM, Henry Saputra  >
> >> wrote:
> >>
> >>> Hi Max,
> >>>
> >>> Is there a way that dev@ list gets email notification if the build
> fail
> >>> for
> >>> the build bot?
> >>>
> >>> - Henry
> >>>
> >>> On Monday, October 26, 2015, Maximilian Michels  > wrote:
> >>>
> >>> > Thanks for reporting, Suneel. On my machine the Java docs build.
> >>> >
> >>> > Here's the build log:
> >>> >
> >>> >
> >>>
> https://ci.apache.org/builders/flink-docs-master/builds/122/steps/Java%20%26%20Scala%20docs/logs/stdio
> >>> >
> >>> >
> >>> > [ERROR]
> >>> >
> >>>
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:35:
> >>> > error: not found: type ILoopCompat
> >>> > [ERROR]   extends ILoopCompat(in0, out0) {
> >>> > [ERROR]   ^
> >>> > [ERROR]
> >>> >
> >>>
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:29:
> >>> > error: too many arguments for constructor Object: ()Object
> >>> > [ERROR] class FlinkILoop(
> >>> > [ERROR] ^
> >>> > [ERROR]
> >>> >
> >>>
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:118:
> >>> > error: value createInterpreter is not a member of AnyRef
> >>> > [ERROR] super.createInterpreter()
> >>> > [ERROR]   ^
> >>> > [ERROR]
> >>> >
> >>>
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:120:
> >>> > error: not found: value addThunk
> >>> > [ERROR] addThunk {
> >>> > [ERROR] ^
> >>> > [ERROR]
> >>> >
> >>>
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:138:
> >>> > error: not found: value intp
> >>> > [ERROR] val vd = intp.virtualDirectory
> >>> > [ERROR]  ^
> >>> > [ERROR]
> >>> >
> >>>
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:186:
> >>> > error: not found: value echo
> >>> > [ERROR] echo(
> >>> > [ERROR] ^
> >>> > [ERROR]
> >>> >
> >>>
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala:151:
> >>> > error: value process is not a member of
> >>> > org.apache.flink.api.scala.FlinkILoop
> >>> > [ERROR]   repl.foreach(_.process(settings))
> >>> > [ERROR]  ^
> >>> > [ERROR]
> >>> >
> >>>
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala:153:
> >>> > error: value closeInterpreter is not a member of
> >>> > org.apache.flink.api.scala.FlinkILoop
> >>> > [ERROR]   repl.foreach(_.closeInterpreter())
> >>> > [ERROR]  ^
> >>> > [ERROR] 8 errors found
> >>> >
> >>> >
> >>> > Not sure what the issue is. I'll try to look into later.
> >>> >
> >>> > Thanks,
> >>> > Max
> >>> >
> >>> > On Mon, Oct 26, 2015 at 7:12 AM, Henry Saputra <
> henry.sapu...@gmail.com 
> >>> > >
> >>> > wrote:
> >>> >
> >>> > > Thanks for the heads up, Suneel.
> >>> > >
> >>> > > Seemed like master Java api (api/java/index.html) is not being
> built:
> >>> > > https://ci.apache.org/projects/
> >>> > >
> >>> > > I have filed ticket with infra to help figure out why.
> >>> > >
> >>> > > - Henry
> >>> > >
> >>> > > On Sat, Oct 24, 2015 at 5:45 PM, Suneel Marthi  
> >>> > > wrote:
> >>> > > > https://ci.apache.org/projects/flink/flink-docs-master/api/java
> >>> > > >
> >>> > > > needs to be fixed.
> >>> > >
> >>> >
> >>>
>

Re: Broken link for master Javadocs

[Henry Saputra]

Ah thanks Max, sending to commits@ is good

- Henry

On Tue, Oct 27, 2015 at 2:35 AM, Maximilian Michels  wrote:
> Hi Henry,
>
> Yes, there is. The Commits@ list actually gets notifications on failures
> and recoveries. I figured sending them to dev@ would bother too many people
> because sometimes the infrastructure is flaky and it fails for no
> particular reason.
>
> Cheers,
> Max
>
> On Tue, Oct 27, 2015 at 4:18 AM, Henry Saputra 
> wrote:
>
>> Hi Max,
>>
>> Is there a way that dev@ list gets email notification if the build fail
>> for
>> the build bot?
>>
>> - Henry
>>
>> On Monday, October 26, 2015, Maximilian Michels  wrote:
>>
>> > Thanks for reporting, Suneel. On my machine the Java docs build.
>> >
>> > Here's the build log:
>> >
>> >
>> https://ci.apache.org/builders/flink-docs-master/builds/122/steps/Java%20%26%20Scala%20docs/logs/stdio
>> >
>> >
>> > [ERROR]
>> >
>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:35:
>> > error: not found: type ILoopCompat
>> > [ERROR]   extends ILoopCompat(in0, out0) {
>> > [ERROR]   ^
>> > [ERROR]
>> >
>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:29:
>> > error: too many arguments for constructor Object: ()Object
>> > [ERROR] class FlinkILoop(
>> > [ERROR] ^
>> > [ERROR]
>> >
>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:118:
>> > error: value createInterpreter is not a member of AnyRef
>> > [ERROR] super.createInterpreter()
>> > [ERROR]   ^
>> > [ERROR]
>> >
>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:120:
>> > error: not found: value addThunk
>> > [ERROR] addThunk {
>> > [ERROR] ^
>> > [ERROR]
>> >
>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:138:
>> > error: not found: value intp
>> > [ERROR] val vd = intp.virtualDirectory
>> > [ERROR]  ^
>> > [ERROR]
>> >
>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:186:
>> > error: not found: value echo
>> > [ERROR] echo(
>> > [ERROR] ^
>> > [ERROR]
>> >
>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala:151:
>> > error: value process is not a member of
>> > org.apache.flink.api.scala.FlinkILoop
>> > [ERROR]   repl.foreach(_.process(settings))
>> > [ERROR]      ^
>> > [ERROR]
>> >
>> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala:153:
>> > error: value closeInterpreter is not a member of
>> > org.apache.flink.api.scala.FlinkILoop
>> > [ERROR]   repl.foreach(_.closeInterpreter())
>> > [ERROR]  ^
>> > [ERROR] 8 errors found
>> >
>> >
>> > Not sure what the issue is. I'll try to look into later.
>> >
>> > Thanks,
>> > Max
>> >
>> > On Mon, Oct 26, 2015 at 7:12 AM, Henry Saputra > > >
>> > wrote:
>> >
>> > > Thanks for the heads up, Suneel.
>> > >
>> > > Seemed like master Java api (api/java/index.html) is not being built:
>> > > https://ci.apache.org/projects/
>> > >
>> > > I have filed ticket with infra to help figure out why.
>> > >
>> > > - Henry
>> > >
>> > > On Sat, Oct 24, 2015 at 5:45 PM, Suneel Marthi > > > wrote:
>> > > > https://ci.apache.org/projects/flink/flink-docs-master/api/java
>> > > >
>> > > > needs to be fixed.
>> > >
>> >
>>

Re: Broken link for master Javadocs

[Henry Saputra]

Hi Max,

Is there a way that dev@ list gets email notification if the build fail for
the build bot?

- Henry

On Monday, October 26, 2015, Maximilian Michels  wrote:

> Thanks for reporting, Suneel. On my machine the Java docs build.
>
> Here's the build log:
>
> https://ci.apache.org/builders/flink-docs-master/builds/122/steps/Java%20%26%20Scala%20docs/logs/stdio
>
>
> [ERROR]
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:35:
> error: not found: type ILoopCompat
> [ERROR]   extends ILoopCompat(in0, out0) {
> [ERROR]   ^
> [ERROR]
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:29:
> error: too many arguments for constructor Object: ()Object
> [ERROR] class FlinkILoop(
> [ERROR] ^
> [ERROR]
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:118:
> error: value createInterpreter is not a member of AnyRef
> [ERROR] super.createInterpreter()
> [ERROR]   ^
> [ERROR]
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:120:
> error: not found: value addThunk
> [ERROR] addThunk {
> [ERROR] ^
> [ERROR]
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:138:
> error: not found: value intp
> [ERROR] val vd = intp.virtualDirectory
> [ERROR]  ^
> [ERROR]
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala:186:
> error: not found: value echo
> [ERROR] echo(
> [ERROR] ^
> [ERROR]
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala:151:
> error: value process is not a member of
> org.apache.flink.api.scala.FlinkILoop
> [ERROR]   repl.foreach(_.process(settings))
> [ERROR]  ^
> [ERROR]
> /home/buildslave2/slave2/flink-docs-master/build/flink-staging/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala:153:
> error: value closeInterpreter is not a member of
> org.apache.flink.api.scala.FlinkILoop
> [ERROR]   repl.foreach(_.closeInterpreter())
> [ERROR]      ^
> [ERROR] 8 errors found
>
>
> Not sure what the issue is. I'll try to look into later.
>
> Thanks,
> Max
>
> On Mon, Oct 26, 2015 at 7:12 AM, Henry Saputra  >
> wrote:
>
> > Thanks for the heads up, Suneel.
> >
> > Seemed like master Java api (api/java/index.html) is not being built:
> > https://ci.apache.org/projects/
> >
> > I have filed ticket with infra to help figure out why.
> >
> > - Henry
> >
> > On Sat, Oct 24, 2015 at 5:45 PM, Suneel Marthi  > wrote:
> > > https://ci.apache.org/projects/flink/flink-docs-master/api/java
> > >
> > > needs to be fixed.
> >
>

Re: [DISCUSS] Introducing a review process for pull requests

[Henry Saputra]

+1 for that one.

The good news is all Flink committers have been very discipline about
reviews :)

On Monday, October 26, 2015, Fabian Hueske  wrote:

> Hi Matthias,
>
> those a good points. I did not really think about the different roles
> (technical, meta-role).
> My reasoning was that the shepherd should be able to give feedback on the
> PR in order to move it forward. This does not work so well if the shepherd
> is also the author of the PR (at least I am not so good at commenting on my
> own work ;-)).
>
> However, you and Henry are of course right. Every committer can commit by
> her/himself.
>
> How about we keep it as follows:
> Committers can of course push hot fixes and minor changes directly.
> Everything larger should go through a PR. If nobody signs up to shepherd
> the PR, the author can drive it forward her/himself. I don't think it is
> necessary that a committer explicitly sign-up as a shepherd of her/his own
> PR.
>
> Cheers, Fabian
>
>
> 2015-10-24 20:48 GMT+02:00 Henry Saputra  >:
>
> > Yes, as committer we trust you to do the right thing when committing the
> > code.
> > If a committer believe he/ she needs review,especially with large
> > patch, then he/ she should send PR for review.
> >
> >
> > - Henry
> >
> > On Sat, Oct 24, 2015 at 3:53 AM, Matthias J. Sax  > wrote:
> > > Great work!
> > >
> > > What puzzles me a little bit, is the shepherding of PR from committers.
> > > Why should it be a different committer?
> > >
> > > 1) As a committer, you don't even need to open a PR for small code
> > > changes at all (especially, if the committer is the most experience one
> > > regard a certain component/library). Just open a JIRA, fix it, and
> > commit.
> > >
> > > 2) It is clear, that if a PR is complex and maybe touches different
> > > components, feedback from the most experiences committer on those
> > > components should be given on the PR to ensure code quality. But the
> > > role of a shepherd is a "meta" role, if a understand the guideline
> > > correctly, and not a technical one (-> everybody should discuss,
> > > comment, accept, mark to get merged, and merge PRs). So why do we need
> a
> > > different shepherd there? I think, committers can drive their own PRs
> by
> > > themselves.
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 10/23/2015 06:00 PM, Fabian Hueske wrote:
> > >> Hi everybody,
> > >>
> > >> I created a wiki page for our Pull Request Process. [1]
> > >> Please review, refine, and comment.
> > >>
> > >> I would suggest to start following the process once 0.10 is released.
> > >> What do you think?
> > >>
> > >> Cheers,
> > >> Fabian
> > >>
> > >> [1]
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Pull+Request+Management
> > >>
> > >> 2015-10-19 17:53 GMT+02:00 Fabian Hueske  >:
> > >>
> > >>> Thanks for your feedback, Alex.
> > >>>
> > >>> Chesnay is right, we cannot modify the GH assignee field at the
> > moment. If
> > >>> this changes at some point, I would support your proposal.
> > >>>
> > >>> Regarding the PR - JIRA rule, this has been discussed as part of the
> > new
> > >>> contribution guidelines discussion [1].
> > >>> I agree, it is not always easy to draw a line here. However, if in
> > doubt I
> > >>> would go for the JIRA issue because it allows everybody to track the
> > issue
> > >>> later, esp. if it solves a bug that other people might run into as
> > well.
> > >>> In your concrete example, you could have referenced the original JIRA
> > >>> issue to remove Spargel from your PR.
> > >>>
> > >>> I also agree that the shepherd should not be the author of the PR.
> > >>> However, every committer can always commit to the master branch
> (unless
> > >>> somebody raised concerns of course). So committers should be free to
> > commit
> > >>> their own PRs, IMO.
> > >>>
> > >>> Cheers, Fabian
> > >>>
> > >>> [1]
> > >>>
> >
> http://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/%3CCAAdrtT0RLgqJgJVrK1zH_e_7VW1k8VTv4ChnumTRh7fLGA_zmw%40mail.gmail.com%3E
> > >>>
> >

Re: Broken link for master Javadocs

[Henry Saputra]

Thanks for the heads up, Suneel.

Seemed like master Java api (api/java/index.html) is not being built:
https://ci.apache.org/projects/

I have filed ticket with infra to help figure out why.

- Henry

On Sat, Oct 24, 2015 at 5:45 PM, Suneel Marthi  wrote:
> https://ci.apache.org/projects/flink/flink-docs-master/api/java
>
> needs to be fixed.

Re: [DISCUSS] Introducing a review process for pull requests

[Henry Saputra]

Yes, as committer we trust you to do the right thing when committing the code.
If a committer believe he/ she needs review,especially with large
patch, then he/ she should send PR for review.


- Henry

On Sat, Oct 24, 2015 at 3:53 AM, Matthias J. Sax  wrote:
> Great work!
>
> What puzzles me a little bit, is the shepherding of PR from committers.
> Why should it be a different committer?
>
> 1) As a committer, you don't even need to open a PR for small code
> changes at all (especially, if the committer is the most experience one
> regard a certain component/library). Just open a JIRA, fix it, and commit.
>
> 2) It is clear, that if a PR is complex and maybe touches different
> components, feedback from the most experiences committer on those
> components should be given on the PR to ensure code quality. But the
> role of a shepherd is a "meta" role, if a understand the guideline
> correctly, and not a technical one (-> everybody should discuss,
> comment, accept, mark to get merged, and merge PRs). So why do we need a
> different shepherd there? I think, committers can drive their own PRs by
> themselves.
>
>
> -Matthias
>
>
> On 10/23/2015 06:00 PM, Fabian Hueske wrote:
>> Hi everybody,
>>
>> I created a wiki page for our Pull Request Process. [1]
>> Please review, refine, and comment.
>>
>> I would suggest to start following the process once 0.10 is released.
>> What do you think?
>>
>> Cheers,
>> Fabian
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/Pull+Request+Management
>>
>> 2015-10-19 17:53 GMT+02:00 Fabian Hueske :
>>
>>> Thanks for your feedback, Alex.
>>>
>>> Chesnay is right, we cannot modify the GH assignee field at the moment. If
>>> this changes at some point, I would support your proposal.
>>>
>>> Regarding the PR - JIRA rule, this has been discussed as part of the new
>>> contribution guidelines discussion [1].
>>> I agree, it is not always easy to draw a line here. However, if in doubt I
>>> would go for the JIRA issue because it allows everybody to track the issue
>>> later, esp. if it solves a bug that other people might run into as well.
>>> In your concrete example, you could have referenced the original JIRA
>>> issue to remove Spargel from your PR.
>>>
>>> I also agree that the shepherd should not be the author of the PR.
>>> However, every committer can always commit to the master branch (unless
>>> somebody raised concerns of course). So committers should be free to commit
>>> their own PRs, IMO.
>>>
>>> Cheers, Fabian
>>>
>>> [1]
>>> http://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/%3CCAAdrtT0RLgqJgJVrK1zH_e_7VW1k8VTv4ChnumTRh7fLGA_zmw%40mail.gmail.com%3E
>>>
>>>
>>>
>>> 2015-10-17 13:00 GMT+02:00 Alexander Alexandrov <
>>> alexander.s.alexand...@gmail.com>:
>>>
 Also, regarding the "Each PR should be backed by a JIRA" rule - please
 don't make this strict and consider the relative effort to opening a JIRA
 as opposed to just opening a PR for small PRs that fix something obvious.

 For example, two days ago I opened two PRs while investigating a reported
 JIRA bug (FLINK-2858 ):

 https://github.com/apache/flink/pull/1259
 https://github.com/apache/flink/pull/1260

 1259 removes obsolete references to the (now removed 'flink-spargel' code
 from the POM), while 1260 updates a paragraph of the "How to Build"
 documentation and fixes some broken href anchors.

 Just my two cents here, but fixes (not only hotfixes) that

 * resolve minor and obvious issues with the existing code or the
 documentation,
 * can be followed by everybody just by looking at the diff

 should be just cherry-picked (and if needed amended) by a committer
 without
 too much unnecessary discussion and excluded from the "shepherding
 process".


 2015-10-17 12:32 GMT+02:00 Alexander Alexandrov <
 alexander.s.alexand...@gmail.com>:

> One suggestion from me: in GitHub you can make clear who the current
> sheppard is through the "Assignee" field in the PR (which can and IMHO
> should be different from the user who actually opened the request).
>
> Regards,
> A.
>
> 2015-10-16 15:58 GMT+02:00 Fabian Hueske :
>
>> Hi folks,
>>
>> I think we can split of the discussion of a PR meeting.
>>
>> Are there any more comments on the proposal itself?
>> Otherwise, I would go ahead and put the described process (modulo the
>> comments) into a document for our website.
>>
>> Cheers, Fabian
>>
>> 2015-10-07 16:57 GMT+02:00 Fabian Hueske :
>>
>>> I like the idea of meeting once a week to discuss about PRs as well.
>>>
>>> Regarding lingering PRs, you can simply sort the Flink PRs in Github
 by
>>> "least recently updated"
>>>
>>> Cheers,
>>> Fabian
>>>
>>> 2015-10-07 16:48 GMT+02:00 Theodore Vasiloudis <
>>> 

Re: [DISCUSS] Java code style

[Henry Saputra]

+1 for adding restriction for Javadoc at least at the header of public
classes and methods.

We did the exercise in Twill and seemed to work pretty well.

On Fri, Oct 23, 2015 at 1:34 AM, Maximilian Michels  wrote:
> I don't think lazily adding comments will work. However, I'm fine with
> adding all the checkstyle rules one module at a time (with a jira
> issue to keep track of the modules already converted). It's not going
> to happen that we lazily add comments because that's the reason why
> comments are missing in the first place...
>
> On Fri, Oct 23, 2015 at 12:05 AM, Henry Saputra  
> wrote:
>> Could we make certain rules to give warning instead of error?
>>
>> This would allow us to cherry-pick certain rules we would like people
>> to follow but not strictly enforced.
>>
>> - Henry
>>
>> On Thu, Oct 22, 2015 at 9:20 AM, Stephan Ewen  wrote:
>>> I don't think a "let add comments to everything" effort gives us good
>>> comments, actually. It just gives us checkmark comments that make the rules
>>> pass.
>>>
>>> On Thu, Oct 22, 2015 at 3:29 PM, Fabian Hueske  wrote:
>>>
>>>> Sure, I don't expect it to be free.
>>>> But everybody should be aware of the cost of adding this code style, i.e.,
>>>> spending a huge amount of time on reformatting and documenting code.
>>>>
>>>> Alternatively, we could drop the JavaDocs rule and make the transition
>>>> significantly cheaper.
>>>>
>>>> 2015-10-22 15:24 GMT+02:00 Till Rohrmann :
>>>>
>>>> > There ain’t no such thing as a free lunch and code style.
>>>> >
>>>> > On Thu, Oct 22, 2015 at 3:13 PM, Maximilian Michels 
>>>> > wrote:
>>>> >
>>>> > > I think we have to document all these classes. Code Style doesn't come
>>>> > > for free :)
>>>> > >
>>>> > > On Thu, Oct 22, 2015 at 3:09 PM, Fabian Hueske 
>>>> > wrote:
>>>> > > > Any ideas how to deal with the mandatory JavaDoc rule for existing
>>>> > code?
>>>> > > > Just adding empty headers to make the checkstyle pass or start a
>>>> > serious
>>>> > > > effort to add the missing docs?
>>>> > > >
>>>> > > > 2015-10-21 13:31 GMT+02:00 Matthias J. Sax :
>>>> > > >
>>>> > > >> Agreed. That's the reason why I am in favor of using vanilla Google
>>>> > code
>>>> > > >> style.
>>>> > > >>
>>>> > > >> On 10/21/2015 12:31 PM, Stephan Ewen wrote:
>>>> > > >> > We started out originally with mixed tab/spaces, but it ended up
>>>> > with
>>>> > > >> > people mixing spaces and tabs arbitrarily, and there is little way
>>>> > to
>>>> > > >> > enforce Matthias' specific suggestion via checkstyle.
>>>> > > >> > That's why we dropped spaces alltogether...
>>>> > > >> >
>>>> > > >> > On Wed, Oct 21, 2015 at 12:03 PM, Gyula Fóra <
>>>> gyula.f...@gmail.com>
>>>> > > >> wrote:
>>>> > > >> >
>>>> > > >> >> I think the nice thing about a common codestyle is that everyone
>>>> > can
>>>> > > set
>>>> > > >> >> the template in the IDE and use the formatting commands.
>>>> > > >> >>
>>>> > > >> >> Matthias's suggestion makes this practically impossible so -1 for
>>>> > > mixed
>>>> > > >> >> tabs/spaces from my side.
>>>> > > >> >>
>>>> > > >> >> Matthias J. Sax  ezt írta (időpont: 2015. okt.
>>>> > > 21.,
>>>> > > >> Sze,
>>>> > > >> >> 11:46):
>>>> > > >> >>
>>>> > > >> >>> I actually like tabs a lot, however, in a "mixed" style together
>>>> > > with
>>>> > > >> >>> spaces. Example:
>>>> > > >> >>>
>>>> > > >> >>> myVar.callMethod(param1, // many more
>>>> > > >> >>>

Re: [DISCUSS] Java code style

[Henry Saputra]

gt; >> >>>>> size (like 4).
>> > > >> >>>
>> > > >> >>>
>> > > >> >>> -Matthias
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>
>> > > >> >>> On 10/21/2015 11:06 AM, Ufuk Celebi wrote:
>> > > >> >>>> To summarize up to this point:
>> > > >> >>>>
>> > > >> >>>> - All are in favour of Google check style (with the following
>> > > possible
>> > > >> >>>> exceptions)
>> > > >> >>>> - Proposed exceptions so far:
>> > > >> >>>>   * Specific line length 100 vs. 120 characters
>> > > >> >>>>   * Keep tabs instead converting to spaces (this would
>> translate
>> > to
>> > > >> >>>> skipping/coming up with some indentation rules as well)
>> > > >> >>>>
>> > > >> >>>> If we keep tabs, we will have to specify the line length
>> relative
>> > > to a
>> > > >> >>> tab
>> > > >> >>>> size (like 4).
>> > > >> >>>>
>> > > >> >>>> Let’s keep the discussion going a little longer. I think it has
>> > > >> >> proceeded
>> > > >> >>>> in a very reasonable manner so far. Thanks for this!
>> > > >> >>>>
>> > > >> >>>> – Ufuk
>> > > >> >>>>
>> > > >> >>>> On Wed, Oct 21, 2015 at 10:29 AM, Fabian Hueske <
>> > fhue...@gmail.com
>> > > >
>> > > >> >>> wrote:
>> > > >> >>>>
>> > > >> >>>>> Thanks Max for checking the modifications by the Google code
>> > > style.
>> > > >> >>>>> It is very good to know, that the impact on the code base
>> would
>> > > not
>> > > >> be
>> > > >> >>> too
>> > > >> >>>>> massive. If the Google code style would have touched almost
>> > every
>> > > >> >> line,
>> > > >> >>> I
>> > > >> >>>>> would have been in favor of converting to spaces. However,
>> your
>> > > >> >>> assessment
>> > > >> >>>>> is a strong argument to continue with tabs, IMO.
>> > > >> >>>>>
>> > > >> >>>>> Regarding the line length limit, I personally find 100 chars
>> too
>> > > >> >> narrow
>> > > >> >>> but
>> > > >> >>>>> would be +1 for having a limit.
>> > > >> >>>>>
>> > > >> >>>>> +1 for discussing the Scala style in a separate thread.
>> > > >> >>>>>
>> > > >> >>>>> Fabian
>> > > >> >>>>>
>> > > >> >>>>> 2015-10-20 18:12 GMT+02:00 Maximilian Michels > >:
>> > > >> >>>>>
>> > > >> >>>>>> I'm a little less excited about this. You might not be aware
>> > but,
>> > > >> for
>> > > >> >>>>>> a large portion of the source code, we already follow the
>> > Google
>> > > >> >> style
>> > > >> >>>>>> guide. The main changes will be tabs->spaces and 80/100
>> > > characters
>> > > >> >>>>>> line limit.
>> > > >> >>>>>>
>> > > >> >>>>>> Out of curiosity, I ran the official Google Style Checkstyle
>> > > >> >>>>>> configuration to confirm my suspicion:
>> > > >> >>>>>>
>> > > >> >>>>>>
>> > > >> >>>>>
>> > > >> >>>
>> > > >> >>
>> > > >>
>> > >
>> >
>> https://github.com/checkstyle/checkstyle/blob/master/src/main/r

Re: [DISCUSS] Java code style

[Henry Saputra]

1) yes. Been dancing this issue for a while. Let's pull the trigger. Did
the exercise with Tachyon while back and did help readability and
homogeneity of code.

2) +1 for Google Java style with documented exceptions and explanation on
why.

On Tuesday, October 20, 2015, Ufuk Celebi  wrote:

> DISCLAIMER: This is not my personal idea, but a community discussion from
> some time ago. Don't kill the messenger.
>
> In March we were discussing issues with heterogeneity of the code [1]. The
> summary is that we had a consensus to enforce a stricter code style on our
> Java code base in order to make it easier to switch between projects and to
> have clear rules for new contributions. The main proposal in the last
> discussion was to go with Google's Java code style. Not all were fully
> satisfied with this, but still everyone agreed on some kind of style.
>
> I think the upcoming 0.10 release is a good point to finally go through
> with these changes (right after the release/branch-off).
>
> I propose to go with Google's Java code style [2] as proposed earlier.
>
> PROs:
> - Clear style guide available
> - Tooling like checkstyle rules, IDE plugins already available
>
> CONs:
> - Fully breaks our current style
>
> The main problem with this will be open pull requests, which will be harder
> to merge after all the changes. On the other hand, should pull requests
> that have been open for a long time block this? Most of the important
> changes will be merged for the release anyways. I think in the long run we
> will gain more than we loose by this (more homogenous code, clear rules).
> And it is questionable whether we will ever be able to do such a change in
> the future if we cannot do it now. The project will most likely grow and
> attract more contributors, at which point it will be even harder to do.
>
> Please make sure to answer the following points in the discussion:
>
> 1) Are you (still) in favour of enforcing stricter rules on the Java
> codebase?
>
> 2) If yes, would you be OK with the Google's Java code style?
>
> – Ufuk
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/flink-dev/201503.mbox/%3ccanc1h_von0b5omnwzxchtyzwhakeghbzvquyk7s9o2a36b8...@mail.gmail.com%3e
>
> [2] https://google.github.io/styleguide/javaguide.html
>

[jira] [Created] (FLINK-2872) Update the documentation for Scala part to add readFileOfPrimitives

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-2872:


 Summary: Update the documentation for Scala part to add 
readFileOfPrimitives
 Key: FLINK-2872
 URL: https://issues.apache.org/jira/browse/FLINK-2872
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Henry Saputra
Assignee: Henry Saputra
Priority: Minor


Currently the Scala part of the ExecutionEnvironment missing 
readFileOfPrimitives to create Dataset from file for primitive types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [URGENT] Documentation is broken for Dataset API programming guide

[Henry Saputra]

Hi Fabian,

Thanks so much for quick fix :)

- Henry

On Friday, October 16, 2015, Fabian Hueske  wrote:

> Hi Henry,
>
> thanks for spotting this problem.
> I forgot to add a closing  tag when adding the documentation for the
> outer joins.
> Just pushed a fix. Should be good again soon when the documentation was
> built and published on the CI servers.
>
> Thanks, Fabian
>
> 2015-10-17 3:48 GMT+02:00 Henry Saputra  >:
>
> > Guys,
> >
> > Need help to figure out what went wrong with the Dataset API programming
> > guide.
> >
> > Before:
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html
> >
> > Current:
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html
> >
> > Missing lots of sections after Transformations section.
> >
> >
> > - Henry
> >
>

[URGENT] Documentation is broken for Dataset API programming guide

[Henry Saputra]

Guys,

Need help to figure out what went wrong with the Dataset API programming guide.

Before:
https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/programming_guide.html

Current:
https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html

Missing lots of sections after Transformations section.


- Henry

Re: Flaky ScalaShellLocalStartupITCase

[Henry Saputra]

HI Sachin,

Let's file ticket for it so we could try to address it.
These flaky tests could cause confusion for contributors.

Thanks,


- Henry

On Tue, Oct 13, 2015 at 7:14 AM, Sachin Goel  wrote:
> Hi all
> Over the past few days, I've observed multiple failures of this test case.
> There is no single reason.
> 1. The assertion to check the output fails [either the finished status or
> the actual job output].
> https://travis-ci.org/apache/flink/jobs/84664370
> 2. JVM runs out of heap space [weirdly].
> 3. Just recently, observed a loss of the task manager.
> https://travis-ci.org/apache/flink/jobs/85108827
>
> Has someone else observed any such issues? It seems a simple enough test to
> display such flaky behavior.
>
> Cheers!
> Sachin
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685

[jira] [Created] (FLINK-2847) Fix flaky test in StormTestBase.testJob

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-2847:


 Summary: Fix flaky test in StormTestBase.testJob
 Key: FLINK-2847
 URL: https://issues.apache.org/jira/browse/FLINK-2847
 Project: Flink
  Issue Type: Bug
  Components: Tests
Reporter: Henry Saputra
Priority: Minor


{code}
testJob(org.apache.flink.storm.wordcount.WordCountLocalITCase)  Time elapsed: 
12.845 sec  <<< FAILURE!
java.lang.AssertionError: Different number of lines in expected and obtained 
result. expected:<801> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:305)
at 
org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:291)
at 
org.apache.flink.storm.wordcount.WordCountLocalITCase.postSubmit(WordCountLocalITCase.java:38)

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.815 sec - in 
org.apache.flink.storm.wordcount.BoltTokenizerWordCountWithNamesITCase
Running org.apache.flink.storm.wordcount.BoltTokenizerWordCountITCase
Running org.apache.flink.storm.wordcount.BoltTokenizerWordCountPojoITCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.55 sec - in 
org.apache.flink.storm.wordcount.BoltTokenizerWordCountPojoITCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.801 sec - in 
org.apache.flink.storm.wordcount.BoltTokenizerWordCountITCase

Results :

Failed tests: 
  
WordCountLocalITCase>StormTestBase.testJob:98->postSubmit:38->TestBaseUtils.compareResultsByLinesInMemory:291->TestBaseUtils.compareResultsByLinesInMemory:305
 Different number of lines in expected and obtained result. expected:<801> but 
was:<0>

Tests run: 11, Failures: 1, Errors: 0, Skipped: 0
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Extending and improving our "How to contribute" page

[Henry Saputra]

.g. Travis,
> >>>> >> Mockito, Guava, Log4j, FlinkMiniCluster, Unit testing vs IT cases,
> >>>> >> etc.
> >>>> >>
> >>>> >> 4 ) Restructuring the how to contribute guide
> >>>> >>
> >>>> >> Good idea to have a meta document that explains how contributing
> >>>> works
> >>>> >> in general, and another document for technical things.
> >>>> >>
> >>>> >>
> >>>> >> Cheers,
> >>>> >> Max
> >>>> >>
> >>>> >>
> >>>> >> On Thu, Sep 24, 2015 at 2:53 PM, Fabian Hueske  >
> >>>> wrote:
> >>>> >>>
> >>>> >>> Thanks everybody for feedback and comments.
> >>>> >>>
> >>>> >>> Regarding 1) and 2):
> >>>> >>>
> >>>> >>> I like the idea of keeping the discussion of new features and
> >>>> >> improvements
> >>>> >>> in JIRA as Kostas proposed.
> >>>> >>> Our coding guidelines [1] already request a JIRA issue for each
> pull
> >>>> >>> request.
> >>>> >>>
> >>>> >>> How about we highlight this requirement more prominently and
> follow
> >>>> this
> >>>> >>> rule more strict from now on.
> >>>> >>> JIRA issues for new features and improvements should clearly
> >>>> specify the
> >>>> >>> scope and requirements for the new feature / improvement.
> >>>> >>> The level of detail is up to the reporter of the issue, but the
> >>>> community
> >>>> >>> can request more detail or change the scope and requirements by
> >>>> >> discussion.
> >>>> >>> When a JIRA issue for a new feature or improvement is opened, the
> >>>> >> community
> >>>> >>> can start a discussion whether the feature is desirable for Flink
> >>>> or not.
> >>>> >>> Any contributor (including the reporter) can also attach a
> >>>> >>> "design-doc-requested" label to the issue. A design document can
> be
> >>>> >>> proposed by anybody, including the reporter or assignee of the
> JIRA
> >>>> >> issue.
> >>>> >>> However, the issue cannot be resolved and a corresponding PR not
> be
> >>>> >> merged
> >>>> >>> before a design document has been accepted by lazy consensus.
> >>>> Hence, an
> >>>> >>> assignee should propose a design doc before starting to code to
> >>>> avoid
> >>>> >> major
> >>>> >>> redesigns of the implementation.
> >>>> >>>
> >>>> >>> This way it is up to the community when to start a discussion
> about
> >>>> >> whether
> >>>> >>> a feature request is accepted or to request a design document. We
> >>>> can
> >>>> >> make
> >>>> >>> design documents mandatory for changes that touch the public API.
> >>>> >>>
> >>>> >>> Regarding 3):
> >>>> >>>
> >>>> >>> I agree with Vasia, that we should collect suggestions for common
> >>>> >> patterns
> >>>> >>> and also continuously update the coding guidelines.
> >>>> >>> @Henry, I had best practices (exception handling, tests, etc.) in
> >>>> mind.
> >>>> >>> Syntactic code style is important as well, but we should have a
> >>>> separate
> >>>> >>> discussion about that, IMO.
> >>>> >>>
> >>>> >>> Proposal for a design document template:
> >>>> >>>
> >>>> >>> - Overview of general approach
> >>>> >>> - API changes (changed interfaces, new / deprecated configuration
> >>>> >>> parameters, changed behavior)
> >>>> >>> - Main components and classes to touch
> >>>> >>>
> >>>> >>> Cheers,
> >>>> >>> Fabian
> >>>> >>>
>

[jira] [Created] (FLINK-2815) [REFACTOR] Remove Pact from class and file names since it is no longer valid reference

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-2815:


 Summary: [REFACTOR] Remove Pact from class and file names since it 
is no longer valid reference
 Key: FLINK-2815
 URL: https://issues.apache.org/jira/browse/FLINK-2815
 Project: Flink
  Issue Type: Task
Reporter: Henry Saputra
Assignee: Henry Saputra
Priority: Minor


Remove Pact word from class and file names in Apache Flink.

Pact was the name used in Stratosphere time to refer to concept of distributed 
datasets (similar to Flink Dataset).

It was used when Pact and Nephele still separate concept.

As part of 0.10 cleanup effort, let's remove the Pact names to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Release Flink 0.10

[Henry Saputra]

Robert, how hard it is to move these missing features to the new front end?

I prefer to remove the old one to prevent another duplicate things in Flink.

- Henry

On Fri, Oct 2, 2015 at 6:53 AM, Robert Metzger  wrote:
> The new one does not have access to the JobManager log file.
> Also, the graphs for the TaskManagers are missing.
>
>
> On Fri, Oct 2, 2015 at 3:51 PM, Stephan Ewen  wrote:
>
>> I would actually like to remove the old one, but I am okay with keeping it
>> and activating the new one by default
>>
>> On Fri, Oct 2, 2015 at 3:49 PM, Robert Metzger 
>> wrote:
>>
>> > The list from Kostas also contained the new JobManager front end.
>> >
>> > Do we want to enable it by default in the 0.10 release?
>> >
>> > Are we going to keep the old interface, or are we removing it?
>> >
>> > I'm voting for enabling the new one by default and keeping the old one
>> for
>> > the next release.
>> > What do you think?
>> >
>> >
>> > On Tue, Sep 29, 2015 at 8:41 PM, Henry Saputra 
>> > wrote:
>> >
>> > > Oops, I meant Spargel [1] =)
>> > >
>> > > [1]
>> > >
>> >
>> https://ci.apache.org/projects/flink/flink-docs-master/libs/spargel_guide.html
>> > >
>> > > On Tue, Sep 29, 2015 at 7:59 AM, Henry Saputra <
>> henry.sapu...@gmail.com>
>> > > wrote:
>> > > > +1 to the idea.
>> > > >
>> > > > I also think we need to remove Pregel if Gelly wants to graduate. It
>> > > already
>> > > > deprecated in 0.9.
>> > > >
>> > > > - Henry
>> > > >
>> > > >
>> > > > On Tuesday, September 29, 2015, Kostas Tzoumas 
>> > > wrote:
>> > > >>
>> > > >> Hi everyone,
>> > > >>
>> > > >> I would like to propose to cancel the 0.10-milestone release and go
>> > > >> directly for a 0.10 release as soon as possible.
>> > > >>
>> > > >> My opinion would be to focus this release on:
>> > > >> - Graduating the streaming API out of staging (depends on some open
>> > pull
>> > > >> requests)
>> > > >> - Master high availability
>> > > >> - New monitoring framework
>> > > >> - Graduating Gelly out of staging
>> > > >>
>> > > >> Flink 1.0 will probably come after 0.10, which gives us time to fix
>> > open
>> > > >> issues and freeze APIs.
>> > > >>
>> > > >> What do you think?
>> > > >>
>> > > >> Best,
>> > > >> Kostas
>> > >
>> >
>>

Re: Pulling Streaming out of staging and project restructure

[Henry Saputra]

+1

On Friday, October 2, 2015, Matthias J. Sax  wrote:

> It think, rename "flink-storm-compatibility-core" to just "flink-storm"
> would be the cleanest solution.
>
> So in flink-contrib there would be two modules:
>   - flink-storm
>   - flink-storm-examples
>
> Please let me know if you have any objection about it.
>
> -Matthias
>
> On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > Sure. Will do that.
> >
> > -Matthias
> >
> > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> >> @Matthias: How about getting rid of the storm-compatibility-parent and
> >> making the core and examples projects directly projects in "contrib"
> >>
> >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann  > wrote:
> >>
> >>> +1 for the new project structure. Getting rid of our code dump is a
> good
> >>> thing.
> >>>
> >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels  >
> >>> wrote:
> >>>
> >>>> +1 Matthias, let's limit the overhead this has for the module
> >>> maintainers.
> >>>>
> >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax  >
> >>> wrote:
> >>>>> I will commit something to flink-storm-compatibility tomorrow that
> >>>>> contains some internal package restructuring. I think, renaming the
> >>>>> three modules in this commit would be a smart move as both changes
> >>>>> result in merge conflicts when rebasing open PRs. Thus we can limit
> >>> this
> >>>>> pain to a single time. If no objections, I will commit those changes
> >>>>> tomorrow.
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> >>>>>> +1
> >>>>>>
> >>>>>> I like the idea moving "staging" projects into appropriate modules.
> >>>>>>
> >>>>>> While we are at it, I would like to propose changing "
> >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket
> list
> >>>>>> but would be nice if it is part of re-org.
> >>>>>> Supporting Hadoop in the connector implicitly means compatibility
> with
> >>>> Hadoop.
> >>>>>> Also same thing with "flink-storm-compatibility" to "flink-storm".
> >>>>>>
> >>>>>> - Henry
> >>>>>>
> >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen  >
> >>> wrote:
> >>>>>>> Hi all!
> >>>>>>>
> >>>>>>> We are making good headway with reworking the last parts of the
> >>> Window
> >>>> API.
> >>>>>>> After that, the streaming API should be good to be pulled out of
> >>>> staging.
> >>>>>>>
> >>>>>>> Since we are reorganizing the projects as part of that, I would
> shift
> >>>> a bit
> >>>>>>> more to bring things a bit more up to date.
> >>>>>>>
> >>>>>>> In this restructure, I would like to get rid of the "flink-staging"
> >>>>>>> project. Anyone who only uses the maven artifacts sees no
> difference
> >>>>>>> whether a project is in "staging" or not, so it does not help much
> to
> >>>> have
> >>>>>>> that directory structure.
> >>>>>>> On the other hand, projects have a tendency to linger in staging
> >>>> forever
> >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> >>>>>>>
> >>>>>>> The new structure could be
> >>>>>>>
> >>>>>>> flink-core
> >>>>>>> flink-java
> >>>>>>> flink-scala
> >>>>>>> flink-streaming-core
> >>>>>>> flink-streaming-scala
> >>>>>>>
> >>>>>>> flink-runtime
> >>>>>>> flink-runtime-web
> >>>>>>> flink-optimizer
> >>>>>>> flink-clients
> >>>>>>>
> >>>>>>> flink-shaded
> >>>>>>>   -> flink-shaded-hadoop

Re: Pulling Streaming out of staging and project restructure

[Henry Saputra]

+1

I like the idea moving "staging" projects into appropriate modules.

While we are at it, I would like to propose changing "
flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket list
but would be nice if it is part of re-org.
Supporting Hadoop in the connector implicitly means compatibility with Hadoop.
Also same thing with "flink-storm-compatibility" to "flink-storm".

- Henry

On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen  wrote:
> Hi all!
>
> We are making good headway with reworking the last parts of the Window API.
> After that, the streaming API should be good to be pulled out of staging.
>
> Since we are reorganizing the projects as part of that, I would shift a bit
> more to bring things a bit more up to date.
>
> In this restructure, I would like to get rid of the "flink-staging"
> project. Anyone who only uses the maven artifacts sees no difference
> whether a project is in "staging" or not, so it does not help much to have
> that directory structure.
> On the other hand, projects have a tendency to linger in staging forever
> (like avro, spargel, hbase, jdbc, ...)
>
> The new structure could be
>
> flink-core
> flink-java
> flink-scala
> flink-streaming-core
> flink-streaming-scala
>
> flink-runtime
> flink-runtime-web
> flink-optimizer
> flink-clients
>
> flink-shaded
>   -> flink-shaded-hadoop
>   -> flink-shaded-hadoop2
>   -> flink-shaded-include-yarn-tests
>   -> flink-shaded-curator
>
> flink-examples
>   -> (have all examples, Scala and Java, Batch and Streaming)
>
> flink-batch-connectors
>   -> flink-avro
>   -> flink-jdbc
>   -> flink-hadoop-compatibility
>   -> flink-hbase
>   -> flink-hcatalog
>
> flink-streaming-connectors
>   -> flink-connector-twitter
>   -> flink-streaming-examples
>   -> flink-connector-flume
>   -> flink-connector-kafka
>   -> flink-connector-elasticsearch
>   -> flink-connector-rabbitmq
>   -> flink-connector-filesystem
>
> flink-libraries
>   -> flink-gelly
>   -> flink-gelly-scala
>   -> flink-ml
>   -> flink-table
>   -> flink-language-binding
>   -> flink-python
>
>
> flink-scala-shell
>
> flink-test-utils
> flink-tests
> flink-fs-tests
>
> flink-contrib
>   -> flink-storm-compatibility
>   -> flink-storm-compatibility-examples
>   -> flink-streaming-utils
>   -> flink-tweet-inputformat
>   -> flink-operator-stats
>   -> flink-tez
>
> flink-quickstart
>   -> flink-quickstart-java
>   -> flink-quickstart-scala
>   -> flink-tez-quickstart
>
> flink-yarn
> flink-yarn-tests
>
> flink-dist
>
> flink-benchmark
>
>
> Let me know if that makes sense!
>
> Greetings,
> Stephan

[jira] [Created] (FLINK-2786) Remove Spargel from source code and update documentation in favor of Gelly

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-2786:


 Summary: Remove Spargel from source code and update documentation 
in favor of Gelly
 Key: FLINK-2786
 URL: https://issues.apache.org/jira/browse/FLINK-2786
 Project: Flink
  Issue Type: Task
  Components: Documentation, Spargel
Reporter: Henry Saputra


With Gelly getting more mature and ready to be top level project for Flink, we 
need to remove deprecated Spargel library from source and documentation.

Gelly copies the library needed from Spargel so there should not be hard 
dependency between the 2 modules.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Release Flink 0.10

[Henry Saputra]

Oops, I meant Spargel [1] =)

[1] 
https://ci.apache.org/projects/flink/flink-docs-master/libs/spargel_guide.html

On Tue, Sep 29, 2015 at 7:59 AM, Henry Saputra  wrote:
> +1 to the idea.
>
> I also think we need to remove Pregel if Gelly wants to graduate. It already
> deprecated in 0.9.
>
> - Henry
>
>
> On Tuesday, September 29, 2015, Kostas Tzoumas  wrote:
>>
>> Hi everyone,
>>
>> I would like to propose to cancel the 0.10-milestone release and go
>> directly for a 0.10 release as soon as possible.
>>
>> My opinion would be to focus this release on:
>> - Graduating the streaming API out of staging (depends on some open pull
>> requests)
>> - Master high availability
>> - New monitoring framework
>> - Graduating Gelly out of staging
>>
>> Flink 1.0 will probably come after 0.10, which gives us time to fix open
>> issues and freeze APIs.
>>
>> What do you think?
>>
>> Best,
>> Kostas

Re: Release Flink 0.10

[Henry Saputra]

+1 to the idea.

I also think we need to remove Pregel if Gelly wants to graduate. It
already deprecated in 0.9.

- Henry

On Tuesday, September 29, 2015, Kostas Tzoumas  wrote:

> Hi everyone,
>
> I would like to propose to cancel the 0.10-milestone release and go
> directly for a 0.10 release as soon as possible.
>
> My opinion would be to focus this release on:
> - Graduating the streaming API out of staging (depends on some open pull
> requests)
> - Master high availability
> - New monitoring framework
> - Graduating Gelly out of staging
>
> Flink 1.0 will probably come after 0.10, which gives us time to fix open
> issues and freeze APIs.
>
> What do you think?
>
> Best,
> Kostas
>

[jira] [Created] (FLINK-2775) [CLEANUP] Cleanup code as part of theme to be more consistent on Utils classes

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-2775:


 Summary: [CLEANUP] Cleanup code as part of theme to be more 
consistent on Utils classes
 Key: FLINK-2775
 URL: https://issues.apache.org/jira/browse/FLINK-2775
 Project: Flink
  Issue Type: Improvement
Reporter: Henry Saputra
Assignee: Henry Saputra
Priority: Minor


As part of the theme to help make the code more consistent, add cleanup to 
Utils classes:
-) Add final class modifier to the XXXUtils and XXXUtil classes to make sure 
can not be extended.
-) Add missing Javadoc header classs to some public classes.
-) Add private constructor to Utils classes to avoid instantiation.
-) Remove unused 
test/java/org/apache/flink/test/recordJobs/util/ConfigUtils.java class



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Extending and improving our "How to contribute" page

[Henry Saputra]

Thanks again, Fabian for starting the discussions.

For (1) and (2) I think it is good idea and will help people to
understand and follow the author thought process.
Following up with Stephan's reply, some new features solutions could
be explained thoroughly in the PR descriptions but some requires
additional reviews of the proposed design.
I like the idea of using tag in JIRA whether new features should or
should not being accompanied by design document.

Agree with (3) and (4).
As for (3) are you thinking about more of style of code syntax via
checkstyle updates, or best practices in term of no mutable state if
possible, throw precise Exception if possible for interfaces, etc. ?

- Henry




On Wed, Sep 23, 2015 at 9:31 AM, Stephan Ewen  wrote:
> Thanks, Fabian for driving this!
>
> I agree with your points.
>
> Concerning Vasia's comment to not raise the bar too high:
> That is true, the requirements should be reasonable. We can definitely tag
> issues as "simple" which means they do not require a design document. That
> should be more for new features and needs not be very detailed.
>
> We could also make the inverse, meaning we explicitly tag certain issues as
> "requires design document".
>
> Greetings,
> Stephan
>
>
>
>
> On Wed, Sep 23, 2015 at 5:05 PM, Vasiliki Kalavri > wrote:
>
>> Hi,
>>
>> I agree with you Fabian. Clarifying these issues in the "How to Contribute"
>> guide will save lots of time both to reviewers and contributors. It is a
>> really disappointing situation when someone spends time implementing
>> something and their PR ends up being rejected because either the feature
>> was not needed or the implementation details were never agreed on.
>>
>> That said, I think we should also make sure that we don't raise the bar too
>> high for simple contributions.
>>
>> Regarding (1) and (2), I think we should clarify what kind of
>> additions/changes require this process to be followed. e.g. do we need to
>> discuss additions for which JIRAs already exist? Ideas described in the
>> roadmaps? Adding a new algorithm to Gelly/Flink-ML?
>>
>> Regarding (3), maybe we can all suggest some examples/patterns that we've
>> seen when reviewing PRs and then choose the most common (or all).
>>
>> (4) sounds good to me.
>>
>> Cheers,
>> Vasia.
>>
>> On 23 September 2015 at 15:08, Kostas Tzoumas  wrote:
>>
>> > Big +1.
>> >
>> > For (1), a discussion in JIRA would also be an option IMO
>> >
>> > For (2), let us come up with few examples on what constitutes a feature
>> > that needs a design doc, and what should be in the doc (IMO
>> > architecture/general approach, components touched, interfaces changed)
>> >
>> >
>> >
>> > On Wed, Sep 23, 2015 at 2:24 PM, Fabian Hueske 
>> wrote:
>> >
>> > > Hi everybody,
>> > >
>> > > I guess we all have noticed that the Flink community is quickly growing
>> > and
>> > > more and more contributions are coming in. Recently, a few
>> contributions
>> > > proposed new features without being discussed on the mailing list. Some
>> > of
>> > > these contributions were not accepted in the end. In other cases, pull
>> > > requests had to be heavily reworked because the approach taken was not
>> > the
>> > > best one. These are situations which should be avoided because both the
>> > > contributor as well as the person who reviewed the contribution
>> invested
>> > a
>> > > lot of time for nothing.
>> > >
>> > > I had a look at our “How to contribute” and “Coding guideline” pages
>> and
>> > > think, we can improve them. I see basically two issues:
>> > >
>> > >   1. The documents do not explain how to propose and discuss new
>> features
>> > > and improvements.
>> > >   2. The documents are quite technical and the structure could be
>> > improved,
>> > > IMO.
>> > >
>> > > I would like to improve these pages and propose the following
>> additions:
>> > >
>> > >   1. Request contributors and committers to start discussions on the
>> > > mailing list for new features. This discussion should help to figure
>> out
>> > > whether such a new feature is a good fit for Flink and give first
>> > pointers
>> > > for a design to implement it.
>> > >   2. Require contributors and committers to write design documents for
>> > all
>> > > new features and major improvements. These documents should be attached
>> > to
>> > > a JIRA issue and follow a template which needs to be defined.
>> > >   3. Extend the “Coding Style Guides” and add patterns that are
>> commonly
>> > > remarked in pull requests.
>> > >   4. Restructure the current pages into three pages: a general guide
>> for
>> > > contributions and two guides for how to contribute to code and website
>> > with
>> > > all technical issues (repository, IDE setup, build system, etc.)
>> > >
>> > > Looking forward for your comments,
>> > > Fabian
>> > >
>> >
>>

[jira] [Created] (FLINK-2650) Fix broken link in the Table API doc

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-2650:


 Summary: Fix broken link in the Table API doc
 Key: FLINK-2650
 URL: https://issues.apache.org/jira/browse/FLINK-2650
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Henry Saputra


In the Table API doc [1] there is a broken link to programming guide:
https://ci.apache.org/projects/flink/flink-docs-master/libs/programming_guide.html

which returns 404

[1] https://ci.apache.org/projects/flink/flink-docs-master/libs/table.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSSION] Release current master as 0.9.1 (mod few changes)

[Henry Saputra]

I am +1 for this idea.


- Henry

On Wed, Aug 26, 2015 at 3:07 AM, Robert Metzger  wrote:
> I'm against using the current master for 0.9.1.
> It contains too many changes, posing the risk of changing APIs/semantics/...
> I agree with Max that there was no consensus or discussion regarding the
> scope of the 0.10 release.
>
> How about we release a 0.9.1 version containing all fixes we can easily
> apply to the release-0.9 branch and a 0.10-milestone1 off our current
> "master".
>
> Ideally we release 0.10-milestone1 before merging new major changes such as
> master high availablilty.
>
>
>
>
>
> On Wed, Aug 26, 2015 at 11:42 AM, Maximilian Michels  wrote:
>
>> I might not have made my point clear but I wrote:
>>
>> >However, I can see applying a subset of carefully selected commits from
>> the
>> >master branch as an option.
>>
>> And you wrote:
>>
>> >We can also try and "rebase" a fork of the maser to the "0.9" branch,
>> where
>> >we select something like 70%-80% of the commits (all fixes and reworks)
>> and
>> >drop the API beaking ones.
>>
>> So we essentially agree on this issue. Or not?
>>
>> Forking the entire branch is not an option for me for the 0.9.1
>> release. I think we could do an 0.10 release since we never decided
>> 0.10 would freeze the streaming API. These are just statements by
>> individual committers and do not represent the community's opinion.
>>
>>
>>
>> On Wed, Aug 26, 2015 at 11:12 AM, Stephan Ewen  wrote:
>> > @mxm: I know the textbook theory ;-)
>> >
>> > The whole point here is that it is not possible to do that. Fixes and
>> major
>> > reworks changes go together so tightly that you can get none or both.
>> >
>> > Not having the fixes voids the purpose of the bugfix release. Having both
>> > means it is hard to guarantee all changes are non-breaking.
>> >
>> > On Wed, Aug 26, 2015 at 11:08 AM, Maximilian Michels 
>> wrote:
>> >
>> >> A bugfix release should not be forked from the current master. It is
>> >> very hard to asses whether we don't break the API because there are
>> >> many small fixes going in almost daily. However, I can see applying a
>> >> subset of carefully selected commits from the master branch as an
>> >> option. Only those commits should be cherry-picked which are required
>> >> to fix the streaming issues.
>> >>
>> >> On Wed, Aug 26, 2015 at 10:50 AM, Stephan Ewen 
>> wrote:
>> >> > @Aljoscha: Correct me if I am wrong, but did the change actually break
>> >> > anything user facing?
>> >> >
>> >> > The source function and source context interface look still the same.
>> The
>> >> > underlying changes to introduce watermarks should not be visible to
>> any
>> >> > user anyways at this point (if we remove the additional source
>> contexts)
>> >> >
>> >> > On Wed, Aug 26, 2015 at 10:46 AM, Stephan Ewen 
>> wrote:
>> >> >
>> >> >> The timestamp thing is one of the biggest questions.
>> >> >>
>> >> >> The fixes that came as part of that pull request are crucial and
>> hard to
>> >> >> pull out of the change.
>> >> >>
>> >> >> On Wed, Aug 26, 2015 at 10:44 AM, Aljoscha Krettek <
>> aljos...@apache.org
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >>> I don't think we had to many API breaking changes. If everyone was
>> >> >>> careful,
>> >> >>> maybe these are even it:
>> >> >>> https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
>> >> >>>
>> >> >>> I added my breaking stuff there. And of course the whole Timestamp
>> >> thing
>> >> >>> is
>> >> >>> a change, but it does not affect the normal source interface.
>> >> >>>
>> >> >>> On Wed, 26 Aug 2015 at 10:24 Stephan Ewen  wrote:
>> >> >>>
>> >> >>> > We can also try and "rebase" a fork of the maser to the "0.9"
>> branch,
>> >> >>> where
>> >> >>> > we select something like 70%-80% of the commits (all fixes and
>> >> reworks)
>> >> >>> and
>> >> >>> > drop the API beaking ones.
>> >> >>> >
>> >> >>> > Let me try this and see how feasible it is...
>> >> >>> >
>> >> >>> > On Wed, Aug 26, 2015 at 9:52 AM, Ufuk Celebi 
>> wrote:
>> >> >>> >
>> >> >>> > > I think you are the best one to assess this at the moment since
>> you
>> >> >>> are
>> >> >>> > > doing the hard work of back porting the changes.
>> >> >>> > >
>> >> >>> > > Are you suggesting this, because it is a) less
>> error-prone/easier
>> >> or
>> >> >>> b)
>> >> >>> > > faster to do?
>> >> >>> > >
>> >> >>> > > For those that haven't followed the discussion: Stephan is back
>> >> >>> porting
>> >> >>> > > fixes for the streaming fault tolerance. There is consensus that
>> >> the
>> >> >>> > > changes need to be in the bug fix release. So it's definitely
>> not
>> >> an
>> >> >>> > option
>> >> >>> > > to skip it.
>> >> >>> > >
>> >> >>> > > In general I would like to keep our established process of back
>> >> >>> porting
>> >> >>> > > fixes to the release-X branch. But given the importance of the
>> to
>> >> be
>> >> >>> back
>> >> >>> > > ported fixes and the difficulty of back porting it, I think your
>> >> >>> > suggestion
>> >> >>> > > is reason

[jira] [Created] (FLINK-2574) Remove Spargel from master in next release

[Henry Saputra (JIRA)]

Henry Saputra created FLINK-2574:


 Summary: Remove Spargel from master in next release
 Key: FLINK-2574
 URL: https://issues.apache.org/jira/browse/FLINK-2574
 Project: Flink
  Issue Type: Task
  Components: Spargel
Reporter: Henry Saputra
 Fix For: 0.10


With Gelly getting more mature and more people start using Flink, I propose to 
remove Spargel from master in next release.

We already deprecate it in 0.9 so I think it is  good time to remove it in 
favor of Gelly.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Gelly Blog Post

[Henry Saputra]

Awesome +1

- Henry

On Tue, Aug 25, 2015 at 6:39 AM, Ufuk Celebi  wrote:
> Blog post is live:
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html
>
> Feel free to spread the word. :)
>
> On Tue, Aug 25, 2015 at 10:29 AM, Ufuk Celebi 
> wrote:
>
>> Andra just opened a PR for the post.
>>
>> Since there was consensus to publish, I will publish it later today. If
>> there are any last minute objections, speak now or forever hold your peace.
>>
>> The PR is here: https://github.com/apache/flink-web/pull/7
>>
>> – Ufuk
>>
>>
>> On 23 Aug 2015, at 16:28, Vasiliki Kalavri 
>> wrote:
>>
>> +1!
>>
>> On 23 August 2015 at 15:30, Kostas Tzoumas  wrote:
>>
>> I think it reads very well, time to publish :-)
>>
>> On Sun, Aug 23, 2015 at 12:37 PM, Martin Junghanns <
>> m.jungha...@mailbox.org>
>> wrote:
>>
>> Hi,
>>
>> this is a very nice blog post! I added some minor comments. I am really
>> excited about the future work on partition-centric computation and graph
>> partitioning!
>>
>> And thanks for guiding me to DataSetUtils.zipWithUniqueIds()! I should
>> switch to 0.10-SNAPSHOT :)
>>
>> Best,
>> Martin
>>
>>
>>
>>
>> On 22.08.2015 22:46, Andra Lungu wrote:
>>
>> Hi,
>>
>> I updated the Gelly Blog Post.
>>
>>
>>
>> https://docs.google.com/document/d/1FMtpwKSE3kY7RfH082LzQpWrY6o-fdZVxqambIiC_rU/edit?usp=sharing
>>
>>
>> Let me know if there are comments or further suggestions. Otherwise,
>>
>> we'll
>>
>> kick this one out of the nest, to see if it flies :)
>>
>>
>> On Wed, Jun 3, 2015 at 11:37 AM, Vasiliki Kalavri <
>> vasilikikala...@gmail.com
>>
>> wrote:
>>
>>
>> Hi all,
>>
>>
>> thank you so much for your comments!
>>
>> Most of them have been addressed already.
>> I would also like to add Fabian's suggestion on providing a library
>> method
>> for initializing vertices with unique IDs and we'd also like to run
>>
>> some
>>
>> experiments with the Echo Nest dataset.
>>
>> After that, I think it'd be ready to publish :)
>>
>> -Vasia.
>>
>> On 26 May 2015 at 14:57, Aljoscha Krettek  wrote:
>>
>> Very good, I made some comments and suggestions inline.
>>
>>
>> On Tue, May 26, 2015 at 2:36 PM, Stephan Ewen 
>>
>> wrote:
>>
>>
>> Wow, this is impressive :-)
>>
>> Amazing work, Gelly folks!
>>
>> On Tue, May 26, 2015 at 10:03 AM, Andra Lungu >
>>
>>
>> wrote:
>>
>>
>> Hey everyone,
>>
>>
>> We are very excited to share the first stable draft of the Gelly
>>
>> blog
>>
>>
>> post
>>
>>
>> with you :D
>>
>>
>>
>>
>>
>>
>>
>>
>> https://docs.google.com/document/d/1FMtpwKSE3kY7RfH082LzQpWrY6o-fdZVxqambIiC_rU/edit?usp=sharing
>>
>>
>>
>> *Feedback* is welcome, as usual!
>> Andra
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Data Artisans GmbH | Tempelhofer Ufer 17 | 10963 Berlin
>
> i...@data-artisans.com
> +49-30-55599146
>
> Registered at Amtsgericht Charlottenburg - HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Stephan Ewen

Re: [ANNOUNCEMENT] Please add more Description and information to JIRA issues

[Henry Saputra]

Thanks Max, Fabian,

As Fabian had mentioned, there are several JIRAs being opened without
details info.
We need to ask for more description of those JIRAs even though they
have Github PRs for proposed solution.

- Henry

On Mon, Aug 24, 2015 at 1:41 AM, Maximilian Michels  wrote:
> +1 for your initiative, Henry.
>
> We should have a very high standard on JIRA descriptions. That helps people
> to understand issues correctly and makes it much easier for new people to
> join the development.
>
> On Mon, Aug 17, 2015 at 12:26 AM, Fabian Hueske  wrote:
>
>> +1 for what Henry said.
>>
>> Recently, several JIRA issues were opened without a detailed description
>> but which were immediately addressed by a pull request.
>> This is probably because our coding guidelines state that each PR should
>> refer to a JIRA issue [1].
>> The rational behind this guideline is to have a JIRA ticket for
>> documentation to which the commit message links. Without a proper
>> description of the issue, this purpose is not met.
>>
>> I would like to ask everybody to provide descriptions that help to
>> understand the issue.
>>
>> Thanks,
>> Fabian
>>
>> [1] http://flink.apache.org/coding-guidelines.html
>>
>> 2015-08-16 10:35 GMT+02:00 Henry Saputra :
>>
>> > Hi All,
>> >
>> > Apology for the cross-posting.
>> >
>> > Please remember to add detail Description in the JIRA ticket to help
>> > reproduce and prioritize the issues.
>> > Assume that some other people will try to solve and reproduce the issues.
>> >
>> > Adding information in JIRA issues is to help describe the issue and
>> > steps to reproduce.
>> > Adding information in Github PR is to describe how and what being
>> > changed or added to solve the problem.
>> >
>> > We will add guide about adding more Description and step to reproduce
>> > for JIRA issues in the how to contribute page [1] to promote open and
>> > transparent development in Apache Flink.
>> >
>> >
>> > Thanks,
>> >
>> > - Henry
>> >
>> >
>> > [1] https://flink.apache.org/how-to-contribute.html
>> >
>>

Re: [ANNOUNCE] New Committer Chesnay Schepler

[Henry Saputra]

Welcome Chesnay!

On Thu, Aug 20, 2015 at 2:18 AM, Robert Metzger  wrote:
> The Project Management Committee (PMC) for Apache Flink has asked Chesnay
> Schepler to become a committer and we are pleased to announce that they
> have accepted.
>
> Chesnay has been very involved with the Flink project since its pre-ASF
> days. He has worked on several components including the Java API,
> documentation, and execution engine. Recently he made a big contribution
> and added a Python API to Flink.
>
> Being a committer enables easier contribution to the project since there is
> no need to go via the pull request submission process. This should enable
> better productivity. Being a PMC member enables assistance with the
> management and to guide the direction of the project.