Re: Flink ML - Vector and DenseVector

We had a discussion about the "LabeledVector" class. But now this is a discussion about the Vector and DenseVector :) Am 18.01.2016 um 22:29 schrieb Till Rohrmann: Didn't we just had the discussion in another email thread? On Jan 18, 2016 8:55 PM, "Hilmi Yildirim"

Re: [ANNOUNCE] Chengxiang Li added as committer

Congrats! Welcome Chengxiang Li! > On Jan 19, 2016, at 7:13 PM, Vasiliki Kalavri > wrote: > > Congratulations! Welcome Chengxiang Li! > > On 19 January 2016 at 11:02, Fabian Hueske wrote: > >> Hi everybody, >> >> I'd like to announce that

Re: [ANNOUNCE] Chengxiang Li added as committer

Congratulations! Welcome Chengxiang Li! On 19 January 2016 at 11:02, Fabian Hueske wrote: > Hi everybody, > > I'd like to announce that Chengxiang Li accepted the PMC's offer to become > a committer of the Apache Flink project. > > Please join me in welcoming Chengxiang Li! >

What is the topic & offset used for in KeyedDeserializationSchema.deserialize()?

Hi devs, I need a little help on clarification of what the arguments "topic" and "offset" is used for in KeyedDeserializationSchema.deserialize(). The main issue is that I'm currently in progress of implementing Flink Kinesis Consumer, and Kinesis offsets, unlike Kafka offsets which are

Re: [ANNOUNCE] Chengxiang Li added as committer

Pleased to have you with us Chengxiang! Cheers, Max On Tue, Jan 19, 2016 at 11:13 AM, Chiwan Park wrote: > Congrats! Welcome Chengxiang Li! > >> On Jan 19, 2016, at 7:13 PM, Vasiliki Kalavri >> wrote: >> >> Congratulations! Welcome Chengxiang

Re: What is the topic & offset used for in KeyedDeserializationSchema.deserialize()?

Hi Gordon, You may use "topic" and "offset" for whatever you like. Note that this is just an interface. If it does not work for your Kinesis adapter, you may create a new interface. For existing usage of the KeyedDeserializationSchema, please have a look at the FlinkKafkaConsumer. Cheers, Max

Re: What is the topic & offset used for in KeyedDeserializationSchema.deserialize()?

Hi Max, Thanks for the quick response and clarification :) I got a bit confused and thought that Flink internals would be accessing this interface too. Cheers, Gordon -- View this message in context:

Questions for a better understanding of the internals of data exchange between tasks

Hello, I list some questions gathered while reading documentation on Flink's internals and I am grateful to receive your answers. 1) How is the JobManager involved in the communication between tasks running in task slots on TaskManagers? >From [1] it appears to me that, as part of the control

Re: What is the topic & offset used for in KeyedDeserializationSchema.deserialize()?

Hi Gordon, thank you for starting the discussion. I think in fact the KeyedDeserializationSchema is located in the wrong package. Its methods are very Kafka specific, maybe I should move them there. How would the deserializationSchema for Kinesis look like? Does the Kinesis API return byte[] ?

Re: What is the topic & offset used for in KeyedDeserializationSchema.deserialize()?

Hi Robert, +1 for a change to where the KeyedDeserializationSchema is located. I was just starting to wonder how I should name the Kinesis's deserializationSchema if I were to create another one in the same package. For Kinesis, the API returns String for key, byte[] for value, String for

Re: What is the topic & offset used for in KeyedDeserializationSchema.deserialize()?

I'll relocate the KeyedDeserializationSchema as part of the Kafka 0.9.0.0 support (its a pending pull request I'll merge soon) On Tue, Jan 19, 2016 at 12:20 PM, Tzu-Li (Gordon) Tai wrote: > Hi Robert, > > +1 for a change to where the KeyedDeserializationSchema is located. I

Re: What is the topic & offset used for in KeyedDeserializationSchema.deserialize()?

While the de-serializations schema is not used by the Flink internals, I think the initial idea was to use it across different sources/sinks (like Kafka, Socket, RabbitMQ, ...) Does it make sense to have a KafkaDeSerializationSchema, and then wrap the common serialization schemata? On Tue, Jan

[jira] [Created] (FLINK-3258) Merge AbstractInvokable's registerInputOutput and invoke

Ufuk Celebi created FLINK-3258: -- Summary: Merge AbstractInvokable's registerInputOutput and invoke Key: FLINK-3258 URL: https://issues.apache.org/jira/browse/FLINK-3258 Project: Flink Issue

Re: [ANNOUNCE] Chengxiang Li added as committer

Good to have you on board! On Tue, Jan 19, 2016 at 11:29 AM, Maximilian Michels wrote: > Pleased to have you with us Chengxiang! > > Cheers, > Max > > On Tue, Jan 19, 2016 at 11:13 AM, Chiwan Park > wrote: > > Congrats! Welcome Chengxiang Li! > > > >> On

[jira] [Created] (FLINK-3259) Redirect programming guides to new layout

Ufuk Celebi created FLINK-3259: -- Summary: Redirect programming guides to new layout Key: FLINK-3259 URL: https://issues.apache.org/jira/browse/FLINK-3259 Project: Flink Issue Type: Improvement

Re: [ANNOUNCE] Chengxiang Li added as committer

Welcome Chengxiang!! On Tue, Jan 19, 2016 at 12:31 PM, Stephan Ewen wrote: > Good to have you on board! > > On Tue, Jan 19, 2016 at 11:29 AM, Maximilian Michels > wrote: > > > Pleased to have you with us Chengxiang! > > > > Cheers, > > Max > > > > On Tue, Jan

Re: [ANNOUNCE] Chengxiang Li added as committer

Congrats and welcome Chengxiang!! :) On 01/19/2016 12:56 PM, Kostas Tzoumas wrote: > Welcome Chengxiang!! > > On Tue, Jan 19, 2016 at 12:31 PM, Stephan Ewen wrote: > >> Good to have you on board! >> >> On Tue, Jan 19, 2016 at 11:29 AM, Maximilian Michels >>

[jira] [Created] (FLINK-3263) Log task statistics on TaskManager

Greg Hogan created FLINK-3263: - Summary: Log task statistics on TaskManager Key: FLINK-3263 URL: https://issues.apache.org/jira/browse/FLINK-3263 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Git force pushing and deletion of branchs

Thank you :D On Tue, Jan 19, 2016 at 6:41 PM, Till Rohrmann wrote: > Thanks Max :-) > > On Tue, Jan 19, 2016 at 6:05 PM, Fabian Hueske wrote: > > > Thanks Max! > > > > 2016-01-19 18:04 GMT+01:00 Maximilian Michels : > > > > > I've filed

Re: Questions for a better understanding of the internals of data exchange between tasks

On Tue, Jan 19, 2016 at 11:37 AM, Camelia Elena Ciolac wrote: > Hello, > > I list some questions gathered while reading documentation on Flink's > internals and I am grateful to receive your answers. > > 1) How is the JobManager involved in the communication between tasks >

Fwd: integration with a scheduler

> > Hi, > > I am planning to integrate flink with our job scheduler product to execute > jobs - especially bathc like - on flink which may be the part of some > other DAG style job chain. > > I need some control ablities like start, stop, suspend, get status... > > Where shold i go through ? >

Re: Weird test-source issue

Yeah, we saw this as well this morning, in a job that triggers checkpoints super fast (50msecs). I think we have a good fix figured out, let's solve this for 1.0... On Tue, Jan 19, 2016 at 3:25 PM, Gyula Fóra wrote: > I just got back to this issue. The problem wasn't with

Re: Weird test-source issue

It is nice to see that we converge on the issues we find. Means that this is getting pretty stable :-) On Tue, Jan 19, 2016 at 8:17 PM, Stephan Ewen wrote: > Yeah, we saw this as well this morning, in a job that triggers checkpoints > super fast (50msecs). > > I think we have

RE: Questions for a better understanding of the internals of data exchange between tasks

Dear Ufuk, Thank you very much for this detailed explanation, it helped me understand. So, many many thanks! Camelia From: Ufuk Celebi [u...@apache.org] Sent: Tuesday, January 19, 2016 7:16 PM To: dev@flink.apache.org Subject: Re: Questions for a

Re: What is the topic & offset used for in KeyedDeserializationSchema.deserialize()?

Hi Stephan, A comment on this. For KeyedDeserializationSchema, I don't think it is necessary. As previously explained, the interfaces for the KeyedDeserializationSchema of Kafka / Kinesis can be quite different, and may also be specific for future external systems that we might implement

[jira] [Created] (FLINK-3260) ExecutionGraph gets stuck in state FAILING

Stephan Ewen created FLINK-3260: --- Summary: ExecutionGraph gets stuck in state FAILING Key: FLINK-3260 URL: https://issues.apache.org/jira/browse/FLINK-3260 Project: Flink Issue Type: Bug

Re: Weird test-source issue

I just got back to this issue. The problem wasn't with the locking but that the StreamTask wasn't in running state before the first checkpoint trigger message. I actually just saw your JIRA as well, funny... :) Regards, Gyula Stephan Ewen ezt írta (időpont: 2016. jan. 8., P,

Re: [ANNOUNCE] Chengxiang Li added as committer

Congrats Chengxiang! Really pleased to have you on board > On 19 Jan 2016, at 13:16, Matthias J. Sax wrote: > > Congrats and welcome Chengxiang!! :) > > On 01/19/2016 12:56 PM, Kostas Tzoumas wrote: >> Welcome Chengxiang!! >> >> On Tue, Jan 19, 2016 at 12:31 PM, Stephan Ewen

[jira] [Created] (FLINK-3261) Tasks should eagerly report back when they cannot start a checkpoint

Stephan Ewen created FLINK-3261: --- Summary: Tasks should eagerly report back when they cannot start a checkpoint Key: FLINK-3261 URL: https://issues.apache.org/jira/browse/FLINK-3261 Project: Flink

[jira] [Created] (FLINK-3262) Remove fuzzy versioning from Bower dependencies

Greg Hogan created FLINK-3262: - Summary: Remove fuzzy versioning from Bower dependencies Key: FLINK-3262 URL: https://issues.apache.org/jira/browse/FLINK-3262 Project: Flink Issue Type:

RE: [ANNOUNCE] Chengxiang Li added as committer

Thanks everyone, it's always great to collaborate with you guys, look forward to contribute more on Flink. Thanks Chengxiang -Original Message- From: Paris Carbone [mailto:par...@kth.se] Sent: Tuesday, January 19, 2016 9:24 PM To: dev@flink.apache.org Subject: Re: [ANNOUNCE]

Re: Emitting to non-declared output stream

Please ignore. Wrong list. Sorry! On 01/19/2016 03:25 PM, Matthias J. Sax wrote: > Hi, > > currently, I am using Storm 0.9.3. For first tests on a new topology, I > use LocalCluster. It happened to me, that I emitted tuples to an output > stream, that I did never declare (and thus not connect

Re: [ANNOUNCE] Chengxiang Li added as committer

Usually the first thing is to add yourself to the "team" page :-) http://flink.apache.org/community.html#people On Tue, Jan 19, 2016 at 4:29 PM, Li, Chengxiang wrote: > Thanks everyone, it's always great to collaborate with you guys, look > forward to contribute more

Re: [DISCUSS] Git force pushing and deletion of branchs

I've filed an issue at infra to protect the master: https://issues.apache.org/jira/browse/INFRA-11088 On Fri, Jan 15, 2016 at 3:40 PM, Maximilian Michels wrote: > +1 for a protected master. > +1 for creating release tags under rel/. > > On Thu, Jan 14, 2016 at 10:07 AM, Chiwan

Re: [DISCUSS] Git force pushing and deletion of branchs

Thanks Max! 2016-01-19 18:04 GMT+01:00 Maximilian Michels : > I've filed an issue at infra to protect the master: > https://issues.apache.org/jira/browse/INFRA-11088 > > On Fri, Jan 15, 2016 at 3:40 PM, Maximilian Michels > wrote: > > +1 for a protected master.

Re: [DISCUSS] Git force pushing and deletion of branchs

Thanks Max :-) On Tue, Jan 19, 2016 at 6:05 PM, Fabian Hueske wrote: > Thanks Max! > > 2016-01-19 18:04 GMT+01:00 Maximilian Michels : > > > I've filed an issue at infra to protect the master: > > https://issues.apache.org/jira/browse/INFRA-11088 > > > > On