custom scheduler in Flink?

Hi everyone, I want to develop a custom task scheduler in Flink, specifically, I want to control the scheduling of components of an topology into specific hosts, where should I start with? Is there an example program or tutorial about this? Thanks! Thanks. Best regards. Shuhao Zhang (Tony)

Re: Tumbling time window cannot group events properly

The output is the timestamps of events in string. (For convenience, the payload of each event is exactly the timestamp of it.) As soon as the folding of a time window is finished, the code will print "# events in this window" indicating the end of the window. The 10s windows should be [19:10:40,

Re: Using standalone single node without HA in production, crazy?

Thank you Jamie and Ufuk both for such helpful answers! I will continue to explore my options and eagerly await out of the box Mesos support. Ryan On Mon, Jul 4, 2016 at 5:05 AM Ufuk Celebi wrote: > On Fri, Jul 1, 2016 at 3:41 PM, Ryan Crumley wrote: > >

Re: Tumbling time window cannot group events properly

Could you please elaborate a bit on what exactly the output means and how you derive that events are leaking into the previous window? On Mon, 4 Jul 2016 at 13:20 Yukun Guo wrote: > Thanks for the information. Strange enough, after I set the time > characteristic to

Re: Data point goes missing within iteration

I have sent you my code in a separate email, I hope you can solve my issue. Thanks a lot Biplob -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Data-point-goes-missing-within-iteration-tp7776p7798.html Sent from the Apache Flink User

Re: Tumbling time window cannot group events properly

Thanks for the information. Strange enough, after I set the time characteristic to EventTime, the events are leaking into the previous window: ... Mon, 04 Jul 2016 19:10:49 CST Mon, 04 Jul 2016 19:10:50 CST # ? Mon, 04 Jul 2016 19:10:50 CST Mon, 04 Jul 2016 19:10:50 CST Mon, 04 Jul 2016 19:10:50

Re: Data point goes missing within iteration

Can you share the complete program with me? Than I would look into it. Could be that you count in a wrong way. The iteration should definitely consume all the initial input once at least. On Mon, Jul 4, 2016 at 12:07 PM, Biplob Biswas wrote: > Can anyone check this

Re: Data point goes missing within iteration

Can anyone check this once, and help me out with this? I would be really obliged. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Data-point-goes-missing-within-iteration-tp7776p7795.html Sent from the Apache Flink User Mailing List

Re: Different results on local and on cluster

Sorry I wanted to write Kryo but I'm on my mobile On 4 Jul 2016 12:34 p.m., "Flavio Pompermaier" wrote: > Because I don't see any good reason for that...maybe also all keyo > serialization errors that I have from time to time could be symptomatic of > some other

Re: Different results on local and on cluster

Because I don't see any good reason for that...maybe also all keyo serialization errors that I have from time to time could be symptomatic of some other error in how Flink manage the ibternal buffers...but also this is just another personal guess I did.. On 4 Jul 2016 12:29 p.m., "Ufuk Celebi"

Re: Flink programm with for loop yields wrong results when run in parallel

I also have a lot of use case where caching a dataset would be definitely useful...maybe using Auxillio (the new name of Tachyon) and write intermediate results to an in-memory fs could be an better than re-read over and over tge input source for the moment...What do you think? On 4 Jul 2016 12:25

Re: Different results on local and on cluster

It's not possible to tell. You would have to look into the logs of the job manager to check what happened. The not killed task manager could have re-connected to the job manager, if it was restarted quickly after the failure. Why do you think that the task manager would influence the job result

Re: Flink programm with for loop yields wrong results when run in parallel

Nested iterations are not supported via a "native iteration" operator. There is no way to avoid the for loop at the moment. I think it's not possible to tell why the results are wrong from the code snippet. How do you propagate the counts back? In general I expect this program to perform very

Re: Documentation for translation of Job graph to Execution graph

On Wed, Jun 29, 2016 at 9:19 PM, Bajaj, Abhinav wrote: > Is their a plan to add the Job id or name to the logs ? This is now part of the YARN client output and should be part of the 1.1 release. Regarding your other question: in standalone mode, you have to manually

Re: How to avoid breaking states when upgrading Flink job?

Judging from the stack trace the state should be part of the operator state and not the partitioned RocksDB state. If you have implemented the Checkpointed interface anywhere, that would be a good place to pinpoint the anonymous class. Is it possible to share the job code? – Ufuk On Fri, Jul 1,

Re: Different results on local and on cluster

I guess Aljoscha was referring to whether you also have broadcasted input or something like it? On Fri, Jul 1, 2016 at 7:05 PM, Flavio Pompermaier wrote: > what do you mean exactly? > > On 1 Jul 2016 18:58, "Aljoscha Krettek" wrote: >> >> Hi, >> do you

Re: Using standalone single node without HA in production, crazy?

On Fri, Jul 1, 2016 at 3:41 PM, Ryan Crumley wrote: > Questions: > 1. Is this a viable approach? Any pitfalls to be aware of? The major pitfall would be future migrations as outlined by Jamie. > 2. What is the correct term for this deployment mode? Single node > standalone?

Re: Failed job restart - flink on yarn

If you just re-submit the job without a savepoint, the Kafka consumer will by default start processing from the latest offset and the operators will be in an empty state. It should be possible to add a feature to Flink, which allows turning the latest checkpoint to a savepoint, from which you then

Flink programm with for loop yields wrong results when run in parallel

Hi, I have a Flink programm, which outputs wrong results once I set the parallelism to a value larger that 1. If I run the programm with parallelism 1, everything works fine. The algorithm works on one input dataset, which will iteratively be split until the desired output split size is

Re: Error submitting stand-alone Flink job to EMR YARN cluster

You can also have a look at the YARN client logs, which should print which JARs are uploaded. The container logs should also log the class path. On Sun, Jul 3, 2016 at 6:04 PM, Jamie Grier wrote: > Hi Bruce, > > I just spun up an EMR cluster and tried this out. Hadoop

Re: Tumbling time window cannot group events properly

Hi, I think it should be as simple as setting event time as the stream time characteristic: env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) The problem is that .timeWindow(Time.seconds(10)) will use processing time if you don't specify a time characteristic. You can enforce using