Re: PartitionNotFoundException when running in yarn-session.

Hey Niels, any update on this? – Ufuk On Mon, Oct 9, 2017 at 10:16 PM, Ufuk Celebi wrote: > Hey Niels, > > thanks for the detailed report. I don't think that it is related to > the Hadoop or Scala version. I think the following happens: > > - Occasionally, one of your tasks seems to be extreme

Windows getting created only on first execution

Hi , I have written a program which reads data from Kafka, parses the json and does some reduce operation. The problem I am facing is, the program executes perfectly for the first time on a day. But when I kill the program and execute it again, an empty file is created. Even after compiling again

Re: Windowing isn't applied per key

Hi Marcus, I think that is an expected result for sliding window in Flink. You can see the example in the document for more details. [1] For your need, I will suggest to use ProcessFunction to implement the sliding window that you expected. You can use key state to buffer elements and onTimer to t

RichMapFunction parameters in the Streaming API

I was looking for withParameters(config) in the Streaming API today. I stumbled across the following thread. http://apache-flink-mailing-list-archive.1008284.n3. nabble.com/withParameters-for-Streaming-API-td9332.html#a9333 It appears that some of the StreamingAPI developers are in favor of remov

Subscribe

Re: Windowing isn't applied per key

Sure, I'm going to use a name as key in this example and just a number as the value aggregated. This is the sample input data 12:00 {"name": "Marcus", "value": 1} 12:01 {"name": "Suzy", "value": 2} 12:03 {"name": "Alicia", "value": 3} 12:04 {"name": "Ben", "value": 1} 12:06 {"name": "Alicia", "val

Re: Windowing isn't applied per key

Hi, Could you maybe give an example of what you expect as output and what you actually get? Best, Aljoscha > On 9. Oct 2017, at 16:09, mclendenin wrote: > > I am using Processing Time, so it is using the default timestamps and > watermarks. I am running it with a parallelism of 3, I can see

Re: How to make my execution graph prettier?

Great, thanks! On Tue, Oct 10, 2017 at 7:52 AM Aljoscha Krettek wrote: > Hi, > > The execution graph looks like this because Flink optimises your graph to > fit all operations within a single Task. This operation is called chaining. > The operation can be applied when there is no shuffle between

Manual checkpoint

Can I trigger a checkpoint based on a specific event? Meaning, if a given event arrives (containing EOF in this case) it would be broadcasted to all downstream operators and trigger a savepoint aftewards. Thanks, Nuno -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble

Re: DataStream joining without window

Hi, Yes, using a TwoInputStreamOperator (or even better, a CoProcessFunction, because TwoInputStreamOperator is a low-level interface that might change in the future) is the recommended way for implementing a stream-stream join, currently. As you already guessed, you need a policy for cleanup

Re: Questions about checkpoints/savepoints

Hi, Flink does not rely on file system operations to list contents, all necessary file paths are stored in the meta data file, as you guessed. This is the reason savepoints also work with file systems that "only" have read-after-write consistency. Best, Aljoscha > On 10. Oct 2017, at 03:01, v

Re: How to make my execution graph prettier?

Hi, The execution graph looks like this because Flink optimises your graph to fit all operations within a single Task. This operation is called chaining. The operation can be applied when there is no shuffle between operations and when the parallelism is the same (roughly speaking). If you wa

Re: Unusual log message - Emitter thread got interrupted

Just FYI: I pushed a change that changes the message and removes the stack trace (and the exception). > On 10. Oct 2017, at 00:58, Ken Krugler wrote: > > Hi Aljoscha, > > Thanks for responding. > >> On Oct 9, 2017, at 7:36 AM, Aljoscha Krettek > > wrote: >> >> Hi,

Best way to setup different log files for distinct jobs

Hello, I'm using logback as my logging framework. I would like to setup Flink so that each job outputs to a different file. Any Ideas on how could I do that? I am running flink in a standalone cluster with version 1.3.2. Regards, Pedro Chaves. - Best Regards, Pedro Chaves -- Sent from: h

Re: async io operator timeouts

Perfect! Thanks a lot Karthik. > On Oct 10, 2017, at 10:41 AM, Karthik Deivasigamani > wrote: > > Thanks Kostas. > Here is the JIRA : https://issues.apache.org/jira/browse/FLINK-7789 > > > ~ > Karthik > > On Mon, Oct 9, 2017 at 7:12 PM, Kos

Re: async io operator timeouts

Thanks Kostas. Here is the JIRA : https://issues.apache.org/jira/browse/FLINK-7789 ~ Karthik On Mon, Oct 9, 2017 at 7:12 PM, Kostas Kloudas wrote: > Hi Karthik, > > Currently there is no way to provide a handler for timed-out requests. > So the behavior is exactly what you described. A request

Re: Consult about flink on mesos cluster

Hi Bo, I think the by saying mesos.constraings.hard.hostattribute: rack:ak03-07,rack:ak16-10, you define two hard constraints which are attribute rack must equal ak03-07 AND rack must equal ak16-10. Since a task offer must come from both racks, it will never complete a task request. So at the mome

Re: Avoid duplicate messages while restarting a job for an application upgrade

Hi, That’s good to hear :) I quickly went through the code and it seems reasonable. I think there might be need to think a little bit more about how this cancel checkpoint should be exposed to the operators and what should be default action - right now by default cancel flag is ignored, I wou