Which window function to use to start a window at anytime

Hi guys, We are trying use Flink to count millions of keyed items of an hour window hourly as `time(SlidingEventTimeWindows.of(1hour, 1hour))`. According to the sliding window doc , all windows are

StandaloneResourceManager failed to associate with JobManager leader

Hi, I am trying to run a cluster of job-manager and task-manager in docker. One of each for now. I got a StandaloneResourceManager error, stating that it can not associate with job-manager. I do not know what was wrong. I am sure that job-manager can be connected. ===

Deleting files in continuous processing

Is there a way to delete a file once it has been processed? streamEnv .readFile(format, args[0], FileProcessingMode.*PROCESS_CONTINUOUSLY*, 2000)

Avoiding duplicates in joined stream

What's the best way to avoid duplicates in joined stream. In below code I get duplicates of "A" because I have multiple of "A" in fileInput3. SingleOutputStreamOperator fileInput3 = streamEnv.fromElements("A", "A") .assignTimestampsAndWatermarks(timestampAndWatermarkAssigner1);

Re: Automate Job submission with container

Flink 1.3 relies on a two-step approach of deploying the cluster on Mesos and then deploying the job. The cron task seems like a good approach; maybe retry until the job is successfully deployed (using the 'detached' option assumedly). One complication is that the slots take some time to come

Re: Time zones problem

Hi Biplob, Yes unix timestamp is what I¹m using now. But the problem is that a time window like '1 day' is defined using different start-end timestamps for users in different time zones Let me try to draw it |1--2-3-4---| 1 and 3 - time frames for European users 2 and 4 -

Re: kerberos yarn - failure in long running streaming application

Thanks for helping fix the issue Eron. //Eron's email on this issue I see two interesting things in the log. One, the TGT has an expiry of 10 hours, according to the Kafka log output: > 2017-08-13 06:14:48,248 INFO org.apache.kafka.common.security.kerberos.Login - TGT valid

Re:How to verify the data to Elasticsearch whether correct or not ?

BTW, ActivityInfo is an PB object build from xxx.proto. And already has it's value setted to itself. At 2017-08-15 21:17:00, "mingleizhang" <18717838...@163.com> wrote: Hi, flink experts! I sinked my data ( PB objects ) to elasticsearch. I dont know whether the sinked data is correct or

How to verify the data to Elasticsearch whether correct or not ?

Hi, flink experts! I sinked my data ( PB objects ) to elasticsearch. I dont know whether the sinked data is correct or incorrect. The codes like following, Could you help me check it please ? Im not familar with ES. Now, I want to install a kibana to view my data. But I dont know the below

Re:Re: IllegalArgumentException when using elasticsearch as a sink

A PB object is an object that is build from Protobuf. It belongs to google's data interchange format. You can get more from https://github.com/google/protobuf At 2017-08-15 13:42:58, "Hai Zhou" wrote: I would like to ask what is “PB object”？ Thanks. Hai Zhou 在

Re: Serialization problem: Using generic that extends a class on POJO.

Hi Ido, thank you for your good example to reproduce the problem. I could find a bug in Flink's type extraction logic and opened an issue for it [0]. The problem seems to be the bounded generics in both the Foo and FooFoo. Foo.someKey has the wrong type information. It is GenericType. As a

Avro Serialization and RocksDB Internal State

Hi, This is somewhat related to my previous query here: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Evolving-serializers-and-impact-on-flink-managed-states-td14777.html I was exploring Avro Serialization and in that regard I enabled the force use of avro using,

Re: Time zones problem

Regarding timezones, you should probably convert your time to the unix timestamp which will be consistent all over the world, and then you can create your window based on this timestamp. -- View this message in context:

Re: Flink - Handling late events - main vs late window firing

I would say so, yes. But I don't consider ProessWindowFunction to be low-level, it's just the function that should be used for processing windows if you need more information about context. Best, Aljoscha > On 14. Aug 2017, at 22:53, M Singh wrote: > > Thanks Aljoscha