Update timeWindow size and trigger value at runtime

Hi, I want to have an opportunity to update timeWindow size and trigger value in KeyedStream dynamically at runtime. For example, I have such a piece of code: DataStream stream = env.addSource(new FlinkKafkaConsumer09<>(TOPIC, new JSONDeserializer(), properties)); Integer numMinutes = .

Re: Exception when using keyby operator

That fixed my issue. Thanks. I also agree we need to fix the Documentation On Thu, Sep 7, 2017 at 6:15 PM, Timo Walther wrote: > Hi Sridhar, > > according to the exception, your "meEvents" stream is not POJO. You can > check that by printing "meEvents.getType()". In general, you can always > che

Cannot deploy Flink on YARN

I am trying to start Flink(Version 1.3.0) on YARN (Hadoop 2.8.1) by issuing the following command: ~/flink-1.3.0/bin/yarn-session.sh -s 4 -n 10 -jm 4096 -tm 4096-d I am seeing a flurry of these Errors: 2017-09-11 08:17:11,410 INFO org.apache.flink.yarn.YarnClusterDescriptor - D

Keyed function type erasure problem.

Just small change from PojoExample: PojoExample.java public class PojoExample { private SelectorContainer selectorContainer; // KeySelector class like this can't be used in flink !!!

Re: Java heap size

UPDATE: I also tried using rocksdb and increasing heap size for job manager and task manager to 3072 mb from 2014. Anyway no good news. 14:07:33,973 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1505131653970 14:07:40,027 INFO org.apache.flink.run

Bulk Iteration

Hello all, using Bulk iteration, is there any way to know the number of iterations? Cheers, Alieh

Jobmanager and Taskmanager

Hi, I'm configuring a cluster composed by just three nodes. Looking at cluster setup guide I'm setting in the jobmanager the addresses of the workers. A jobmanager can be used also as an additional taskmanager? i.

Re: Jobmanager and Taskmanager

Yes, a given host may be used as both a job manager and as a task manager. On Mon, Sep 11, 2017 at 8:27 AM, AndreaKinn wrote: > Hi, > I'm configuring a cluster composed by just three nodes. > Looking at cluster setup guide > release-0.8/cluste

Re: Installing Apache Flink on Mesos Cluster without DC/OS

Hi, You do not need to install Flink onto the nodes. The approach used by Flink is that the task managers download a copy of Flink from the appmaster. The entire installation tree of Flink is downloaded (i.e. the bin/lib/conf directories). The only assumed dependency is Java, which may be prov

Re: Update timeWindow size and trigger value at runtime

You cannot change the size/trigger count while a job is running. For this to work you will have to take a savepoint, modify the parameters and reload from the savepoint. On 11.09.2017 09:27, victor.reut wrote: Hi, I want to have an opportunity to update timeWindow size and trigger value in K

Re: Best way to deriving streams from another one

Have a look at side outputs in the documentation, they allow you to emit to multiple streams (of different types!) with a ProcessFunction. On 10.09.2017 22:15, AndreaKinn wrote: Hi, I have a data stream resulting from an operation executed on a data stream of data. Essentially I want to obtain

Re: Is State access synchronized?

Hello, state is local to each parallel instance of an operator. Coupled with the fact that the "map" method is always called by the same thread (and never concurrently) the ValueState (or any state for that matter) will always return the latest values. On 10.09.2017 14:39, Federico D'Ambrosi

Re: Delay in Flink timers

It is true that onTimer and processElement are never called at the same time. I'm not entirely sure whether there is any prioritization/fairness between these methods (if not if could be that onTimer is starved) , looping in Aljoscha who hopefully knows more about this. On 10.09.2017 09:31,

Re: Handle event time

As mentioned earlier, the watermark is the basis for reasoning about the overall progression of time. Many operators use the watermark to correctly organize records, e.g. into the correct time-based window. Within that window the records may still be unordered. That said, some operators do take

Re: Queryable State

Hi All, Any suggestions would really be helpful. Thanks On Sun, Sep 10, 2017 at 12:04 AM, Navneeth Krishnan < reachnavnee...@gmail.com> wrote: > Hi All, > > I'm running a streaming job on flink 1.3.2 with few queryable states. > There are 3 task managers and a job manager. I'm getting timeout ex

Re: Is State access synchronized?

Hi, Thank you very much, Chesnay, for this clarification. 2017-09-11 19:36 GMT+02:00 Chesnay Schepler : > Hello, > > state is local to each parallel instance of an operator. Coupled with the > fact that the "map" method is always called by the same thread (and never > concurrently) the ValueStat

Re: ETL with changing reference data

Thanks Chen We add our reference data to a JVM global hash map, one map per reference data type (we found flink table joins were too slow as they were doing a table scan) so a side pipeline to update the reference data is a nice idea but may suffer from concurrency issues. If there are pending r

Flink flick cancel vs stop

I was wondering about the status of the flink stop command. At first blush it would seem as the preferable way to shutdown a Flink job, but it depends on StoppableFunction being implemented by sources and I notice that the Kafka source does not seem to implement it. In addition, the command does

heap dump shows StoppableSourceStreamTask retained by java.lang.finalizer

Hi , I was using Chaos Monkey to test Flink's behavior against frequent killing of task manager nodes. I found that stopped/disposed StreamTask got retained by java finalizer. It is kind like a memory leak. Since each StreamTask retains 2.6 MB memory. With 20 kills (and job restarts) for 8-CPU con