Re: Slow Flink program

Hi, If you didn't configure your program to use RocksDB then you're already not using RocksDB. I think the main issue, as others have pointed out, is that by keying on a constant key you're essentially turning your program into a parallelism-of-1 program, thereby wasting almost all cluster

Re: Task Slots allocation

Hi, Could you maybe send a screenshot from the dashboard that shows how tasks are being assigned to slots? Aljoscha > On 4. Mar 2018, at 18:06, vijayakumar palaniappan > wrote: > > I am using flink 1.4.0 in standalone cluster mode > > I have a job with a graph like

Re: Which test cluster to use for checkpointing tests?

Fyi, this is the Jira issue for tracking the issue: https://issues.apache.org/jira/browse/FLINK-2491 Aljoscha > On 6. Mar 2018, at 02:32, Nico Kruber wrote: > > There are still some upcoming changes for the network

Re: Restart hook and checkpoint

Hi, Have you looked into fine-grained recovery? https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures Stefan cc'ed might be able to give you

Re: Record Delivery Guarantee with Kafka 1.0.0

Hi, How are you checking that records are missing? Flink should flush to Kafka and wait for all records to be flushed when performing a checkpoint. Best, Aljoscha > On 13. Mar 2018, at 21:31, Chirag Dewan wrote: > > Hi, > > Still stuck around this. > > My

Re: state backend configuration

Hi, This is not possible on versions of Flink prior to 1.5 but starting with that (as of yet unreleased) versions you'll be able to configure this via state.backend.incremental: true. Configurations that you do in code will take precedence. Best, Aljoscha > On 14. Mar 2018, at 02:33, Jayant

design flink cep pattern for transaction life cycle

0down votefavorite I have a scenario where I need to detect following pattern for a transaction. My basic transaction event look like this => *Transaction(id,status, time)* Transaction goes through

Re: flink sql: "slow" performance on Over Window aggregation with time attribute set to event time

Hi, It is typically not a good idea to generate watermarks based on system (machine) time. Watermarks should be data driven. As soon as the clock of one of your machines is 1 second behind the other machines, its watermarks will also be 1 second behind and hence the complete stream. Best, Fabian

Re: Too many open files on Bucketing sink

I have seen this before as well. My workaround was to limit the number of parallelism but it is the unfortunate effect of limiting the number of processing tasks also (and so slowing things down) Another alternative is to have bigger buckets (and smaller number of buckets) Not sure if there

Re: Question regarding effect of 'restart-strategy: none' on Flink (1.4.1) JobManager HA

Hi Timo, My observation was based on standalone cluster. Regards, Shaswata Get Outlook for Android From: Timo Walther Sent: Wednesday, March 14, 22:11 Subject: Re: Question regarding effect of 'restart-strategy: none' on Flink (1.4.1) JobManager HA To:

Re: flink sql: "slow" performance on Over Window aggregation with time attribute set to event time

Hi Fabian, Thank you for answering the question. However, I think it doesn't explain my situation. the source tasks' watermark are set to 10 ms behind the system time. Assigners allowing a fixed amount of

Re: Question regarding effect of 'restart-strategy: none' on Flink (1.4.1) JobManager HA

Hi Shaswata, are you using a standalone Flink cluster or how does your deployement look like? E.g. YARN has its own restart attempts [1]. Regards, Timo [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/jobmanager_high_availability.html#yarn-cluster-high-availability Am

Re: Flink SSL Setup on a standalone cluster

Hi Vinay, do you have any exception or log entry that describes the failure? Regards, Timo Am 14.03.18 um 15:51 schrieb Vinay Patil: Hi, I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing

Re: activemq connector not working..

Hi Puneet, are you running this job on the cluster or locally in your IDE? Regards, Timo Am 14.03.18 um 13:49 schrieb Puneet Kinra: Hi I used apache bahir connector  below is the code.the job is getting finished and not generated the output as well ,ideal it should keep on running below

Flink SSL Setup on a standalone cluster

Hi, I have keystore for each of the 4 nodes in cluster and respective trustore. The cluster is configured correctly with SSL , verified this by accessing job manager using https and also see the TM path as akka.ssl.tcp, however the job is not getting submitted to the cluster. I am not allowed to

Re: PartitionNotFoundException when restarting from checkpoint

Unfortunately the stack trace was swallowed by the java timer in the LocalInputChannel[1], the real error is forwarded out to the main thread but I couldn’t figure out how to see that in my logs. However, I believe I am close to having a reproducible example. Run a 1.4 DataStream, sinking to

[Proposal] CEP library changes - review request

Hi, We've been facing issues* w.r.t watermarks not supported per key, which led us to: Either (a) run the job in Processing time for a KeyedStream -> compromising on use cases which revolve around catching time-based patterns or (b) run the job in Event time for multiple data streams (one data

activemq connector not working..

Hi I used apache bahir connector below is the code.the job is getting finished and not generated the output as well ,ideal it should keep on running below the code. import org.apache.activemq.ActiveMQConnectionFactory; import org.apache.flink.streaming.api.datastream.DataStream; import

Re: flink sql: "slow" performance on Over Window aggregation with time attribute set to event time

Hi, Flink advances watermarks based on all parallel source tasks. If one of the source tasks lags behind the others, the event time progresses as determined by the "slowest" source task. Hence, records ingested from a faster task might have a higher processing latency. Best, Fabian 2018-03-14

Re: sorting data into sink

Hi, To be honest, I did not understand your requirements and what you are looking for. stream.keyBy("partition").addSink(...) will partition the output on the "partition" attribute before handing it to the sink. Hence, all records with the same "partition" value will be handled by the same

Re: State serialization problem when we add a new field in the object

Hi Konstantin, What you could do, is that you write and intermediate job that has the old ValueState “oldState” and the new one “newState”, with the new format. When an element comes in this intermediate job, you check the oldState if it is empty for that key or not. If it is null (empty),

Re: State serialization problem when we add a new field in the object

Hi, Flink supports upgrading of serializers [1] [2] since version 1.3. You probably need to upgrade to Flink 1.3 before you can use the feature. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/custom_serialization.html [2]

Re: Can i use Avro serializer as default instead of kryo serializer in Flink for scala API ?

Hi, when I forcefully enable the Avro. That code doesn't work cause there are some dependencies in Flink-CEP library which needs Generic serializer also. So I have a question again? We are using Scala for Flink program we need evolution schema support for our manage state, Cause variable changes

Re: Global Window, Trigger and Watermarks, Parallelism

I see you replied on https://stackoverflow.com/questions/48028061/flink-cep-greedy-matching with a known bug issue on the https://issues.apache.org/jira/browse/FLINK-8914 In my case my pattern looks like Pattern tripPattern =

state backend configuration

Hi, Can I enable incremental checkpoint for rocksdb via flink-conf.yaml file? Also, I can set state.backend via the conf file as well as in my code using env.setStateBackend(). Which of these take precedence? Jayant Ameta

Re: State serialization problem when we add a new field in the object

Hi Aljoscha, Thanks for your reply. Do you have a suggestion how can we workaround it ? We have a production system running with Flink and it is mandatory to add one more field in the state. Maybe some how we can write our own serializer? Thanks, Konstantin -- Sent from:

Re: Extremely large job serialization produced by union operator

Can you share the operator plan (StreamExecutionEnvironment.getExecutionPlan()) for both cases? Thanks, Fabian 2018-03-14 9:08 GMT+01:00 杨力 : > I understand complex SQL queries would be translated into large DAGs. > However, the submission succeeds in my case if I don't

Re: Extremely large job serialization produced by union operator

I understand complex SQL queries would be translated into large DAGs. However, the submission succeeds in my case if I don't use union operator. It might be a potential bug related to it. For example, following code submisses successfully with the default limitations of akka.framesize. val sqls:

Re: migrate AbstractStreamOperator from 1.0 to 1.4

Hi, if some can be interested, restoreState was removed from issue https://issues.apache.org/jira/browse/FLINK-4196 and snapshotOperatorState was replace from snapshotState in 1.2 --Filippo 2018-03-12 15:37 GMT+01:00 Filippo Balicchia : > Hi, > > I'm newbie in Flink and