Kafka Offset commit failed on partition

Hello, Since the last update to the universal Kafka connector, I'm getting the following error fairly often. /2019-11-18 15:42:52,689 ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-10, groupId=srx-consumer-group] Offset commit failed on partit

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Unfortunately the audit logs for hdfs were not enabled. We will enable them and post he results when the problem happens again. Nonetheless, we don't have ay other process using hadoop besides flink - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

The issue happened again. /AsynchronousException{java.lang.Exception: Could not materialize checkpoint 47400 for operator ENRICH (1/4).} at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) at org.

Re: Received fatal alert: certificate_unknown

We found the issue. It was using the DNSName for the certificate validation and we were accessing via localhost. - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hello Andrey, The audit log doesn't have anything that would point to it being deleted. The only thing worth mentioning is the following line. /2019-05-15 10:01:39,082 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK* blk_1248714854_174974084 is COMMITTED but not COMPLETE(numNodes=

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hello, Thanks for the help. I've attached the logs. Our cluster has 2 job managers (HA) and 4 task managers. logs.tgz Regards, Pedro - Best Regards, Pedro Chaves -- Sent from: http://apache-flink

Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hello, Every once in a while our checkpoints fail with the following exception: /AsynchronousException{java.lang.Exception: Could not materialize checkpoint 65912 for operator AGGREGATION-FILTER (2/2).} at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler

Received fatal alert: certificate_unknown

Every time that I access Flink's WEB UI I get the following exception: /2019-05-14 12:31:47,837 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint- Unhandled exception org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fat

Re: CodeCache is full - Issues with job deployments

Hello Stefan, Thank you for the reply. I've taken a heap dump from a development cluster using jmap and analysed it. To do the tests we restarted the cluster and then left a job running for a few minutes. After that, we restarted the job a couple of times and stopped it. After leaving the cluster

CodeCache is full - Issues with job deployments

Hello, Every time I deploy a flink job the code cache increases, which is expected. However, when I stop and start the job or it restarts the code cache continuous to increase. Screenshot_2018-12-11_at_11.png

Re: Get savepoint status fails - Flink 1.6.2

Hello, I've tried with different (jobId, triggerId) pairs but it doesn't work. Regards, Pedro Chaves. - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Get savepoint status fails - Flink 1.6.2

Hello, I am trying to get the status for a savepoint using the rest api but the GET request is failing with an error as depicted bellow. /curl -k https://localhost:8081/jobs/c78511cf0dc10c1e9f7db17566522d5b/savepoints/51c174eab1efd2c1354282f52f37fadb {"errors":["Operation not found under key: org

Re: ***UNCHECKED*** Error while confirming Checkpoint

Hello, Find attached the jobmanager.log. I've omitted the log lines from other runs, only left the job manager info and the run with the error. jobmanager.log Thanks again for your help. Regards

Re: Could not retrieve the redirect address - No REST endpoint has been started

Hello, Thank you for the reply. The problem sometimes happens when there is a jobmanager failover. I've attached the jobmanager logs for further debugging. flink-flink-jobmanager-1-demchcep00-01.log

Re: ***UNCHECKED*** Error while confirming Checkpoint

Hello Stefan, Thank you for the help. I've actually lost those logs to due several cluster restarts that we did, which cause log rotation up (limit = 5 versions). Those log lines that i've posted were the only ones that showed signs of some problem. *The configuration of the job is as follows:

***UNCHECKED*** Error while confirming Checkpoint

Hello, I have a running Flink job that reads data form one Kafka topic, applies some transformations and writes data back into another Kafka topic. The job sometimes restarts due to the following error: /java.lang.RuntimeException: Error while confirming checkpoint at org.apache.flink.run

Flink JMX Metrics

Hello, I've upgraded my cluster to version 1.5.3 from 1.4.2. After the upgrade I notice that some of the metrics reported via JMX, like the number of running jobs, are missing. I've listed all of the domains and this is what I have: /$>domains #following domains are available JMImplementation co

Re: Could not retrieve the redirect address - No REST endpoint has been started

Hello, It happens whether the WEB UI is opened or not and it no longer works. When this happens I have to restart the job managers. regards, Pedro. - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Could not retrieve the redirect address - No REST endpoint has been started

Hello, I have a running standalone Flink cluster with 2 task managers and 2 job manages (one task manager and job manager per machine). Sometimes, when I restart the cluster I get the following error message: / java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: No RE

Re: Substasks - Uneven allocation

That is only used to split the load across all of the subtasks, which am already doing. It is not related with the allocation. - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Substasks - Uneven allocation

Hello, I have a job that has one async operational node (i.e. implements AsyncFunction). This Operational node will spawn multiple threads that perform heavy tasks (cpu bound). I have a Flink Standalone cluster deployed on two machines of 32 cores and 128 gb of RAM, each machine has one task man

Re: Metric Registry Warnings

I have multiple sources but with distinct names and UIDs. More information about my execution environment: Flink Version: 1.4.2 bundled with hadoop 2.8 State backend: Hadoop 2.8 Job compiled for version 1.4.2 using the Scala version libs from Scala version 2.11. Am using the com.github.davidb t

Error while reporting metrics - ConcorrentModificationException

Hello, I have the following error while trying to report metrics to influxdb using the DropwizardReporter. 2018-03-20 13:51:00,288 WARN org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while reporting metrics java.util.ConcurrentModificationException at java.util.Lin

Re: Metric Registry Warnings

Hello, I still have the same issue with Flink Version 1.4.2. java.lang.IllegalArgumentException: A metric named .taskmanager.6aa8d13575228d38ae4abdfb37fa229e.CDC.Source: EVENTS.1.numRecordsIn already exists at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) at

Re: Getting warning messages (...hdfs.DataStreamer - caught exception) while running Flink with Hadoop as the state backend

No. It is just a log message with no apparent side effects. - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Getting warning messages (...hdfs.DataStreamer - caught exception) while running Flink with Hadoop as the state backend

While my flink job is running I keep getting the following warning message in the log: /2018-02-23 09:08:11,681 WARN org.apache.hadoop.hdfs.DataStreamer - Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lan

Best way to setup different log files for distinct jobs

Hello, I'm using logback as my logging framework. I would like to setup Flink so that each job outputs to a different file. Any Ideas on how could I do that? I am running flink in a standalone cluster with version 1.3.2. Regards, Pedro Chaves. - Best Regards, Pedro Chaves -- Sent from: h

Re: Could not initialize keyed state backend.

Hello, I thought that the checkpoints would be propagated to all the machines in the cluster when using a local filesystem. Thank you, Regards. - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Could not initialize keyed state backend.

Hello, The state of my RichWindowFunction operator fails to initialize upon a failure. I see the following error in the log file: / java.lang.IllegalStateException: Could not initialize keyed state backend. at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(

Re: FlinkKafkaConsumer010 - Memory Issue

Hello, Sorry for the delay. The buffer memory of the Kafka consumer was piling up. Once I updated to the 1.3.2 version the problem no longer occurred. Pedro. - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: FlinkKafkaConsumer010 - Memory Issue

Hello, Thank you for the reply. The problem is not that the task manager uses a lot of memory, the problem is that every time I cancel and re-submit the Job the task manager does not release the previously allocated memory. Regards, Pedro Chaves. - Best Regards, Pedro Chaves -- View this

FlinkKafkaConsumer010 - Memory Issue

Hello, Whenever I submit a job to Flink that retrieves data from Kafka the memory consumption continuously increases. I've changed the max heap memory from 2gb, to 4gb, even to 6gb but the memory consumption keeps reaching the limit. An example of a simple Job that shows this behavior is depicte

Re: Can AsyncFunction be applied to connected streams

Hello, I wanted to keep the data locally, if I associate the fetched metadata with eachevent (in an enrichment phase) it would considerably increase their size since the metadata that I need to process the event in the async I/O is to large. Regards, Pedro. - Best Regards, Pedro Chaves --

Can AsyncFunction be applied to connected streams

Hello, Is there a way to apply the AsyncFunction to connected streams like in a CoFlatMap? I would like to connect streams from different types and process one of them based on the state created by the other in an asynchronous fashion. Regards, Pedro Chaves - Best Regards, Pedro Chaves --

Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers

Can YARN provide task isolation? - Best Regards, Pedro Chaves -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-Run-Distinct-Tasks-in-Different-Task-Managers-tp12104p12201.html Sent from the Apache Flink User Mailing List arc

Re: Remove Accumulators at runtime

Hi, I'm building a system that maintains a set of rules that can be dynamically added/removed. I wanted to count every element that matched each rule in an accumulator ( I have several parallel instances). If the rule is removed so should the accumulator. - Best Regards, Pedro Chaves -- V

Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers

Thanks for the response. I would like to assure that the map operator is not in the same task manager as the window/apply operator, regardless of the number of slots of each task manager. - Best Regards, Pedro Chaves -- View this message in context: http://apache-flink-user-mailing-list-a

Remove Accumulators at runtime

Hello, We can add an accumulator using the following call: getRuntimeContext().addAccumulator(NAME, ACCUMULATOR); Is there a way to remove the added accumulators at runtime? Regards, Pedro Chaves - Best Regards, Pedro Chaves -- View this message in context: http://apache-flink-user-mail

Isolate Tasks - Run Distinct Tasks in Different Task Managers

Hello, Assuming that I have the following Job Graph, (Source) -> (map) -> (KeyBy | Window | apply) -> (Sink) Is there a way to assure that the map operator (and all its subtasks) run on a different task manager than the operator (map | window | apply)? This would allow JVM memory isolati

Re: Problem - Negative currentWatermark if the watermark assignment is made before connecting the streams

Hi Vinay , I'm simply using Netbeans Debugger. Regards, Pedro - Best Regards, Pedro Chaves -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Problem-Negative-currentWatermark-if-the-watermark-assignment-is-made-before-connecting-the-strea

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

The best answer I can give you is the one given in the post. Currently, there is no way of dynamically changing the patterns. The only way would be to dive into Flink's core code and change the way operators are shipped to the cluster. On Thu, Nov 24, 2016 at 3:34 PM, kaelumania [via Apache Flink

Problem - Negative currentWatermark if the watermark assignment is made before connecting the streams

Hello, I have an application which has two different streams of data, one represents a set of events and the other a set of rules that need to be matched against the events. In order to do this I use a coFlatMapOperator. The problem is that if I assign the timestamps and watermarks after the strea

Processing streams of events with unpredictable delays

Hello, I have a stream source of events. Each event is assigned a timestamp by the machine that generated the event and then those events are retreived by other machines (collectors). Finally those collectors will send the events to flink. In flink, when I receive those events I extract their time

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

ave a field for the > window). Then, you can write a custom WindowAssigner that will simply get > the window from an event and assign that as the internal window. > > Please let me know if you need more details. > > Cheers, > Aljoscha > > On Thu, 3 Nov 2016 at 18:40

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

Hello, Your tip was very helpful and I took a similar approach. I have something like this: class Processor extends RichCoFlatMapFunction { public void flatMap1(Event event, Collector out) { process(event,out); // run the javscript (rules) against the incoming events } publ

Re: Best Practices/Advice - Execution of jobs

Thank you. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-Practices-Advice-Execution-of-jobs-tp9822p9873.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: NotSerializableException: jdk.nashorn.api.scripting.NashornScriptEngine

Hello, I'm having the exact same problem. I'm using a filter function on a datastream. My flink version is 1.1.3. What could be the problem? Regards, Pedro Chaves. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/NotSerializableException-

Best Practices/Advice - Execution of jobs

Hello, I'm trying to build a stream event correlation engine with Flink and I have some questions regarding the for the execution of jobs. In my architecture I need to have different sources of data, lets say for instance: /firewallStream= environment.addSource([FirewalLogsSource]); proxyStream

Elasticsearch sink: Java.lang.NoSuchMethodError: org.elasticsearch.common.settings.Settings.settingsBuilder

Hello, I am using Flink to write data to elasticsearch. Flink version : 1.1.3 Elasticsearch version: 2.4.1 But I am getting the following error: 1/0/28/2016 18:58:56 Job execution switched to status FAILING. java.lang.NoSuchMethodError: org.elasticsearch.common.settings.Settings.settingsBui

Re: FlinkKafkaConsumerBase - Received confirmation for unknown checkpoint

Hello, I Am using the version 1.2-SNAPSHOT. I will try with a stable version to see if the problem persists. Regards, Pedro Chaves. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkKafkaConsumerBase-Received-confirmation-for-unknown-c

FlinkKafkaConsumerBase - Received confirmation for unknown checkpoint

Hello, Am getting the following warning upon executing a checkpoint /2016-10-21 16:31:54,229 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 5 @ 1477063914229 2016-10-21 16:31:54,233 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator -

Re: Flink SQL Stream Parser based on calcite

Thank you. Great presentation about the high-level translation process. Regards, Pedro -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-Stream-Parser-based-on-calcite-tp9592p9608.html Sent from the Apache Flink User Mailing List arc

Re: Flink SQL Stream Parser based on calcite

Thank you for the response. I'm not understanding where does something like this, /SELECT * WHERE action='denied' / gets translated to something similar in the Flink Stream API, /filter.(new FilterFunction() { public boolean filter(Event event) {

Flink SQL Stream Parser based on calcite

Hello, I am pretty new to Apache Flink. I am trying to figure out how does Flink parses an Apache Calcite sql query to its own Streaming API in order to maybe extend it, because, as far as I know, many operations are still being developed and not currently supported (like TUMBLE windows). I need

JsonMappingException: No content to map due to end-of-input

Hello, I recently started programming with Apache Flink API. I am trying to get input directly from kafka in a JSON format with the following code: /private void kafkaConsumer(String server, String topic) { Properties properties = new Properties(); properties.setP

Re: Error with table sql query - No match found for function signature TUMBLE(, )

I have this so far: result = eventData .filter(new FilterFunction(){ public boolean filter(Event event){ return event.action.equals("denied");

Re: Error with table sql query - No match found for function signature TUMBLE(, )

Hi, Thanks for the response. What would be the easiest way to do this query using the DataStream API? Thank you. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Error-with-table-sql-query-No-match-found-for-function-signature-TUMBLE-TIME

Error with table sql query - No match found for function signature TUMBLE(, )

Hello, I am trying to build an query using the StreamTableEnvironment API. I Am trying to build this queries with tableEnvironment.sql("QUERY") so that I can in the future load those queries from a file. Code source: Table accesses = tableEnvironment.sql ("SELEC

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

I've been thinking in several options to solve this problem: 1. I can use Flink savepoints in order to save the application state , change the jar file and submit a new job (as the new jar file with the patterns added/changed). The problem in this case is to be able to correctly handle the savepoi

What is the best way to load/add patterns dynamically (at runtime) with Flink?

Hello, I am new to Apache Flink and am trying to build a CEP using Flink's API. One of the requirements is the ability to add/change patterns at runtime for anomaly detection (maintaining the systems availability). Any Ideas of how could I do that? For instance, If I have a stream of security eve