Re: Native kubernetes setup

1) -Dkubernetes.namespace 2) The -D syntax is actually just a way to specify configurations options from the command-line. As such, the configuration page lists all options. 3) if the diff between the conf

Re: unknown process kdevtmpfsi is taking more cpu

on that particular server itself. On Mon, Nov 2, 2020 at 6:48 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: I could not find any traces of *kdevtmpfsi* in our official docker image. The only references I could find on the web for *kdevtmpfsi* are that it is some

Re: Missing help about run-application action in Flink CLI client

Good find, this is an oversight in the CliFrontendParser; no help is printed for the run-application action. Can you file a ticket? On 11/4/2020 10:53 AM, Flavio Pompermaier wrote: Hello everybody, I was looking into currently supported application-modes when submitting a Flink job so I tried

Re: Flink job percentage

|The "mismatch" is due to you mixing job and vertex states. | |These are the states a job can be in (based on org.apache.flink.api.common.JobStatus):| |[ "CREATED", "RUNNING", "FAILING", "FAILED", "CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED", "RECONCILING" ]|| | |T

Re: Flink job percentage

Admittedly, it can be out-of-sync if someone forgets to regenerate the documentation, but they cannot be mixed up. On 11/5/2020 11:31 AM, Chesnay Schepler wrote: |The "mismatch" is due to you mixing job and vertex states. | |These are the states a job can be in

Re: Flink job percentage

Nov 5, 2020 at 11:31 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: |The "mismatch" is due to you mixing job and vertex states. | |These are the states a job can be in (based on org.apache.flink.api.common.JobStatus):| |[ "CREATED&qu

Re: Long blocking call in UserFunction triggers HA leader lost?

I'd go with the network congestion theory for the time being; then the only remedy is throttling the download of said list, or somehow reducing the size of it significantly . What the task thread is doing doesn't matter in regards to HA; it may cause checkpoints to time out, but should have no

Re: Flink TLS in K8s

It is unlikely that this is a port issue, and I would currently suspect that something in your SSL setup is not correct. @Nico: do you have a suggestion on how to debug this? On 11/5/2020 4:23 PM, Patrick Eifler wrote: Hi, I did set up a flink session cluster on K8s. Now I added the ssl conf

Re: Is possible that make two operators always locate in same taskmanager?

It would be good if you could elaborate a bit more on your use-case. Are you using batch or streaming? What kind of "message" are we talking about? Why are you thinking of using a static variable, instead of just treating this message as part of the data(set/stream)? On 11/5/2020 12:55 PM, Si-

Re: union stream vs multiple operators

I don't think the first option has any benefit. On 11/5/2020 1:19 AM, Alexey Trenikhun wrote: Hello, I have two Kafka topics ("A" and "B") which provide similar structure wise data but with different load pattern, for example hundreds records per second  in first topic while 10 records per sec

Re: prometheus variable value is "normalized"

Essentially, we started with this behavior, and kept it to not break existing Flink setups. You can disable the filtering of label values with the filterLabelValueCharacters setting. https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#prometheus-orgapacheflinkm

Re: Join a datastream with tables stored in Hive

According to the documentation this is already implemented. On 12/1/2020 3:53 PM, maverick wrote: Hi Kurt, Is there any Jira task for tracking progress of adding event time support

Re: Running Flink job as a rest

What you are asking for is an HTTP(s) source. This currently does not exist for Flink, so you would have to implement it yourself. Additionally you would have to figure out the host on which the source runs on yourself. It may be easier to setup a separate HTTP(s) server that accepts data, whi

Re: user defined metrics showed in Flink UI but not datadog

It appears that the datadog reporter does not report histograms. I'll file an issue to fix that. On 12/8/2020 4:42 AM, Fanbin Bu wrote: Hi, I followed [1] to define my own metric as: val dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500)) histogram = ge

Re: user defined metrics showed in Flink UI but not datadog

the ticket once you create it. Meanwhile, is there any workaround for now? Fanbin On Tue, Dec 8, 2020 at 2:56 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: It appears that the datadog reporter does not report histograms. I'll file an issue to fix that. On

Re: How to reduce number of metrics pushed to Prometheus Push Gateway

There is no way to reduce the number of metrics. The only thing you can do is exclude specific variables (e.g., task_name), like this: metrics.reporter..scope.variables.exclude:task_name[; On 12/9/2020 7:15 PM, Alexander Filipchik wrote: Hi, Is there a way to reduce cardinality (preaggregate

Re: How to reduce number of metrics pushed to Prometheus Push Gateway

wrote: Thank you for replying! Will exclusion produce proper aggregates? If I drop subtask_index, will resulting metric be a sum of all the subtasks, or it will be just data from one that was reported the last? Alex On Thu, Dec 10, 2020 at 4:28 AM Chesnay Schepler <mailto:ches...@apache.

Re: flink-cdc-connector 使用场景和限制是什么？

Moving to chinese user mailing ist. On 12/14/2020 3:19 AM, 陈帅 wrote: 传统CDC方式是通过 mysql -> debezium -> kafka, 这样便于DBA管控资源，因为像postgres库需要创建slot资源，但如果像 flink-cdc-connector 每（几）张表就创建一个CDC流的话对数据库的资源要求很高，而且不可控。所以我的理解flink-cdc-connector更适合少量的即席cdc处理，而不是大规模的cdc处理，不知我的理解对不对？

Re: Never terminating test ...

My guess would be that the consumer does not stop running once it exhausted the kinesis stream. Which makes sense since this isn't a batch job. (Wouldn't want the source to shut down just because it happened to catch up with your input ;) ) On 12/14/2020 8:09 AM, Avi Levi wrote: I have the f

Re: Flink 1.12

1) It is compiled with Java 8 but runs on Java 8 & 11. 2) Docker images are not yet published. 3) It is mentioned at the top of the Kubernetes HA Services documentation that it also works for the native Kubernetes integration. /Kubernetes high availability services can only be used when d

Re: unsubscribe

To unsubscribe from this mailing list, please send a mail to /user-unsubscr...@flink.apache.org/ . On 12/14/2020 10:25 AM, yuguangyuan wrote: unsubscribe

Re: How to tell when flink is done restoring from a savepoint

I do not believe there is anything in the UI, or the logs for that matter, that give a definite answer to that. I suppose if a new checkpoint was completed then you can be sure the state was restored. FLINK-19013 On 12/14/2020 6:40 AM, Rex Fe

Re: Flink 1.12

lready released!). We are not sure yet whether we can get the PR accepted and defer further changes to 1.12.1 . On 12/15/2020 5:17 PM, Boris Lublinsky wrote: Thanks. Do you have ETA for docker images? On Dec 14, 2020, at 3:43 AM, Chesnay Schepler <mailto:ches...@apache.org>> wrote: 1)

Re: Is working with states supported in pyflink1.12?

It is currently not possible to access state with the Python API. A proposal has recently been made to enhance the API with state access (under FLIP-152), but at this time I cannot provide a prediction for when it might be released. On 12/15/2020 7:55 PM, Nadia Mostafa wrote: Hello, I'm new

Re: Distribute Parallelism/Tasks within RichOutputFormat?

Essentially I see 2 options here: a) split your output format such that each format is it's own sink, and then follow Flavio's suggestion to filter the stream and apply each sink to one of the streams, with the respective parallelism. This would be the recommended approach. b) modify your (cust

Re: Throwing Recoverable Exceptions from Tasks

There is no way to have an exception appear in the REST API without restarting the job; that field is exactly defined as the exception causing the job to fail. Using asynchronous by itself is fine, so long as you don't wait for any confirmation. In any case you could remedy the issue by writin

Re: Chaining 2 flink jobs through a KAFKA topic with checkpoint enabled

I don't particularly know the our Kafka connector, but it sounds like a matter of whether a given consumer does dirty reads. Flink does not, whereas the other tools you're using do. On 12/28/2020 7:57 AM, Daniel Peled wrote: Hello, We have 2 flink jobs that communicate with each other through

Re: Compiling Error for Flink 1.11.3

Please delete the flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/generated directory, and then recompile the module. On 12/28/2020 8:22 AM, Le Xu wrote: Hello! I was trying to compile flink 1.11.3 from github (branch release-1.11.3) but I'm getting the following e

Re: Flink 1.12.0 docker image MISSING

This is a known issue: FLINK-20632 On 1/5/2021 11:27 AM, Alexandru Vasiu wrote: Hi, Docker image for flink 1.12 is missing from Docker Hub. Thank you, Alex Vasiu ComplyAdvantage is a trading name of IVXS TECHNOLOGY ROMANIA. This message, i

Re: Submitting a job in non-blocking mode using curl and the REST API

All jobs going through the web-submission are run in detached mode for technical reasons (blocking of threads, and information having to be transported back to the JobManager for things like collect()). You unfortunately cannot run non-detached/attached/blocking jobs via the web submission, wh

Re: Submitting a job in non-blocking mode using curl and the REST API

preciated ----- Original message - From: Chesnay Schepler To: Adam Roberts , user@flink.apache.org Cc: Subject: [EXTERNAL] Re: Submitting a job in non-blocking mode using curl and the REST API Date: Tue, Jan 5, 2021 4:07 PM All jobs going through the web-submissio

Re: Normalizing Metric Identifies for Reporting Systems

One thing you could do is take the first N characters and hash the remaining ones; I don't think there is a better solution at the moment. The size of job/task/operator names is a rather fundamental issue that makes a lot of things complicated (metrics, logging, UI), but we haven't made any prog

Re: mark kafka stream as idle if no data comes in for a while in flink 1.10

The idleTimeout you found is from an earlier attempt at implementing idleness, but making it configurable was aborted midway through as there were some API issues. The effort was subsumed by a new source interface and watermark generators that were introduced in 1.12. Some more details can be

Re: mark kafka stream as idle if no data comes in for a while in flink 1.10

: thank you @chesnay I tried in vain to find the issue about introduction of new watermark strategy, can you provide some details about it ? Chesnay Schepler mailto:ches...@apache.org>> 于2021年1月11日周一 下午9:43写道： The idleTimeout you found is from an earlier attempt at implemen

Re: Testing Flink Jobs

1) You can either execute the job in a separate thread, or set DeploymentOptions.ATTACHED to false in the MiniCluster configuration. 2) The cluster not being ready is /usually/ not really an issue. I wouldn't worry about it for the time being. (The reason being that the MiniCluster resource alr

Re: Log length

Have a look at RollingFileAppenders . These have become the default in 1.12 . On 1/12/2021 12:53 AM, Rex Fenley wrote: Hello, We've collected over 150 MiB of log lines in 5 days. Is there a way to tell Flink to e

Re: state reset(lost) on TM recovery

Just do double-check, are you aware that ValueState within a Keyed*Function is scoped to the key of the input element(s)? I.e., any stored value is only accessible if an element with the same key is processed? On 1/10/2021 7:18 PM, Alexey Trenikhun wrote: Hello, I'm using Flink 1.11.3, state

Re: Log length

A normal FileAppender that does not do any rolling or limiting of the log file. On 1/12/2021 3:09 AM, Rex Fenley wrote: Thanks, I'll check them out. What's the default in 1.11.2? On Mon, Jan 11, 2021 at 4:26 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote:

Re: state reset(lost) on TM recovery

ser parser = Unchecked.cast(prototype.getParserForType());   final byte[] data =new byte[serializedSize]; dataInputView.read(data);   return parser.parseFrom(CodedInputStream.newInstance(data)); } ---- *From:* Chesnay Schepler *Sen

Re: Metrics for average time taken by flatMap function

Sure, that might work. Be aware though that time measurements are, compared to the logic within a function, usually rather expensive and may impact performance. On 1/12/2021 10:57 AM, Manish G wrote: Hi All, I have implemented a flatmap function and I want to collect metrics for average time

Re: Metrics for average time taken by flatMap function

, 2021 at 3:53 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: Sure, that might work. Be aware though that time measurements are, compared to the logic within a function, usually rather expensive and may impact performance. On 1/1

Re: Metrics for average time taken by flatMap function

at 4:12 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: A gauge just returns a value, and Flink exposes it as is. As such you need to calculate the average over time yourself, taking 2 time measurements (before and after the processing of each). On 1/12/2021

Re: Metrics for average time taken by flatMap function

consumed by each method call and then expose it. Something like this: timeSoFar += timeConsumedByCurrentInvocation this.simpleGaug.setValue( timeSoFar ); On Tue, Jan 12, 2021 at 4:24 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: That approach will generally not work for jobs

Re: Metrics for average time taken by flatMap function

The cumulative time probably isn't that useful to detect changes in the behavior of the application. On 1/12/2021 12:30 PM, Chesnay Schepler wrote: I mean the difference itself, not cumulative. On 1/12/2021 12:08 PM, Manish G wrote: Can you elaborate the second approach more? Currently

Re: Testing Flink Jobs

In that case you will have to query the REST API instead; you can retrieve the address via MiniCluster#getRestAddress. Something along these should work: try (final RestClient restClient = new RestClient( RestClientConfiguration.fromConfiguration(new Configuration()), E

Re: Metrics for average time taken by flatMap function

ldi...@gmail.com>> wrote: Prometheus provides avg_over_time for a range vector. That seems to be better suited for this usecase. On Tue, Jan 12, 2021 at 6:53 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: The cumulative time probably isn't t

Re: Histogram has count data but not sum

What exactly do you mean with "count metrics" and "sum"? Given a Histogram H, you should see: - one time series named H_count - one time series named H, with various labels for the different quantiles. What do you see in Prometheus, and what do you consider to be missing? On 1/13/2021 4:10 PM

Re: Histogram has count data but not sum

_sum data. Though it is prometheus histogram, I expected flink histogram too would provide me same. On Wed, Jan 13, 2021 at 8:50 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: What exactly do you mean with "count metrics" and "sum"? Given a Hist

Re: state reset(lost) on TM recovery

*From:* Chesnay Schepler *Sent:* Tuesday, January 12, 2021 2:20 AM *To:* Alexey Trenikhun ; Flink User Mail List *Subject:* Re: state reset(lost) on TM recovery Are the hashes of these object equal as well? On 1/12/2021 3:59 AM

Re: Flink Docker job fails to launch

Where are you starting the task executor? On 1/15/2021 11:57 AM, Manas Kale wrote: Hi all, I've got a job that I am trying to run using docker as per [1]. Here's the dockerfile: # Start from base Flink image. FROM flink:1.11.0 # Add fat JAR and logger properties file to image. ADD ./target/fli

Re: Flink Docker job fails to launch

ext step (starting taskmanager) would be of no use. Please correct me if my understanding is wrong. On Fri, Jan 15, 2021 at 4:37 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Where are you starting the task executor? On 1/15/2021 11:57 AM, Manas Kale wrote:

Re: flink-python_2.12-1.12.0.jar

The flink-python jar is only required for running python jobs. If you don't use such jobs you can safely remove it. On 1/18/2021 11:42 PM, Alexey Trenikhun wrote: Hello, Is flink-python_2.12-1.12.0.jar in docker image needed to run Java based stream processor? Will Flink work if we will remove

Re: Counter metrics for prometheus having unexepcted gaps in grafana

Not sure whether it would solve your issue, but you could maybe exclude the pod id (I assume you mean the host?) from being reported by setting this: metrics.reporter..scope.variables.excludes: On 1/20/2021 7:16 AM, Manish G wrote: Hi All, I am facing an issue with counter metrics I have a

Re: Job execution graph state - INITIALIZING

INITIALIZING is the very first state a job is in. It is the state of a job that has been accepted by the JobManager, but the processing of said job has not started yet. In other words, INITIALIZING = submitted job, CREATED = data-structures and components required for scheduling have been create

Re: Seeing Rocks Native Metrics in Data Dog

Anything metric that is shown in the Flink UI should also appear in DataDog. If this is not the case then something goes wrong within the reporter. Is there anything suspicious in the Flink logs? Can you give some example of metrics that /do/ show up in DataDog? On 1/26/2021 6:32 PM, Rex Fenley

Re: Seeing Rocks Native Metrics in Data Dog

ut of date. Thanks On Tue, Jan 26, 2021 at 10:28 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Anything metric that is shown in the Flink UI should also appear in DataDog. If this is not the case then something goes wrong within the reporter. Is there

Re: Seeing Rocks Native Metrics in Data Dog

The system scopes must be remapped for your Flink metrics to be supported, otherwise they are submitted as custom metrics." Could we instead add host and ids as tags to our metrics? Thanks for your help! On Tue, Jan 26, 2021 at 2:49 PM Chesnay Schepler <mailto:ches...@apache.org>&

Re: Datadog reporter timeout & OOM issue

Yes, I could see how the memory issue can occur. However, it should be limited to buffering 64 requests; this is the default limit that okhttp imposes on concurrent calls. Maybe lowering this value already does the trick. On 1/27/2021 5:52 AM, Xingcan Cui wrote: Hi all, Recently, I tried to

Re: Datadog reporter timeout & OOM issue

(setting this field is currently not possible from a Flink user perspective; it is something I will investigate) On 1/27/2021 10:30 AM, Chesnay Schepler wrote: Yes, I could see how the memory issue can occur. However, it should be limited to buffering 64 requests; this is the default limit

Re: JobManager seems to be leaking temporary jar files

The problem of submitted jar files not being closed is a known one: https://issues.apache.org/jira/browse/FLINK-9844 IIRC it's not exactly trivial to fix since class-loading is involved. It's not strictly related to the REST API; it also occurs in the CLI but is less noticeable since jars are us

Re: Timers not firing until stream end

Based on your description you aren't doing anything obviously wrong. Would it be possible for you to share the code with us? On 1/27/2021 1:02 PM, Pilgrim Beart wrote: A newbie question: I've created a basic Flink DataStream job for an IoT use-case, with file source and sink for testing. I ke

Re: rocksdb block cache usage

I don't quite understand the question; all 3 metrics you listed are the same one? On 1/27/2021 9:23 AM, ?? wrote: hi, all ?0?2 ?0?2I've enable state.backend.rocksdb.metrics.block-pinned-usage metric , ?0?2and the flink_taskmanager_job_task_operator_window_contents_rocksdb_block_cache_pinne

Re: Flink Job Manager & Task Manager heap size

Generally I see 2 options: a) There's a memory leak somewhere. It would be good to know how the baseline heap usage during idleness evolves over time. Are the same 20 jobs running continuously or are they (or others) periodically re-submitted? b) The JVM just doesn't feel like running garbage

Re: Timers not firing until stream end

WatermarkGenerators and use event-time, then it appears that you *must* set this interval. This behavior is...less than ideal I must admit, and it does not appear to be properly documented. On 1/27/2021 1:56 PM, Chesnay Schepler wrote: Based on your description you aren't doing anything obviously

Re: Timers not firing until stream end

d":"2","value":0.24,"is_online":true,"log":"prevMsg.ts 8000 msg_in.ts 9000 Cancelling previous timer. "} {"ts":1,"id":"0","value":0.25,"is_online":true,"log":"prevMsg.ts 9000

Re: Timers not firing until stream end

Note that while this does fix the issue of timers not firing while the job is running, it seems to be firing too many timers... On 1/27/2021 6:49 PM, Chesnay Schepler wrote: My bad, I was still using the custom WatermarkStrategy that emits a watermark for each event

Re: Timers not firing until stream end

Actually, if the parallelism is 1 then it works as it should. sigh On 1/27/2021 6:52 PM, Chesnay Schepler wrote: Note that while this does fix the issue of timers not firing while the job is running, it seems to be firing too many timers... On 1/27/2021 6:49 PM, Chesnay Schepler wrote: My

Re: Setting app Flink logger

@Eyal: The image you are using is for 1.9.2, but the logging configuration you fetched was from master. In 1.9.2 we use Log4j1, but on master we switched to Log4j2 instead, which uses a different configuration syntax. Log4j1 pretty much ignores the entire file, causing the error. Please us

Re: datadog metrics

Please open a JIRA; we may have to split the datatog report into several chunks. On 09/03/2020 07:47, Fanbin Bu wrote: quote from the following link: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SQL-Query-named-operator-exceeds-80-characters-td24807.html#a24818 "This is

Re: datadog http reporter metrics

Do you see anything in the logs? In another thread a user reported that the datadog reporter could stop working when faced with a large number of metrics since datadog was rejecting the report due to being too large. On 15/03/2020 12:22, Yitzchak Lieberman wrote: Anyone? On Wed, Mar 11, 2020

Re: datadog metrics

t the metrics in chunks and it worked fine after. Would be interested in seeing a ticket on this as well. - Steve On Wed, Mar 11, 2020 at 5:13 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Please open a JIRA; we may have to split the datatog report into several chunks.

Re: [EXT.MSG] Re: datadog http reporter metrics

rejected metrics, nothing... tor that case there should be an error, right? (when report is too large) I saw that there are some changes on version 1.10 for datadog reporter, maybe I should upgrade to this version? On Mon, Mar 16, 2020 at 11:47 AM Chesnay Schepler <mailto:ches...@apache.

Re: Hadoop user jar for flink 1.9 plus

You can download flink-shaded-hadoop from the downloads page: https://flink.apache.org/downloads.html#additional-components On 17/03/2020 15:56, Vishal Santoshi wrote: We have been on flink 1.8.x on production and were planning to go to flink 1.9 or above. We have always used hadoop uber jar fr

Re: Flink1.10 Cluster's Received is zero in the web when consume from Kafka0.11

This is a known limitation, see https://issues.apache.org/jira/browse/FLINK-7286 . As a crude workaround you may either break the chain after the source / before the sink, or query the numRecordsOut metric for the source / numRecordsIn metric for the sink via the WebUI metrics tab or REST API.

Re: flink-shaded-hadoop2 for flink 1.10

flink-shaded-hadoop2 was released as part of Flink until 1.8 (hence why it followed the Flink version scheme), after which it was renamed to flink-shaded-hadoop-2 and is now being released separately from Flink as part of flink-shaded (a project that bundles various dependencies to be used by F

Re: [1.10.0] flink-dist source jar is empty

That should not be a problem since the flink-dist module does not contain any java sources On 14/04/2020 06:42, Steven Wu wrote: We build and publish flink-dist locally. But the source jar turns out empty. Other source jars (like flink-core) are good. Anyone else experienced similar problem?

Re: Javadocs Broken?

I'm looking into it. On 10/04/2020 11:27, tison wrote: Hi guys, Right now when I click "JavaDocs" in out docsite[1] it jumps to a page[2] I think is definitely not out api documentation. Any thoughts? Best, tison. [1] https://ci.apache.org/projects/flink/flink-docs-master/ [2] https://ci.ap

Re: Can I use Apache-Flink for Android API-Level < 26?

I agree with your conclusion that you cannot use Flink on an API Level below 26. I do not know whether it will work even with Level 26 though, as I'm not aware of anyone having tried it. On 14/04/2020 11:03, Alexander Borgschulze wrote: I am trying to use Apache-Flink in my Android-Project wi

Re: Upgrading Flink

@Robert Why would he have to recompile the jobs? Shouldn't he be fine soo long as he isn't using any API for which we broke binary-compatibility? On 09/04/2020 09:55, Robert Metzger wrote: Hey Stephen, 1. You should be able to migrate from 1.8 to 1.10: https://ci.apache.org/projects/flink/fli

Re: [1.10.0] flink-dist source jar is empty

contains the source files for the flink modules that it bundles. On Tue, Apr 14, 2020 at 1:34 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: That should not be a problem since the flink-dist module does not contain any java sources On 14/04/2020 06:42, Steven

Re: Upgrading Flink

x ), it is better to update your project dependencies to the new version (1.10.x), and hence you need to recompile the jobs. On Tue, Apr 14, 2020 at 3:29 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: @Robert Why would he have to recompile the jobs? Shouldn&#x

Re: Jars replication

Jars are not replicated to all JobManagers, this is currently expected, but generally undesirable for the use-case you out-lined. IIRC the important part is that the upload goes directly against the leader, the run request can be sent anywhere and it will be redirected internally to the leader

Re: Jars replication

f I try to send a run request to the one that doesn't have the jar, it responds with a "jar not found" error. I didn't find documentation about this, so I don't know if it is the expected behavior or something wrong in my cluster configuration. Regards On Fri, A

Re: Suppressing illegal Access Warnings

I do not know where this function comes from (DisableAccessWarning().disableAccessWarnings()),**so we can't be sure. ** On 21/04/2020 00:27, Zahid Rahman wrote: Hi, *I was getting these warnings, I think these are due to certain version of Maven libraries which is impacting Java frameworks ev

Re: Unable to unmarshall response (com.ctc.wstx.stax.WstxInputFactory cannot be cast to javax.xml.stream.XMLInputFactory)

Which Flink version are you using? On 21/04/2020 11:11, Fu, Kai wrote: Hi, I’m running Flink application on AWS Kinesis Flink platform to read a kinesis stream from another account with assumed role, while I’m getting exception like below. But it works when I’m running the application local

Re: Suppressing illegal Access Warnings

Volatile.invoke(unsafe, loggerClass, offset, null); }catch (Exception ignored) { //ignore } } } Backbutton.co.uk <http://Backbutton.co.uk> ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org <http://makeuse.org> <http://www.backbutton.co.uk> On Tue, 21 Apr 2020

Re: History Server Not Showing Any Jobs - File Not Found?

Which Flink version are you using? Have you checked the history server logs after enabling debug logging? On 21/04/2020 17:16, Hailu, Andreas [Engineering] wrote: Hi, I’m trying to set up the History Server, but none of my applications are showing up in the Web UI. Looking at the console, I s

Re: RocksDB default logging configuration

AFAIK this is not possible; the client doesn't know anything about the cluster configuration. FLINK-15747 proposes to add an additional config option for controlling the logging behavior. The only workaround I can think of would be to create a custom Flink distribution with a modified RocksD

Re: History Server Not Showing Any Jobs - File Not Found?

wrote: I’m having a further look at the code in HistoryServerStaticFileServerHandler - is there an assumption about where overview.json is supposed to be located? *// *ah** *From:*Hailu, Andreas [Engineering] *Sent:* Wednesday, April 22, 2020 1:32 PM *To:* 'Chesnay Schepler' ; Hail

Re: History Server Not Showing Any Jobs - File Not Found?

/ historyserver.web.tmpdir: /local/scratch/flink_historyserver_tmpdir/ Did you have anything else in mind when you said pointing somewhere funny? *// *ah** *From:*Chesnay Schepler *Sent:* Monday, April 27, 2020 5:56 AM *To:* Hailu, Andreas [Engineering] ; user@flink.apache.org *Subject:* Re: History Server Not

Re: History Server Not Showing Any Jobs - File Not Found?

nk_hs/0004be6ce01ba9677d1eb619ad0fa757 ... *// *ah** *From:*Chesnay Schepler *Sent:* Monday, April 27, 2020 10:28 AM *To:* Hailu, Andreas [Engineering] ; user@flink.apache.org *Subject:* Re: History Server Not Showing Any Jobs - File Not Found? If historyserver.web.tmpdir is not set then java.io.tmpdir is use

Re: History Server Not Showing Any Jobs - File Not Found?

weeks. Could you clarify on how you want to try local filesystem archives? As in changing jobmanager.archive.fs.dir and historyserver.web.tmpdir to the same local directory? *// *ah** *From:*Chesnay Schepler *Sent:* Wednesday, April 29, 2020 8:26 AM *To:* Hailu, Andreas [Engineering] ;

Re: Flink REST API side effect?

This is expected, the backing data structure is cached for a while so we never hammer the JobManager with requests. IIRC this is controlled via "web.refresh-interval", with the default being 3 seconds. On 11/05/2020 14:10, Tomasz Dudziak wrote: Hi, I found an interesting behaviour of the R

Re: Flink REST API side effect?

:*Chesnay Schepler *Sent:* 11 May 2020 13:20 *To:* Tomasz Dudziak ; user@flink.apache.org *Subject:* Re: Flink REST API side effect? This is expected, the backing data structure is cached for a while so we never hammer the JobManager with requests. IIRC this is controlled via "web.refresh-int

[CVE-2020-1960] Apache Flink JMX information disclosure vulnerability

CVE-2020-1960: Apache Flink JMX information disclosure vulnerability Severity: Medium (CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:L/A:H) Vendor: The Apache Software Foundation Versions Affected: Flink 1.1.0 to 1.1.5 Flink 1.2.0 to 1.2.1 Flink 1.3.0 to 1.3.3 Flink 1.4.0 to 1.4.2 Flink 1.5.0 to 1.5.6

Re: Flink performance tuning on operators

Generally there should be no difference. Can you check whether the maps are running as a chain (as a single task)? If they are running in a chain, then I would suspect that /something/ else is skewing your results. If not, then the added network/serialization pressure would explain it. I will a

Re: Flink suggestions;

Am I understanding you correctly in that, if one sensor of on factory raises an alert, then you want all sensors in that same factory to raise an alert? How big is this dataset that maps sensors to factories? Maybe you can just load them into a Map in say a FlatMap, enrich the sensor data str

Re: flink setup errors

This looks like a connectivity issue to maven central. If you follow the http address you also get a 501 HTTPS required error, so maybe adjust the URL in your settings.xml to use https. On 18/05/2020 07:12, Prasanna kumar wrote: I tried to setup flink locally as mentioned in the link https://

Re: Testing process functions

We don't publish sources for test classes. Have you considered that the sink will be serialized on job submission, meaning that your myTestSink instance is not the one actually used by the job? This typically means that have to store stuff in a static field instead. Alternatively, depending on

Re: Apache Flink - Error on creating savepoints using REST interface

You also have to set the boolean cancel-job parameter. On 22/05/2020 22:47, M Singh wrote: Hi: I am using Flink 1.6.2 and trying to create a savepoint using the following curl command using the following references (https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/rest_a

<    1   2   3   4   5   6   7   8   9   10   >