Re: IOException when using Prometheus Monitor

From what I found so far this appears to be an incompatibility between the pushgateway and client version. So you can either a) use an older version of the pushgateway b) bump the version of the prometheus reporter. Unfortunately I cannot tell you which version you would need. On 20/11/2019

Re: Completed job wasn't saved to archive

saving) work fine. But I don't see archived job or message about archiving failure at all. It looks like Flink just didn't try to save job to archive. 21.11.2019, 17:17, "Chesnay Schepler" : If the archiving fails there should be some log message, like "Failed to a

Re: Read multiline JSON/XML

Why vino? He's specifically asking whether Flink offers something _like_ spark. On 29/11/2019 14:39, vino yang wrote: Hi Flavio, IMO, it would take more effect to ask this question in the Spark user mailing list. WDYT? Best, Vino Flavio Pompermaier >

Re: Flink 'Job Cluster' mode Ui Access

lowing procedure for the deployment: https://github.com/apache/flink/blob/master/flink-container/docker/README.md And i tried accessing the path you mentioned: # curl :4081/#/overview {"errors":["Not found."]} Best Regards, Jatin On Thu, Nov 28, 2019 at 10:21 PM Chesnay Scheple

Re: ProcessFunction collect and close, when to use?

1) You should never call close() on the collector; Flink will do that automatically. 2) No, it shouldn't block anything. Flink will look at the next record to process, notice it's a barrier and pass it on immediately. On 29/11/2019 05:29, shuwen zhou wrote: Hi Community, In ProcessFunction

Re: Apache Flink - High Available K8 Setup Wrongly Marked Dead TaskManager

art cause not enough task manager on cluster. Thanks Chesnay Schepler mailto:ches...@apache.org>>, 28 Kas 2019 Per, 18:55 tarihinde şunu yazdı: The akka.watch configuration options haven't been used for a while irrespective of FLINK-13883 (but I can't quite tell atm since when).

Re: Read multiline JSON/XML

I know that at least the Table API can read json, but I don't know how well this translates into other APIs. On 29/11/2019 12:09, Flavio Pompermaier wrote: Hi to all, is there any out-of-the-box

Re: Flink 'Job Cluster' mode Ui Access

Could you try accessing :/#/overview ? The REST API is obviously accessible, and hence the WebUI should be too. How did you setup the session cluster? Are you using some custom Flink build or something, which potentially excluded flink-runtime-web from the classpath? On 28/11/2019 10:02,

Re: Apache Flink - High Available K8 Setup Wrongly Marked Dead TaskManager

The akka.watch configuration options haven't been used for a while irrespective of FLINK-13883 (but I can't quite tell atm since when). Let's start with what version of Flink you are using, and what the taskmanager/jobmanager logs say. On 25/11/2019 12:05, Eray Arslan wrote: Hi, I have

Re: ***UNCHECKED*** Error while confirming Checkpoint

This looks to me like the TwoPhaseCommitSinkFunction is a bit too strict. The notification for complete checkpoints is not reliable; it may be late, not come at all, possibly even in different order than expected. As such, if you a simple case of snapshot -> snapshot -> notify -> notify the

Re: Questions about how to use State Processor API

1. Only the Java API is supported. 2. As far as I can tell you are correct, the given checkpoint path isn't really used. On 04/10/2019 10:39, Tony Wei wrote: Hi, I'm recently trying to use State Processor API, but I have some questions during the development. 1. Does

Re: kinesis consumer metrics user variables

What exactly would you prefer? Without the stream name and shard id you'd end up with name clashes all over the place. Why can you not aggregate them? Surely Datadog supports some way to define a wildcard when definying the tags to aggregate. On 03/10/2019 09:09, Yitzchak Lieberman wrote:

Re: POJO serialization vs immutability

This question should only be relevant for cases where POJOs are used as keys, in which case they /must not/ return a class-constant nor effectively-random value, as this would break the hash partitioning. This is somewhat alluded to in the keyBy() documentation

Re: kinesis consumer metrics user variables

ean option to control it. On Mon, Oct 7, 2019 at 11:05 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: What exactly would you prefer? Without the stream name and shard id you'd end up with name clashes all over the place. Why can you not aggregate them? Surely Datado

Re: POJO serialization vs immutability

correct me if I'm wrong). Jan [1] http://web.mit.edu/6.031/www/sp17/classes/15-equality/ On 10/7/19 1:37 PM, Chesnay Schepler wrote: This question should only be relevant for cases where POJOs are used as keys, in which case they /must not/ return a class-constant nor effectively-ra

Re: containThrowable missing in ExceptionUtils

The listed method no longer exists and was subsumed by ExceptionUtils#findThrowable, which also gives access to the Throwable if it could be found. I have filed FLINK-14334 for updating the documentation. On 02/10/2019 15:48, Nicholas Walton wrote: Hi, I’m trying to implement a failure

Re: Difficult to debug reason for checkpoint decline

There does indeed appear to be a code path in the StreamTask where an exception might not be logger on the TaskExecutor. (StreamTask#handleExecutionException) In FLINK-10753 the CheckpointCoordinator was adjusted to log the full stacktrace, and is part of 1.5.6. On 07/10/2019 09:51, Daniel

Re: Building Flink 1.6.4 fails with "object scala in compiler mirror not found"

The only cause I know for errors such as this are corrupt jars. Try cleaning the maven cache and see if the issue persists. On 07/10/2019 17:36, Aikio, Torste wrote: Hi, I'm trying to build Flink 1.6.4 from source and some of the tests for flink-scala module are failing for me. Are there

Re: Implementing CheckpointableInputFormat

You have to use StreamExecutionEnvironment#createFileInput for implementing CheckpointableInputFormat to have any effect. This internally results in it being used by the MonitoringFileSource. If you use StreamExecutionEnvironment#createInput nothing will be checkpointed for the source; and yes

Re: Computing two aggregate functions on the same window

There doesn't seem to be a built-in way to apply multiple aggregations to a window. You could use an aggregate function that combines other aggregate functions, but admittedly this will get unwieldy as the number of functions increase: public static class MultiAggregateFunction, ACC2, OUT2,

Re: Passing parameters to filter function (in DataStreams)

You can compute the threshold ahead of time and reference it directly in the filter function. (Below are 2 examples, depending on whether you like lambdas or not) final int threshold =computeThreshold(); temperatureStream.filter(new FilterFunction() { @Override public boolean

Re: [QUESTION] How to parallelize with explicit punctuation in Flink?

In other words, you need a way to partition the stream such that a series of items followed by a barrier are never interrupted. I'm wondering whether you could just apply DataStream#partitionCustom to your source: public static class BarrierPartitionerimplements Partitioner { private int

Re: Hiding the user Configurations from the flink dashboard

No, parameters stored in the global job parameters cannot be hidden. Only options configured in flink-conf.yaml are hidden, iff their key contains "password" or "secret". On 09/10/2019 08:26, vivekanand yaram wrote: Hello All, I m just wondering , is there a way to hide the user

Re: flink 1.9

Java 11 support will be part of Flink 1.10 (FLINK-10725). You can take the current master and compile it on Java 11. We have not investigated later Java versions yet. On 09/10/2019 14:14, Vishal Santoshi wrote: Thank you. A related question, has flink been tested with jdk11 or above. ? On

Re: Best way to initialize a custom metric task for all task managers and job manager

A proper solution will required a custom Flink build, were you want to modify org.apache.flink.runtime.metrics.util.MetricUtils#instantiateProcessMetricGroup and org.apache.flink.runtime.metrics.util.MetricUtils#instantiateTaskManagerMetricGroup to add your custom metrics. This is where the

Re: Usage of KafkaDeserializationSchema and KafkaSerializationSchema

That's a fair question; the interface is indeed weird in this regard and does have some issues. From what I can tell you have 2 options: a) have the user pass the topic to the serialization schema constructor, which in practice would be identical to the topic they pass to the producer. b)

Re: REST rescale with Flink on YARN

Older versions of Jetty don't support PATCH requests. You will either have to update it or create a custom Flink version that uses POST for the rescale operation. On 23/01/2020 13:23, Vasily Melnik wrote: Hi all. I'm using Flink 1.8 on YARN with CDH 5.12 When i try to perform rescale request:

Re: Apache Flink - Sharing state in processors

1. a/b) No, they are deserialized into separate instances in any case and are independent afterwards. 2. a/b) No, see 1). 3. a/b) No, as individual tasks are isolated by different class-loaders. On 23/01/2020 09:25, M Singh wrote: Thanks Yun for your answers. By processor I did mean user

Re: Flink RocksDB logs filling up disk space

Please see https://issues.apache.org/jira/browse/FLINK-15068 On 27/01/2020 12:22, Ahmad Hassan wrote: Hello, In our production systems, we see that flink rocksdb checkpoint IO logs are filling up disk space very very quickly in the order of GB's as the logging is very verbose. How do we

Re: Flink TaskManager Logs polluted with InfluxDB metrics reporter errors

What InfluxDB version are you using? On 05/02/2020 19:41, Morgan Geldenhuys wrote: I am trying to setup metrics reporting for Flink using InflixDB, however I am receiving tons of exceptions (listed right at the bottom). Reporting is setup as recommended by the documentation:

Re: Flink TaskManager Logs polluted with InfluxDB metrics reporter errors

3.9 log4j log4j 1.2.17 junit junit 4.12 On 06.02.20 10:58, Chesnay Schepler wrote: Setup-wise, are there any differences to what you had a few months ago? On 06/02/2020 10:40, Morgan Geldenhuys wrote: Further info, the flink cluster (1.9) is running on

Re: 回复: Difference between JobManager and JobMaster

They are similar but not quite the same. The JobManager was responsible for managing multiple jobs, whereas the JobMaster only manages a single job. The responsibility for managing multiple jobs (==JobMasters) and accepting job submissions was moved into the Dispatcher. On 31/01/2020 09:39,

Re: Flink 1.10 on MapR secure cluster with high availability

if I include MapR specific zookeeper in job dependencies and still use out-of-box Flink binary distribution ? Regards, Maxim. On Wed, Feb 5, 2020 at 3:25 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: You must rebuild Flink while overriding zookeeper.version property

Re: Flink 1.10 on MapR secure cluster with high availability

You must rebuild Flink while overriding zookeeper.version property to match your MapR setup. For example: mvn clean package -Dzookeeper.version=3.4.5-mapr-1604 Note that you will also have to configure the MapR repository in your local setup as described here

Re: FlinkKinesisConsumer throws "Unknown datum type org.joda.time.DateTime"

instead of avro POJOs if I use FlinkKinesisConsumer? Sent from my iPhone On Feb 20, 2020, at 4:34 AM, Chesnay Schepler wrote:

Re: Tests in FileUtilsTest while building Flink in local

Is the stacktrace identical in both tests? Did these fail on the command-line or in the IDE? Can you check what directory the java.io.tmpdir points to? On 19/02/2020 20:42, Arujit Pradhan wrote: Hi all, I was trying to build Flink in my local machine and these two unit tests are failing. *

Re: Side Outputs from RichAsyncFunction

I don't think this is possible. At the very least you should be able to workaround this by having your AsyncFunction return an Either, and having a subsequent ProcessFunction do the side-output business. On 19/02/2020 22:25, KristoffSC wrote: Hi, any thoughts about this one? Regards,

Re: JDBC source running continuously

Can you show us where you found the suggestion to use iterate()? On 20/02/2020 02:08, Fanbin Bu wrote: Hi, My app creates the source from JDBC inputformat and running some sql and print out. But the source terminates itself after the query is done. Is there anyway to keep the source running?

Re: build error for flink 1.10

MapR has a known problem with their HTTPS repository where certificates are being rejected in some (not well-defined) cases. The only reliable workaround is to work against the HTTP repository. Do also note that the vendor-repos profile no longer exists in 1.10; additional repositories must be

Re: FlinkKinesisConsumer throws "Unknown datum type org.joda.time.DateTime"

From the stack trace it appears that flink-avro is used, which uses avro 1.8.2 internally by default, for which this appears to be a known issue that was fixed in 1.9.?. Are you sure that avro 1.9.2 is actually being used? On 19/02/2020 23:53, Lian Jiang wrote: Hi, I use a

Re: AWS Client Builder with default credentials

First things first, we do not intend for users to use anything in the S3 filesystem modules except the filesystems itself, meaning that you're somewhat treading on unsupported ground here. Nevertheless, the S3 modules contain a large variety of AWS-provided CerentialsProvider implementations,

Re: [DISCUSS] decentralized scheduling strategy is needed

This is a known issue that's will be fixed in 1.9.2/1.10.0; see https://issues.apache.org/jira/browse/FLINK-12122 . On 15/01/2020 10:07, HuWeihua wrote: Hi, All We encountered some problems during the upgrade from Flink 1.5 to Flink 1.9. Flink's scheduling strategy has changed. Flink 1.9

Re: Read CSV file and and create customized field

You should add an extra map function. On 16/01/2020 17:10, Soheil Pourbafrani wrote: Hi friends, I'm going to read a CSV file that has 3 columns. I want the final loaded datatype to have other columns inferred by that 3 columns. For example, I would split the first column of the CSV file and

Re: How to get Task metrics with StatsD metric reporter?

I presume your job/task names contains a space, which is included in the metrics scope? You can either configure the metric scope such that the job/task ID is included instead, or create a modified version of the StatsDReporter that filters out additional characters(i.e., override

Re: Custom Metrics outside RichFunctions

It is not possible to access metrics from within a schema. I can't think of a non-hacky workaround (the hacky one being to create a custom kafka consumer that checks the schema class, casts it to your specific class, and then calls a method on your schema that accepts a metric group). On

Re: Flink Metrics - PrometheusReporter

Metrics are exposed via reporters by each process separately, whereas the WebUI aggregates metrics. As such you have to configure Prometheus to also scrape the TaskExecutors. On 22/01/2020 16:58, Sidney Feiner wrote: Hey, I've been trying to use the PrometheusReporter and when I used in

Re: Influxdb reporter not honouring the metrics scope

The solution for 1.9 and below is to create a customized version of the influx db reporter which excludes certain tags. On 21/01/2020 19:27, Yun Tang wrote: Hi, Gaurav InfluxDB metrics reporter has a fixed format of reporting metrics which cannot be controlled by the scope. If you don't

Re: Running Flink on java 11

In regards to what we test: We run our tests against Java 8 *and *Java 11, with the compilation and testing being done with the same JDK. In other words, we don't check whether Flink compiled with JDK 8 runs on JDK 11, but we currently have no reason to believe that there is a problem (and

Re: Running Flink on java 11

The error you got is due to an older asm version which is fixed for 1.10 in https://issues.apache.org/jira/browse/FLINK-13467 . On 10/01/2020 15:58, KristoffSC wrote: Hi, Yangze Guo, Chesnay Schepler thank you very much for your answers. I have actually a funny setup. So I have a Flink Job

Re: kafka: how to stop consumption temporarily

Are you asking how to detect from within the job whether the dump is complete, or how to combine these 2 jobs? If you had a way to notice whether the dump is complete, then I would suggest to create a custom source that wraps 2 kafka sources, and switch between them at will based on your

Re: Yarn Kerberos issue

From what I understand from the documentation, if you want to use delegation tokens you always first have to issue a ticket using kinit; so you did everything correctly? On 02/01/2020 13:00, Juan Gentile wrote: Hello, Im trying to submit a job (batch worcount) to a Yarn cluster. I’m trying

Re: Migrate custom partitioner from Flink 1.7 to Flink 1.9

You should be able to implement this on the DataStream API level using DataStream#broadcast and #union like this: input = ... singleChannel = input.filter(x -> !x.isBroadCastPartitioning); broadcastChannel = input.filter(x -> x.isBroadCastPartitioning); result =

Re: Change Akka Ask Timeout for Job Submission Only

There are 3 communication layers involved here: 1) client <=> server (REST API) This goes through REST and does not use timeouts AFAIK. We wait until a response comes or the connection terminates. 2) server (REST API) <=> processes (JM, Dispatcher) This goes through akka, with "web.timeout"

Re: Flink task node shut it self off.

The logs show 2 interesting pieces of information: ... 2019-12-19 18:33:23,278 INFO org.apache.kafka.clients.FetchSessionHandler  - [Consumer clientId=consumer-4, groupId=ccdb-prod-import] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0:

Re: Flink Prometheus metric doubt

In practice the documentation is incorrect. While technically the metric _would_ emit -1 if the job is in a failed/finished state, the reality is that at this point the metric is unregistered and no longer updated, since the owning component (the jobmanager) is shutting down. I can't think of

Re: Elasticsink sometimes gives NoClassDefFoundError

Could you clarify under what circumstances you see this issue? You say "sometimes"; is the job running normally and then failing due to this error? Does it happen when submitting/canceling/restarting a job etc. On 08/01/2020 12:20, Jayant Ameta wrote: Hi, I see the following error sometimes 

Re: Setting app Flink logger

@Eyal: The image you are using is for 1.9.2, but the logging configuration you fetched was from master. In 1.9.2 we use Log4j1, but on master we switched to Log4j2 instead, which uses a different configuration syntax. Log4j1 pretty much ignores the entire file, causing the error. Please

Re: datadog metrics

Please open a JIRA; we may have to split the datatog report into several chunks. On 09/03/2020 07:47, Fanbin Bu wrote: quote from the following link: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SQL-Query-named-operator-exceeds-80-characters-td24807.html#a24818 "This is

Re: datadog metrics

the metrics in chunks and it worked fine after. Would be interested in seeing a ticket on this as well. - Steve On Wed, Mar 11, 2020 at 5:13 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Please open a JIRA; we may have to split the datatog report into several chunks. On 0

Re: [EXT.MSG] Re: datadog http reporter metrics

for rejected metrics, nothing... tor that case there should be an error, right? (when report is too large) I saw that there are some changes on version 1.10 for datadog reporter, maybe I should upgrade to this version? On Mon, Mar 16, 2020 at 11:47 AM Chesnay Schepler <mailto:ches...@apache.

Re: datadog http reporter metrics

Do you see anything in the logs? In another thread a user reported that the datadog reporter could stop working when faced with a large number of metrics since datadog was rejecting the report due to being too large. On 15/03/2020 12:22, Yitzchak Lieberman wrote: Anyone? On Wed, Mar 11, 2020

Re: Upgrading Flink

er to update your project dependencies to the new version (1.10.x), and hence you need to recompile the jobs. On Tue, Apr 14, 2020 at 3:29 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: @Robert Why would he have to recompile the jobs? Shouldn't he be f

Re: [1.10.0] flink-dist source jar is empty

That should not be a problem since the flink-dist module does not contain any java sources On 14/04/2020 06:42, Steven Wu wrote: We build and publish flink-dist locally. But the source jar turns out empty. Other source jars (like flink-core) are good. Anyone else experienced similar

Re: Can I use Apache-Flink for Android API-Level < 26?

I agree with your conclusion that you cannot use Flink on an API Level below 26. I do not know whether it will work even with Level 26 though, as I'm not aware of anyone having tried it. On 14/04/2020 11:03, Alexander Borgschulze wrote: I am trying to use Apache-Flink in my Android-Project

Re: Upgrading Flink

@Robert Why would he have to recompile the jobs? Shouldn't he be fine soo long as he isn't using any API for which we broke binary-compatibility? On 09/04/2020 09:55, Robert Metzger wrote: Hey Stephen, 1. You should be able to migrate from 1.8 to 1.10:

Re: Javadocs Broken?

I'm looking into it. On 10/04/2020 11:27, tison wrote: Hi guys, Right now when I click "JavaDocs" in out docsite[1] it jumps to a page[2] I think is definitely not out api documentation. Any thoughts? Best, tison. [1] https://ci.apache.org/projects/flink/flink-docs-master/ [2]

Re: [1.10.0] flink-dist source jar is empty

contains the source files for the flink modules that it bundles. On Tue, Apr 14, 2020 at 1:34 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: That should not be a problem since the flink-dist module does not contain any java sources On 14/04/2020 06:42, Steven

Re: Suppressing illegal Access Warnings

nvoke(unsafe, loggerClass, offset, null); }catch (Exception ignored) { //ignore } } } Backbutton.co.uk <http://Backbutton.co.uk> ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org <http://makeuse.org> <http://www.backbutton.co.uk> On Tue, 21 Apr 2020 at 09:04,

Re: Suppressing illegal Access Warnings

I do not know where this function comes from (DisableAccessWarning().disableAccessWarnings()),**so we can't be sure. ** On 21/04/2020 00:27, Zahid Rahman wrote: Hi, *I was getting these warnings, I think these are due to certain version of Maven libraries which is impacting Java frameworks

Re: Unable to unmarshall response (com.ctc.wstx.stax.WstxInputFactory cannot be cast to javax.xml.stream.XMLInputFactory)

Which Flink version are you using? On 21/04/2020 11:11, Fu, Kai wrote: Hi, I’m running Flink application on AWS Kinesis Flink platform to read a kinesis stream from another account with assumed role, while I’m getting exception like below. But it works when I’m running the application

Re: Jars replication

Jars are not replicated to all JobManagers, this is currently expected, but generally undesirable for the use-case you out-lined. IIRC the important part is that the upload goes directly against the leader, the run request can be sent anywhere and it will be redirected internally to the

Re: Jars replication

ry to send a run request to the one that doesn't have the jar, it responds with a "jar not found" error. I didn't find documentation about this, so I don't know if it is the expected behavior or something wrong in my cluster configuration. Regards On Fri, Apr 17, 2020 at 10:05

Re: flink-shaded-hadoop2 for flink 1.10

flink-shaded-hadoop2 was released as part of Flink until 1.8 (hence why it followed the Flink version scheme), after which it was renamed to flink-shaded-hadoop-2 and is now being released separately from Flink as part of flink-shaded (a project that bundles various dependencies to be used by

Re: Flink1.10 Cluster's Received is zero in the web when consume from Kafka0.11

This is a known limitation, see https://issues.apache.org/jira/browse/FLINK-7286 . As a crude workaround you may either break the chain after the source / before the sink, or query the numRecordsOut metric for the source / numRecordsIn metric for the sink via the WebUI metrics tab or REST

Re: History Server Not Showing Any Jobs - File Not Found?

wrote: I’m having a further look at the code in HistoryServerStaticFileServerHandler - is there an assumption about where overview.json is supposed to be located? *// *ah** *From:*Hailu, Andreas [Engineering] *Sent:* Wednesday, April 22, 2020 1:32 PM *To:* 'Chesnay Schepler' ; Hailu, Andreas

Re: History Server Not Showing Any Jobs - File Not Found?

/ historyserver.web.tmpdir: /local/scratch/flink_historyserver_tmpdir/ Did you have anything else in mind when you said pointing somewhere funny? *// *ah** *From:*Chesnay Schepler *Sent:* Monday, April 27, 2020 5:56 AM *To:* Hailu, Andreas [Engineering] ; user@flink.apache.org *Subject:* Re: History Server

Re: History Server Not Showing Any Jobs - File Not Found?

Which Flink version are you using? Have you checked the history server logs after enabling debug logging? On 21/04/2020 17:16, Hailu, Andreas [Engineering] wrote: Hi, I’m trying to set up the History Server, but none of my applications are showing up in the Web UI. Looking at the console, I

Re: RocksDB default logging configuration

AFAIK this is not possible; the client doesn't know anything about the cluster configuration. FLINK-15747 proposes to add an additional config option for controlling the logging behavior. The only workaround I can think of would be to create a custom Flink distribution with a modified

Re: History Server Not Showing Any Jobs - File Not Found?

. Could you clarify on how you want to try local filesystem archives? As in changing jobmanager.archive.fs.dir and historyserver.web.tmpdir to the same local directory? *// *ah** *From:*Chesnay Schepler *Sent:* Wednesday, April 29, 2020 8:26 AM *To:* Hailu, Andreas [Engineering] ; user

Re: History Server Not Showing Any Jobs - File Not Found?

/0004be6ce01ba9677d1eb619ad0fa757 ... *// *ah** *From:*Chesnay Schepler *Sent:* Monday, April 27, 2020 10:28 AM *To:* Hailu, Andreas [Engineering] ; user@flink.apache.org *Subject:* Re: History Server Not Showing Any Jobs - File Not Found? If historyserver.web.tmpdir is not set then java.io.tmpdir is used, so

[CVE-2020-1960] Apache Flink JMX information disclosure vulnerability

CVE-2020-1960: Apache Flink JMX information disclosure vulnerability Severity: Medium (CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:L/A:H) Vendor: The Apache Software Foundation Versions Affected: Flink 1.1.0 to 1.1.5 Flink 1.2.0 to 1.2.1 Flink 1.3.0 to 1.3.3 Flink 1.4.0 to 1.4.2 Flink 1.5.0 to

Re: flink setup errors

This looks like a connectivity issue to maven central. If you follow the http address you also get a 501 HTTPS required error, so maybe adjust the URL in your settings.xml to use https. On 18/05/2020 07:12, Prasanna kumar wrote: I tried to setup flink locally as mentioned in the link

Re: Testing process functions

We don't publish sources for test classes. Have you considered that the sink will be serialized on job submission, meaning that your myTestSink instance is not the one actually used by the job? This typically means that have to store stuff in a static field instead. Alternatively, depending

Re: Flink REST API side effect?

This is expected, the backing data structure is cached for a while so we never hammer the JobManager with requests. IIRC this is controlled via "web.refresh-interval", with the default being 3 seconds. On 11/05/2020 14:10, Tomasz Dudziak wrote: Hi, I found an interesting behaviour of the

Re: Flink REST API side effect?

:*Chesnay Schepler *Sent:* 11 May 2020 13:20 *To:* Tomasz Dudziak ; user@flink.apache.org *Subject:* Re: Flink REST API side effect? This is expected, the backing data structure is cached for a while so we never hammer the JobManager with requests. IIRC this is controlled via "web.refresh-int

Re: Flink performance tuning on operators

Generally there should be no difference. Can you check whether the maps are running as a chain (as a single task)? If they are running in a chain, then I would suspect that /something/ else is skewing your results. If not, then the added network/serialization pressure would explain it. I will

Re: Flink suggestions;

Am I understanding you correctly in that, if one sensor of on factory raises an alert, then you want all sensors in that same factory to raise an alert? How big is this dataset that maps sensors to factories? Maybe you can just load them into a Map in say a FlatMap, enrich the sensor data

Re: Hadoop user jar for flink 1.9 plus

You can download flink-shaded-hadoop from the downloads page: https://flink.apache.org/downloads.html#additional-components On 17/03/2020 15:56, Vishal Santoshi wrote: We have been on flink 1.8.x on production and were planning to go to flink 1.9 or above. We have always used hadoop uber jar

Re: Apache Flink - Error on creating savepoints using REST interface

You also have to set the boolean cancel-job parameter. On 22/05/2020 22:47, M Singh wrote: Hi: I am using Flink 1.6.2 and trying to create a savepoint using the following curl command using the following references

Re: RestClusterClient locks file after calling `submitJob(JobGraph)` method on Windows OS

This is a known issue: https://issues.apache.org/jira/browse/FLINK-9844 On 9/11/2020 8:34 PM, Vladislav Keda wrote: Hi Flink Community, I was trying to submit a flink job on a standalone cluster using RestClusterClient. After waiting for job submission, I got JobID correctly and tried to

Re: Checkpoint dir is not cleaned up after cancel the job with monitoring API

Yes, the patch call only triggers the cancellation. You can check whether it is complete by polling the job status via jobs/ and checking whether state is CANCELED. On 9/27/2020 7:02 PM, Eleanore Jin wrote: I have noticed this: if I have Thread.sleep(1500); after the patch call returned 202,

Re: Runtime Dependency Issues Upgrading to Flink 1.11.2 from 1.9.2

I'm not aware of any Flink module bundling this class. Note that this class is also bundled in jersey-core (which is also on your classpath), so it appears that there is a conflict between this jar and your shaded one. Have you changed the Hadoop version you are using or how you provide them to

Re: Correct way to handle RedisSink exception

You will have to create a custom version of the redis connector that ignores such exceptions. On 10/15/2020 1:27 PM, Manas Kale wrote: Hi, I have a streaming application that pushes output to a redis cluster sink. I am using the Apache Bahir[1] Flink redis connector for this. I want to

Re: StatsD metric name prefix change for task manager after upgrading to Flink 1.11

The TaskExecutor host being exposed is directly wired to what the RPC system for addresses, which may have changed due to (FLINK-15911; NAT support). If the problem is purely about the periods in the IP, then I would suggest to create a custom reporter that extends the StatsDReporter and

Re: TaskManager takes abnormally long time to register with JobManager on Kubernetes for Flink 1.11.0

The InetAddress caches the result of getCanonicalHostName(), so it is not a problem to call it twice. On 10/15/2020 1:57 PM, Till Rohrmann wrote: Hi Weike, thanks for getting back to us with your findings. Looking at the `TaskManagerLocation`, we are actually calling

Re: Runtime Dependency Issues Upgrading to Flink 1.11.2 from 1.9.2

lients in some of our modules since we were relying  on the transitive dependency. Our 1.9.2 jobs are still able to run just fine, which is interesting. *// *ah** *From:*Chesnay Schepler *Sent:* Thursday, October 15, 2020 7:34 AM *To:* Hailu, Andreas [Engineering] ; user@flink.apache.org *S

Re: Guide on writing Flink plugins

I would recommend looking at one of the existing reporter modules, like flink-metrics-slf4j. The important bit plugin-wise is to have a factory class and the META-INF/services entry. On 10/6/2020 5:23 AM, Kien Truong wrote: Hi all, We want to write a Flink plugins to integrate Flink jobs

Re: [DISCUSS] Remove flink-connector-filesystem module.

Are older versions of the module compatible with 1.12+? On 10/12/2020 4:30 PM, Kostas Kloudas wrote: Hi all, As the title suggests, this thread is to discuss the removal of the flink-connector-filesystem module which contains (only) the deprecated BucketingSink. The BucketingSin is deprecated

Re: [DISCUSS] Remove flink-connector-filesystem module.

opening a discussion. I think that if we supported backwards compatibility, this would have been an easier process. Kostas On Mon, Oct 12, 2020 at 4:32 PM Chesnay Schepler wrote: Are older versions of the module compatible with 1.12+? On 10/12/2020 4:30 PM, Kostas Kloudas wrote: Hi all

Re: [DISCUSS] Remove flink-connector-filesystem module.

is also suffering from bad memory). Did we manage to add them in the meantime? If not, then it feels rushed to remove it at this point. On Tue, Oct 13, 2020 at 2:33 PM Kostas Kloudas wrote: @Chesnay Schepler Off the top of my head, I cannot find an easy way to migrate from the BucketingSink

<    2   3   4   5   6   7   8   9   10   11   >