Re: Is RestClusterClient recommended?

Don't use the RestClusterClient; you can generate one from the openapi spec (see the docs). On 16/11/2023 20:36, Adrian Alexandru Vasiliu wrote: Hello, For a programmatic use in Java of the Flink REST API, which of these options would be the best choice? 1. Direct use via a REST client

Re: Metrics with labels

> I think this is a general issue with the Flink metrics. Not quite. There are a few instance in Flink were code wasn't updated to encode metadata as additional labels, and the RocksDB metrics may be one of them. Also for RocksDB, you could try setting

Re: Java 21 for flink

At this time, no. On 08/07/2023 04:00, Prasanna kumar wrote: Hi all, Java 21 plans to support light weight thread called fiber based on Project LOOM which will increase the concurrency to great extent. Is there any plan for flink to leverage it? Thanks, Prasanna.

Re: Changing vertex parallelism

On 03/07/2023 19:13, Nikolaos Paraskakis wrote: Is there any way changing job vertex parallelism during runtime (downtime included)? For example, via REST API? At this time, no. 1.18.0 will ship with FLIP-291

Re: Flink migration 1.15 to 1.17 - Os version change

The openjdk images are deprecated, which prevented us from releasing our docker images via some distribution channels. https://issues.apache.org/jira/browse/FLINK-29137 When we switched to the Temurin images we didn't have to change a whole lot, so you might be able to reconstruct a

Re: [DISCUSS] Status of Statefun Project

If you were to fork it /and want to redistribute it/ then the short version is that 1. you have to adhere to the Apache licensing requirements 2. you have to make it clear that your fork does not belong to the Apache Flink project. (Trademarks and all that) Neither should be significant

Re: StatsdMetricsReporter is emitting all metric types as gauges

nit: Whether the Flink counter abstraction is cumulative or not is irrelevant w.r.t. what we send to metric backends. For example, for Datadog we do send count diffs instead of the current total count. On 10/05/2023 09:33, Hang Ruan wrote: Hi, Iris, The Flink counter is cumulative. There

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

>> *From:* Jing Ge via user >> *Sent:* Monday, April 24, 2023 11:15 PM >> *To:* Chesnay Schepler >> *Cc:* Piotr Nowojski ; Alexis Sarda-Espinosa < >> sarda.espin...@gmail.com>; Martijn Visser ; >>

Re: Can I setup standby taskmanagers while using reactive mode?

Reactive mode doesn't support standby taskmanagers. As you said it always uses all available resources in the cluster. I can see it being useful though to not always scale to MAX but (MAX - some_offset). I'd suggest to file a ticket. On 26/04/2023 00:17, Wei Hou via user wrote: Hi Flink

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

As it turns out Kryo isn't a blocker; we ran into a JDK bug. On 31/03/2023 08:57, Chesnay Schepler wrote: https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide Kroy themselves state that v5 likely can't read v2 data. However, both versions can be on the classpath

Re: [Discussion] - Take findify/flink-scala-api under Flink umbrella

> they require additional hop to serialize Scala objects This doesn't necessarily mean that we need a Scala API, because a beefed up type extraction could also solve this. > This single committer is now with us and ready to maintain it in open source. The best situation to be :-) Have you

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide Kroy themselves state that v5 likely can't read v2 data. However, both versions can be on the classpath without classpath as v5 offers a versioned artifact that includes the version in the package. It probably

Re: Disable the chain of the Sink operator

As far as I know that chain between committer and writer is also required for correctness. On 16/02/2023 14:53, weijie guo wrote: Hi wu, I don't think it is a good choice to directly change the strategy of chain. Operator chain usually has better performance and resource utilization. If we

Re: Disable the chain of the Sink operator

As far as I know that chain between committer and writer is also required for correctness. On 16/02/2023 14:53, weijie guo wrote: Hi wu, I don't think it is a good choice to directly change the strategy of chain. Operator chain usually has better performance and resource utilization. If we

Re: Kryo version 2.24.0 from 2015?

.x aren't supposed to compile nor run correctly under Java 11+. On Thu, Feb 9, 2023 at 2:34 AM Chesnay Schepler wrote:  > you can't reasonably stay on the 2015 version forever, refuse to adopt any of the updates or fixes in the 8 years since then, and reasonably expect things

Re: Kryo version 2.24.0 from 2015?

> you can't reasonably stay on the 2015 version forever, refuse to adopt any of the updates or fixes in the 8 years since then, and reasonably expect things to continue to work well. We are well aware that Kryo is a ticking time bomb. > Is there any possibility a future release of Flink can

Re: Docker image Flink 1.15.4

1.15.4 is not released yet. On 26/01/2023 16:06, Peng Zhang wrote: Hi, We would like to use Flink 1.15.4 docker image. The latest seems 1.15.3. Could you make a docker release Flink 1.1.5.4? Thanks! There is a blocking bug https://issues.apache.org/jira/browse/FLINK-28695

Re: Problem with custom SerializationSchema in Flink 1.15

It's a known issue that various connectors/wrappers/etc did not respect the schema lifecycle. This was fixed in 1.16.0 in https://issues.apache.org/jira/browse/FLINK-28807. You will have to lazily initialize the mapper in the serialize() method for previous versions. On 24/01/2023 11:52,

Re: Metrics exposed by flink containing long label values

See https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/#scope-variables-excludes On 06/01/2023 12:09, Surendra Lalwani via user wrote: Hi Team, We are exposing metrics by Flink to prometheus but we are seeing it contains various labels even including

Re: Async IO & Retry: how to get error details

Retry logic and per-request timeouts should be setup within asyncInvoke() (with all error-handling being done via plain CompletableFuture logic), with timeout() sort of acting as a global timeout after which you want the job to fail (e.g., to guard against mistakes in the asyncInvoke() logic).

Re: Using filesystem plugin with MiniCluster

There is no good way in 1.15 IIRC. Adding a dependency on flink-s3-fs-hadoop _can_ work, if you dont run into dependency conflicts. Otherwise you have to create a plugin manager yourself, point it to some local directory via a system property (I think?), and then eagerly call

Re: Flink reactive mode for application clusters on AWS EKS

The adaptive scheduler and reactive mode both already support application clusters since 1.13. https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/ On 19/12/2022 10:17, Tamir Sagi wrote: Hey, We are running stream jobs on application clusters (v1.15.2) 

[ANNOUNCE] Apache flink-connector-cassandra 3.0.0 released

|The Apache Flink community is very happy to announce the release of Apache flink-connector-cassandra 3.0.0. | |Apache Flink® is an open-source stream processing framework ||for| |distributed, high-performing, always-available, and accurate data streaming applications.| |The release is

Re: Query about flink job manager dashboard

, I have a similar question on this topic. Is there an option to disable the frontend altogether but still use REST APIs? Thanks On Wed, Nov 30, 2022 at 1:37 AM Chesnay Schepler wrote: There's no way to disable the jar submission in the UI but have it still work via the REST API

Re: Query about flink job manager dashboard

There's no way to disable the jar submission in the UI but have it still work via the REST API. On 30/11/2022 06:16, naga sudhakar wrote: After disabling the cancel, submit flags facing issues with below api calls. 1) /jars giving 404 2) /jars/upload 3) /jars/{jarid}/run Is there any config

[ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Elasticsearch Connector 3.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available

[ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Elasticsearch Connector 3.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available

Re: Kinesis Connector does not work

ns-logging Or if this is something more general, it might be something to talk about in the Python section of the docs because most Python users are not going to understand the interplay between Java classes. On Tue, 8 Nov 2022 at 18:50, Chesnay Schepler wrote: Said dependency (on commons-logging) is

Re: Kinesis Connector does not work

Said dependency (on commons-logging) is not meant to be provided by the docker image, but bundled in your user-jar (along with the connector). On 08/11/2022 02:14, Matt Fysh wrote: Hi, I'm following the kinesis connector instructions as documented here:

Re: [Security] - Critical OpenSSL Vulnerability

We just push new images with the same tags. On 01/11/2022 14:35, Matthias Pohl wrote: The Docker image for Flink 1.12.7 uses an older base image which comes with openssl 1.1.1k. There was a previous post in the OpenSSL mailing list reporting a low vulnerability being fixed with 3.0.6 and

Re: flink-table-api-scala-bridge sources

Thanks for reporting the issue; I've filed a ticket (FLINK-29803). On 25/10/2022 09:40, Clemens Valiente wrote: Hi everyone I noticed when going through the scala datastream/table api bridge in my IDE I cannot see the source of the code. I believe it is because the Sources are missing on

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Support for records has not been investigated yet. We're still at the stage of getting things to run at all on Java 17. It _may_ be possible, it _may_ not be. On 13/10/2022 07:39, Salva Alcántara wrote: Hi Martijn, Maybe a bit of an off-topic, but regarding Java 17 support, will it be

Re: Flink falls back on to kryo serializer for GenericTypes

There's no alternative to Kryo for generic types, apart from implementing your Flink serializer (but technically at that point the type is no longer treated as a generic type). enableForAvro only forces Avro to be used for POJO types. On 11/10/2022 09:29, Sucheth S wrote: Hello, How to

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

be against consistency with metrics in other operators in Flink but maybe it’s acceptable to have the sink as a special case. Best, Qingsheng On Oct 10, 2022, 19:13 +0800, Chesnay Schepler , wrote: > I’m with Xintong’s idea to treat numXXXSend as an alias of numXXXOut But that's not possi

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

> I’m with Xintong’s idea to treat numXXXSend as an alias of numXXXOut But that's not possible. If it were that simple there would have never been a need to introduce another metric in the first place. It's a rather fundamental issue with how the new sinks work, in that they emit data to the

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

On 10/10/2022 11:24, Martijn Visser wrote: Sidenote: metric names are not mentioned in the FLIP process as a public API. Might make sense to have a separate follow-up to add that to the list (I do think we should list them there). That's a general issue we have. There's a lot of things we _

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

> It's possible that for the sake of the Scala API, we would occasionally require some changes in the Java API. As long as those changes are not detrimental to Java users, they should be considered. That's exactly the model we're trying to get to. Don't fix scala-specific issues with scala

Re: How to read flink state data without setting uid?

You will need to reload the savepoint with the original job and add uids to all operators (while also setting the uid hashes on all operators to properly restore the state). On 22/09/2022 11:06, Chesnay Schepler wrote: Currently the state processor API does not support that. On 22/09/2022 11

Re: How to read flink state data without setting uid?

Currently the state processor API does not support that. On 22/09/2022 11:02, BIGO wrote: I didn't set the uid for my flink operator, is there any way to read the flink state data? State Processor API requires uid. Thanks.

Re: Fail to build Flink 1.15.1

PM, Chesnay Schepler wrote: hmm...we've only seen that error in older Flink version: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/flinkdev/ide_setup/#compilation-fails-with-cannot-find-symbol-symbol-method-defineclass-location-class-sunmiscunsafe Please double-check whether

Re: New licensing for Akka

7, 2022 at 4:30 PM Robin Cassan via user wrote: Thanks a lot for your answers, this is reassuring! Cheers Le mer. 7 sept. 2022 à 13:12, Chesnay Schepler a écrit : Just to squash concerns, we will make sure this license change will not affect Flink

Re: Fail to build Flink 1.15.1

hmm...we've only seen that error in older Flink version: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/flinkdev/ide_setup/#compilation-fails-with-cannot-find-symbol-symbol-method-defineclass-location-class-sunmiscunsafe Please double-check whether you actually checked out

[NOTICE] Blog post regarding Akka's licensing change

Hello, You may have heard about a recent change to the licensing of Akka. We just published a blog-post regarding this change and what it means for Flink. https://flink.apache.org/news/2022/09/08/akka-license-change.html TL;DR: Flink is not in any immediate danger and we will ensure that

Re: [NOTICE] Switch docker image base to Eclipse Temurin

2022 at 1:21 PM Chesnay Schepler wrote: * September 7th On 05/09/2022 11:27, Chesnay Schepler wrote: > On Wednesday, September 9th, the Flink 1.14.5/1.15.2 Docker images > will switch bases > > FROM openjdk:8/11-jar (Debian-based) > TO eclipse-temur

Re: Cassandra sink with Flink 1.15

Are you running into this in the IDE, or when submitting the job to a Flink cluster? If it is the first, then you're probably affected by the Scala-free Flink efforts. Either add an explicit dependency on flink-streaming-scala or migrate to Flink tuples. On 07/09/2022 14:17, Lars Skjærven

Re: Slow Tests in Flink 1.15

The test that gotten slow; how many test cases does it actually contain / how many jobs does it actually run? Are these tests using the table/sql API? On 07/09/2022 14:15, Alexey Trenikhun wrote: We are also observing extreme slow down (5+ minutes vs 15 seconds) in 1 of 2 integration tests .

Re: New licensing for Akka

Just to squash concerns, we will make sure this license change will not affect Flink users in any way. On 07/09/2022 11:14, Robin Cassan via user wrote: Hi all! It seems Akka have announced a licensing change https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka If I

Re: New licensing for Akka

We'll have to look into it. The license would apply to usages of Flink. That said, I'm not sure if we'd even be allowed to use Akka under that license since it puts significant restrictions on the use of the software. If that is the case, then it's either use a fork created by another party or

Re: [NOTICE] Switch docker image base to Eclipse Temurin

* September 7th On 05/09/2022 11:27, Chesnay Schepler wrote: On Wednesday, September 9th, the Flink 1.14.5/1.15.2 Docker images will switch bases FROM openjdk:8/11-jar (Debian-based) TO eclipse-temurin:8/11-jre-jammy (Ubuntu-based) due to the deprecation of the OpenJDK images. Users

[NOTICE] Switch docker image base to Eclipse Temurin

On Wednesday, September 9th, the Flink 1.14.5/1.15.2 Docker images will switch bases FROM openjdk:8/11-jar (Debian-based) TO eclipse-temurin:8/11-jre-jammy (Ubuntu-based) due to the deprecation of the OpenJDK images. Users that customized the images are advised to check for breaking changes.

Re: Exception when calculating throughputEMA in 1.14.3

Since 1.14.6 has not been released yet your best bet is to either disable the debloating feature, upgrade to 1.15, or build Flink yourself. On 23/08/2022 08:48, Chesnay Schepler wrote: https://issues.apache.org/jira/browse/FLINK-25454 On 23/08/2022 04:54, Liting Liu (litiliu) wrote: Hi, we

Re: Exception when calculating throughputEMA in 1.14.3

https://issues.apache.org/jira/browse/FLINK-25454 On 23/08/2022 04:54, Liting Liu (litiliu) wrote: Hi, we are using 1.14.3, but got "Time should be non negative" after the job has been running for days. What should i do to get rid of this Exception? Do i have to disable the network-debloating

Re: Flink Histogram not emitting sum while using Metrics Reporter

We currently do not compute sums for Histograms, and hence this isn't exported to Prometheus. You'd need to use custom histogram (that actually computes sums) and a custom prometheus reporter (that extracts said sum if present) to implement this yourself. On 18/08/2022 18:37, Sarang Vadali

Re: Metrics OOM java heap space

The granularity setting isn't relevant because it only matters when you enable latency metrics, but they are opt-in and the default config is used. You can only enable/disable specific metrics in the upcoming 1.16.0. @Yuriy: You said you had 270k Strings in the StreamConfig; is that accurate?

Re: Making Kafka source respect offset changed externally

uggest updating the documentation to include that statement. I imagine dynamic partition discovery has no effect on this? Regards, Alexis. Am Do., 21. Juli 2022 um 10:03 Uhr schrieb Chesnay Schepler : Flink only reads the offsets from Kafka when the job is initially started from a clear

Re: Job id in logs

No, that is not possible. There are too man shared components (many of which not being aware of jobs at all) for this to be feasible. On 21/07/2022 10:49, Lior Liviev wrote: Hello, is there a way to add job Id to logs to distinguish between different jobs?

Re: Making Kafka source respect offset changed externally

Flink only reads the offsets from Kafka when the job is initially started from a clear slate. Once checkpoints are involved it only relies on offsets stored in the state. On 20/07/2022 14:51, Alexis Sarda-Espinosa wrote: Hello again, I just performed a test using 

Re: Did the semantics of Kafka's earliest offset change with the new source API?

I'm not sure about the previous behavior, but at the very least according to the documentation the behavior is identical. 1.12: https://nightlies.apache.org/flink/flink-docs-release-1.12/dev/connectors/kafka.html#kafka-consumers-start-position-configuration /|setStartFromEarliest()|///

Re: Unaligned checkpoint waiting in 'start delay' with AsyncDataStream

2 10:00 AM To: 'Chesnay Schepler' ; user@flink.apache.org Subject: RE: Unaligned checkpoint waiting in 'start delay' with AsyncDataStream Thank you for trying it out! Hopefully, there is just some setting that needs to be changed. I have an Ubuntu VM where I created a single node Docker swarm. T

Re: Unaligned checkpoint waiting in 'start delay' with AsyncDataStream

I ran your code in the IDE and it worked just fine; checkpoints are being completed and results are printed to the console. Can you expand on how you run the job? On 02/07/2022 00:26, Nathan Sharp wrote: I am attempting to use unaligned checkpointing with AsyncDataStream, but the checkpoints

Re: How to mock new DataSource/Sink

It is indeed not easy to mock sources/sink with the new interfaces. There is an effort to make this easier for sources in the future (FLIP-238 ). For the time being I'd stick with the

Re: Sporadic issues with savepoint status lookup in Flink 1.15

D4C.C00AC080 cidimage002.jpg@01D78D4C.C00AC080 <http://www.genesys.com/> *From: *Chesnay Schepler *Date: *Thursday, June 16, 2022 at 11:32 AM *To: *Peter Westermann , user@flink.apache.org *Subject: *Re: Sporadic issues with savepoint status lookup in Flink 1.15 * EXTERNAL EMAIL - Please use

Re: Sporadic issues with savepoint status lookup in Flink 1.15

is no longer IN_PROGRESS. Peter Westermann Analytics Software Architect cidimage001.jpg@01D78D4C.C00AC080 peter.westerm...@genesys.com <mailto:peter.westerm...@genesys.com> cidimage001.jpg@01D78D4C.C00AC080 cidimage002.jpg@01D78D4C.C00AC080 <http://www.genesys.com/> *From: *Chesnay Sch

Re: Sporadic issues with savepoint status lookup in Flink 1.15

Are there any log messages from the CompletedOperationCache in the logs? On 16/06/2022 16:54, Chesnay Schepler wrote: There is an expected case where this might happen: if too much time has elapsed since the savepoint was completed (default 5 minutes; controlled by rest.async.store-duration

Re: Sporadic issues with savepoint status lookup in Flink 1.15

There is an expected case where this might happen: if too much time has elapsed since the savepoint was completed (default 5 minutes; controlled by rest.async.store-duration) Did this happen earlier than that? On 16/06/2022 15:53, Peter Westermann wrote: We recently upgraded one of our

Re: Flink Shaded dependencies and extending Flink APIs

Can we find a more robust way to support this? Both flink-shaded, any relocation pattern and JsonRowDataSerializationSchema are Flink internals that users shouldn't use/rely on. On 13/06/2022 12:26, Qingsheng Ren wrote: Hi Andrew, This is indeed a tricky case since Flink doesn't provide

Re: Needed help with skipping savepoint state (unsure how to set --allowNonRestoredState in Docker)

You are on the right path with using the --allowNonRestoredState flag; we'll just have to find the right place to put it w.r.t. your setup. Which docker images are you using (flink/statefun/something custom), and how do you submit the job? On 03/06/2022 01:17, Bhavani Balasubramanyam wrote:

Re: Unable to retrieve savepoint status from non-leader/standby in HA with Flink 1.15

I think your analysis is correct; I'll file a ticket. On 03/06/2022 15:28, Nick Birnberg wrote: Hello everyone! Our current setup has us running Flink on Kubernetes in HA mode (Zookeeper) with multiple JobManagers. This appears to be a regression from 1.14. We can use the flink CLI to 

Re: Slow tests on Flink 1.15

Can you give us a more complete stacktrace so we can see what call in Flink is waiting for something? Does this happen to all of your tests? Can you provide us with an example that we can try ourselves? If not, can you describe the test structure (e.g., is it using a MiniClusterResource).

Re: SV: How to enable statefun metrics for the Slf4j reporter

. *Från:* Chesnay Schepler *Skickat:* den 23 maj 2022 14:29:15 *Till:* Christopher Gustafson; user@flink.apache.org *Ämne:* Re: How to enable statefun metrics for the Slf4j reporter You shouldn't have to do more than that. Flink metrics are showing up as expected? Including metrics

Re: flink sql api, exception when setting "table.exec.state.ttl"

You're probably mixing Flink versions. From the stack trace we can see that Flink classes are being loaded from 2 different jars (rocketmq-flink-1.0.0-SNAPSHOT.jar/flink-dist_2.12-1.13.5.jar); I'd suggest to resolve that first and see if the error persists. On 23/05/2022 14:32, 李诗君 wrote:

Re: How to enable statefun metrics for the Slf4j reporter

You shouldn't have to do more than that. Flink metrics are showing up as expected? Including metrics from tasks? On 23/05/2022 14:03, Christopher Gustafson wrote: Hi! I am trying to enable the StateFun metrics in the documentation to be logged using the Slf4j reporter but I cannot figure

Re: Prometheus metrics does not work in 1.15.0 taskmanager

ble options for source / sink works. Remaining metrics are exposed on 9200. The thing is I wanted to investigate in the consumer behavior in the first place :D That`s how I came across the bug. Anyways, big thanks for your greate support! On Wed, May 4, 2022 at 1:53 PM Chesnay Schepler wr

Re: Prometheus metrics does not work in 1.15.0 taskmanager

the Kafka metrics, but I was not quite successful. I am using the table api Kafka connector. Do you have any suggestions on how to overcome this? Could you also provide the ticket number after creation? Thanks, Peter On Wed, May 4, 2022 at 1:22 PM Chesnay Schepler wrote: Yes

Re: Prometheus metrics does not work in 1.15.0 taskmanager

https://issues.apache.org/jira/browse/FLINK-27487 On 04/05/2022 13:22, Chesnay Schepler wrote: Yes, that looks like a new bug in 1.15. The migration to the new non-deprecated Kafka API in the KafkaMetricMutableWrapper was done incorrectly. This should affect every job that uses the new kafka

Re: Prometheus metrics does not work in 1.15.0 taskmanager

this if no other hints or ideas arise. Thanks & Best, Peter On Tue, May 3, 2022 at 4:01 PM Chesnay Schepler wrote: > I noticed that my config of the PrometheusReporter is different here. I have: `metrics.reporter.prom.c

Re: Prometheus metrics does not work in 1.15.0 taskmanager

stems. I will check the option of creating a sample job to reproduce the problem. Best, Peter On Tue, May 3, 2022 at 12:48 PM Chesnay Schepler wrote: You'd help me out greatly if you could provide me with a sample job that runs into the issue. So far I wasn't able to reproduce the issue

Re: Prometheus metrics does not work in 1.15.0 taskmanager

You'd help me out greatly if you could provide me with a sample job that runs into the issue. So far I wasn't able to reproduce the issue, but it should be clear that there is some given 3 separate reports, although it is strange that so far it was only reported for Prometheus. If one of you

Re: Prometheus metrics does not work in 1.15.0 taskmanager

Is there any warning in the logs containing "Error while handling metric"? On 03/05/2022 10:18, ChangZhuo Chen (陳昌倬) wrote: On Tue, May 03, 2022 at 01:00:42AM -0700, Mason Chen wrote: Hi ChangZhou, The warning log indicates that the metric was previously defined and so the runtime is handling

Re: How to debug Metaspace exception?

There are cases where user-code is run on the JobManager. I'm not sure whether though that applies to the JDBC sources. On 02/05/2022 15:45, John Smith wrote: Why do the JDBC jars need to be on the job manager node though? On Mon, May 2, 2022 at 9:36 AM Chesnay Schepler wrote: yes

Re: How to debug Metaspace exception?

Just to be sure I'm running a standalone cluster using zookeeper. So I have 3 zookeepers, 3 job managers and 3 task managers. On Mon, May 2, 2022 at 2:57 AM Chesnay Schepler wrote: And you do should make sure that it is set for both processes! On 02/05/2022 08:43, Ch

Re: How to debug Metaspace exception?

And you do should make sure that it is set for both processes! On 02/05/2022 08:43, Chesnay Schepler wrote: The setting itself isn't taskmanager specific; it applies to both the job- and taskmanager process. On 02/05/2022 05:29, John Smith wrote: Also just to be sure this is a Task Manager

Re: How to debug Metaspace exception?

fig/#classloader-parent-first-patterns-additional> On Thu, Apr 28, 2022 at 10:17 AM Chesnay Schepler wrote: I think what I meant was "either add it to /lib, or [if it is already in /lib but also bundled in the jar] add it to the parent-fir

Re: How to debug Metaspace exception?

I think what I meant was "either add it to /lib, or [if it is already in /lib but also bundled in the jar] add it to the parent-first patterns." On 28/04/2022 15:56, Chesnay Schepler wrote: Pretty sure, even though I seemingly documented it incorrectly :) On 28/04/2022 15:49, John S

Re: How to debug Metaspace exception?

either or On Wed, Apr 27, 2022 at 3:44 AM Chesnay Schepler wrote: You're misinterpreting the docs. The parent/child-first classloading controls where Flink looks for a class /first/, specifically whether we first load from /lib or the user-jar. It does not allow you t

Re: How to debug Metaspace exception?

should put the Apache ignite and my Microsoft drivers in the lib folders of my task managers? And then in my job jar only include them as compile time dependencies? On Tue, Apr 26, 2022 at 10:42 AM Chesnay Schepler wrote: JDBC drivers are well-known for l

Re: How to debug Metaspace exception?

so https://shopify.engineering/optimizing-apache-flink-applications-tips might be helpful (has a section on profiling, as well as classloading). On Tue, Apr 19, 2022 at 4:35 AM Chesnay Schepler wrote: We have

Re: Problems with PrometheusReporter

Please check the logs for warnings. It could be that a metric registered by a job is throwing exceptions. On 20/04/2022 18:45, Peter Schrott wrote: Hi kuweiha, Just to confirm, you tried with 1.15 - none of the rcs are working for me? This port is definitely free as it was already used on

Re: How to debug Metaspace exception?

We have a very rough "guide" in the wiki (it's just the specific steps I took to debug another leak): https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks On 19/04/2022 12:01, huweihua wrote: Hi, John Sorry for the late reply. You can use MAT[1] to analyze the dump

Re: Broadcast state corrupted ?

*From:* Alexey Trenikhun *Sent:* Tuesday, April 12, 2022 7:10:17 AM *To:* Chesnay Schepler ; Flink User Mail List *Subject:* Re: Broadcast state corrupted ? I’ve tried to restore job in environment A (where we observe problem) from savepoint taken in environment B - restored fine. So looks

Re: Discuss making KafkaSubscriber Public

Could you expand a bit on possible alternative implementations that require this interface to become public, opposed to providing more built-in ways to subscribe? On 13/04/2022 11:26, Qingsheng Ren wrote: Thanks for the proposal Mason! I think exposing `KafkaSubscriber` as public API is

Re: Broadcast state corrupted ?

Am I understanding things correctly in that the same savepoint cannot be restored from in 1 environment, while it works fine in 3 others? If so, are they all relying on the same file, or copies of the savepoint? On 10/04/2022 22:39, Alexey Trenikhun wrote: Hello, We have 

Re: Unsubscribe

To unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org.

Re: Unsubscribed

To unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org.

Re: Issue with Flink UI for Flink 1.14.0

e001.jpg@01D78D4C.C00AC080 cidimage002.jpg@01D78D4C.C00AC080 <http://www.genesys.com/> *From: *Chesnay Schepler *Date: *Friday, January 21, 2022 at 3:28 AM *To: *Peter Westermann , Dawid Wysakowicz , user@flink.apache.org *Subject: *Re: Issue with Flink UI for Flink 1.14.0 While FLINK-2

Re: scala shell not part of 1.14.4 download

The Scala Shell only works with Scala 2.11. You will need to use the Scala 2.11 Flink distribution. On 18/03/2022 12:42, Georg Heiler wrote: Hi, https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/repls/scala_shell/ mentions: |bin/start-scala-shell.sh local | |a

Re: Using another FileSystem configuration while creating a job

Or an entry in the Flink Documentation? Thanks again for your help, Gil On Thu, 10 Mar 2022 at 10:57, Chesnay Schepler wrote: The FileSystem class is essentially one big singleton, with only 1 instance of each FileSystem implementation being loaded, shared across all jobs. For that

Re: Customizing backpressure mechanism for RichParallelSourceFunction

It's not possible to send events to sources; data only flows in 1 direction. On 03/03/2022 06:31, Le Xu wrote: Hello! I have a dataflow pipeline built using Flink's RichParallelSourceFunction as parallel sources. I'm wondering if there are any mechanisms that I could use to implement

Re: Using another FileSystem configuration while creating a job

The FileSystem class is essentially one big singleton, with only 1 instance of each FileSystem implementation being loaded, shared across all jobs. For that reason we do not support job-specific FileSystem configurations. Note that we generally also don't really support configuring the

Re: Savepoint API challenged with large savepoints

That all sounds very interesting; I'd go ahead with creating tickets. On 08/03/2022 13:43, Schwalbe Matthias wrote: Dear Flink Team, In the last weeks I was faced with a large savepoint (around 40GiB) that contained lots of obsolete data points and overwhelmed our infrastructure (i.e.

  1   2   3   4   5   6   7   8   9   10   >