Re: sideOutputLateData not propagating late reports once window expires

Please try this: val windowedStream = stream   .keyBy(…)   .window(TumblingEventTimeWindows./of/(…))   .allowedLateness(…)   .sideOutputLateData(lateTag)   .trigger(new myTrigger) val lateStream =   windowedStream.getSideOutput(lateTag); val aggregatedStream = windowedStream.aggregate(new

Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

I believe this is due to FLINK-21037; we did not consider the possibility of users mounting the configuration directly, and instead assumed that modifications to the config always go through the FLINK_PROPERTIES environment variable. That would also be the workaround for your issue. On

Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

You could also configure the number of slots via the TASK_MANAGER_NUMBER_OF_TASK_SLOTS environment variable. On 5/12/2021 2:19 PM, Chesnay Schepler wrote: I believe this is due to FLINK-21037; we did not consider the possibility of users mounting the configuration directly, and instead

Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

format I could provide parameters via FLINK_PROPERTIES ? I'm thinking of abandoning the idea to copy flink-conf in Dockerfile. Is it limited to a specific set of parameters or generic ? ср, 12 мая 2021 г. в 15:20, Chesnay Schepler <mailto:ches...@apache.org>>: You could also

Re: Flink 1.13.0: check flamegraph on WebUI shows 404 not found

Did you enable the flamegraph feature in the configuration? On 5/12/2021 10:51 AM, Yik San Chan wrote: Hi community, Flink 1.13.0 releases flamegraph. However, when I run Flink 1.13.0 locally, and try to check the flamegraph of an operator that is running, I got this "404 not found" error.

Re: [EXTERNAL] Re: sideOutputLateData not propagating late reports once window expires

} @Override public void cancel() {} } On 5/11/2021 6:42 PM, Slotterback, Chris wrote: Hi Chesnay, That doesn’t compile, as WindowedStream doesn’t have the operator getSideOutput, only SingleOutputStreamOperator has that operation. Chris *From: *Chesnay Schepler *Date: *

Re: Prometheus Reporter Enhancement

There is already a ticket for this. Note that this functionality should be implemented in a generic fashion to be usable for all reporters. https://issues.apache.org/jira/browse/FLINK-17495 On 5/18/2021 8:16 PM, Andrew Otto wrote: Sounds useful! On Tue, May 18, 2021 at 2:02 PM Mason Chen

Re: The problem of getting data by Rest API

The WebUI also retrieves all data from the REST API, which should be updated with a minimum interval of 10 seconds. On 5/7/2021 3:57 AM, penguin. wrote: On the Web UI page, we can see that the relevant data is updated every 3S, such as the read-bytes of each operator. But when I get data

Re: The problem of getting data by Rest API

On 5/7/2021 9:14 AM, Chesnay Schepler wrote: The WebUI also retrieves all data from the REST API, which should be updated with a minimum interval of 10 seconds. On 5/7/2021 3:57 AM, penguin. wrote: On the Web UI page, we can see that the relevant data is updated every 3S, such as the read-bytes

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

Can you show us the dependency tree of your project? (If you are using maven, run "mvn dependency:tree") On 5/7/2021 2:15 PM, Ragini Manjaiah wrote: The scala version is same across the pom file . 2.11 On Fri, May 7, 2021 at 5:06 PM Chesnay Schepler <mailto:ches...@apach

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

It looks like you have different scala versions on the classpath. Please check that all your dependencies use the same scala version. On 5/7/2021 1:25 PM, Ragini Manjaiah wrote: Hi , I am surfacing when submitting flink from intellij  IDE . what cloud the issues. Do need to change the scala

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

] \- org.springframework:spring-test:jar:5.0.9.RELEASE:test [INFO]    \- org.springframework:spring-core:jar:5.0.9.RELEASE:test [INFO]       \- org.springframework:spring-jcl:jar:5.0.9.RELEASE:test On Fri, May 7, 2021 at 5:58 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Can yo

Re: The problem of getting data by Rest API

e update 2 T12: Receive update1 T14: Receive update2 On 5/7/2021 9:14 AM, Chesnay Schepler wrote: The WebUI also retrieves all data from the REST API, which should be updated with a minimum interval of 10 seconds. On 5/7/2021 3:57 AM, penguin. wrote: On the Web UI page,

Re: Job failing after enabling Checkpointing

Please have a look at https://stackoverflow.com/questions/32453030/using-an-collectionsunmodifiablecollection-with-apache-flink On 5/10/2021 10:48 AM, sudhansu jena wrote: Hi Team, We have recently enabled check pointing in our flink job using S3 as the state backend, but while submitting

Re: Job failing after enabling Checkpointing

object. Thanks, Sudhansu On Mon, May 10, 2021 at 2:24 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Please have a look at https://stackoverflow.com/questions/32453030/using-an-collectionsunmodifiablecollection-with-apache-flink <https://stackoverflow.c

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

, Ragini Manjaiah wrote: hi , true.. but iam using where scala.version is 2.11 . wondering from where this 2.12 is added org.apache.kafka kafka_${scala.version} ${kafka.version} On Fri, May 7, 2021 at 6:24 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: I see a seve

Re: Unsubscribe

Could you reach out to the Apache Infrastructure team about not being able to unsubscribe? Maybe this functionality is currently broken. On 5/6/2021 12:48 PM, Andrew Kramer wrote: I have been unable to unsubscribe as well. Have tried emailing

Re: Unused Checkpointed folder in S3

Which Flink version are you using? On 5/11/2021 4:09 PM, sudhansu jena wrote: Hi Team, We have recently enabled Check Pointing in our flink job using FSStateBackend pointing to S3 bucket. Below is the sample code for enabling the checkpointing for the job. The query is each time we cancel

Re: Prometheus Reporter Enhancement

mes longer. Would it make sense to remove the tag related information (getAllVariables())? On May 18, 2021, at 3:45 PM, Chesnay Schepler <mailto:ches...@apache.org>> wrote: There is already a ticket for this. Note that this functionality should be implemented in a generic fashion t

Re: CoGroupedStreams$TaggedUnion

Please file a ticket. This is not something you need to worry about in terms of correctness; if anything it may impede performance (and I'm not sure if that is the case). On 5/11/2021 4:43 PM, jinxw wrote: Whenever I use cogroup function to join two datastreams of POJOs, it seems like

Re: After configuration checkpoint strategy, Flink Job cannot restart when job failed

The default number of restart attempts is 1. You need to explicitly configure it to allow more failures. On 6/7/2021 11:53 AM, 1095193...@qq.com wrote: Hi community, I

Re: Best practice for adding support for Kafka variants

The FLIP-27 were primarily aimed at the DataStream API; the integration into the SQL/Table APIs will happen at a later date. On 6/1/2021 5:59 PM, deepthi Sridharan wrote: Thank you, Roman. I should have said our own flavor of Kafka and not version. Thanks for the reference of the new source

Re: Corrupted unaligned checkpoints in Flink 1.11.1

Is there anything in the Flink logs indicating issues with writing the checkpoint data? When the savepoint could not be created, was anything logged from Flink? How did you shut down the cluster? On 6/3/2021 5:56 AM, Alexander Filipchik wrote: Hi, Trying to figure out what happened with our

Re: Prometheus Reporter Enhancement

iscuss implementation details here or in the ticket, but I was thinking about modifying the ReporterScopedSettings to enable this generic tag support. Best, Mason On May 20, 2021, at 4:36 AM, Chesnay Schepler <mailto:ches...@apache.org>> wrote: There is no plan to generally exclude

Re: StreamingFileSink output formatting to CSV

This is handled by the StringEncoder; the one you use (SimpleStringEncoder) just calls toString on the input element. I don't think Flink provides a CSV StringEncoder, but if all you want is remove the parenthesis, then you could wrap the SimpleStringEncoder and trim the first and last

Re: streaming file sink OUT metrics not populating

This is a known issue, and cannot be fixed on the user side. The underlying problem is that this needs to be implemented separately for each source/sink and we haven't gotten around to doing that yet, but some progress is being made for 1.14 to that end. On 6/3/2021 6:06 PM, Vijayendra Yadav

Re: Flattening of events

Have a look at flatMaps: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/operators/overview/#datastream-rarr-datastream-1 On 6/3/2021 8:28 PM, Satish Saley wrote: Hi team, I am trying to figure out a way to flatten events in my Flink app. Event that i am

Re: [VOTE] Release 1.13.1, release candidate #1

+1 (binding) - reviewed website PR - checked repository contents for presence of scala 2.11/2.12 artifacts, and that they are indeed using different scala versions - checked release commit (a7f31926eced15fa2df353b06c71b86c4f9a8e0c) On 5/25/2021 3:32 PM, Dawid Wysakowicz wrote: |Hi everyone,|

Re: Getting Abstract method Error with flink 1.13.0 and 1.12.0

It appears as if flink-siddhi was build against Flink 1.9.0, so it may just not be compatible with later versions of Flink. You will either need to reach out to the maintainers of flink-siddhi, or migrate it to a later Flink version yourself. On 6/1/2021 3:11 PM, Dipanjan Mazumder wrote:

Re: Flink Metrics Naming

etric names in my experience... An example just for clarity, flink__group1_group2_metricName{group1=…, group2=…, flink tags} => flink__metricName{group_info=group1_group2, group1=…, group2=…, flink tags} On Jun 1, 2021, at 9:57 AM, Chesnay Schepler <mailto:ches...@apache.org>> wrote:

Re: Flink Metrics Naming

The uniqueness of metrics and the naming of the Prometheus reporter are somewhat related but also somewhat orthogonal. Prometheus works similar to JMX in that the metric name (e.g., taskmanager.job.task.operator.numRecordsIn) is more or less a _class_ of metrics, with tags/labels allowing you

Re: Unexpected end of ZLIB input stream

What filesystem are you using? Is it possible that the source tries to read a file that is the process of being written to disk? n 6/3/2021 11:32 PM, Billy Bain wrote: We are getting this exception occasionally after a job runs for a month or more. The files don't seem to be corrupt from

Re: Lot of java.util.zip.ZipFile$Source in JobManager's Heap

Those are normal and point to the JDK and Flink classes which remain loaded for the lifetime of the JVM. On 08/07/2021 14:31, Pranjul Ahuja wrote: Hi, On analyzing the heap dump of the JobManager process, I am seeing a lot of instances of java.util.zip.ZipFile$Source which includes open file

Re: owasp-dependency-check is flagging flink 1.13 for scala 2.12.7

Its unlikely to be relevant for you since the vulnerability only affects the scaladocs, i.e., documentation. On 7/2/2021 2:10 PM, Debraj Manna wrote: Hi, I was running owasp-dependency-check in a java application based on flink-1.13.0 (scala

Re: owasp-dependency-check is flagging flink 1.13 for scala 2.12.7

18-17196 <http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2018-17196>* On Fri, Jul 2, 2021 at 7:19 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Its unlikely to be relevant for you since the vulnerability only affects the scaladocs, i.e., documentation. On 7/2

Re: Savepoint failure with operation not found under key

Ordinarily this happens because the status request is sent to a different JM than the one who received the request for creating a savepoint. The meta information for such requests is only stored locally on each JM and neither distributed to all JMs nor persisted anywhere. Did you send both

Re: Savepoint failure with operation not found under key

How much time has passed between the requests? (You can only query the status for about 5 minutes) On 6/29/2021 6:37 AM, Rainie Li wrote: Thanks for the context Chesnay. Yes, I sent both requests to the same JM. Best regards Rainie On Mon, Jun 28, 2021 at 8:33 AM Chesnay Schepler

Re: ValueState is null; checkpointing issues

1) The ValueState can only return a non-null value if a prior value with the same key (in your case, "x.id") has been received. Have you double-checked that this is the case? 2) Checkpointing does not alleviate the need to restart all operators, it alleviates having to reprocess all data. It

Re: Error when trying to setup and run wordcount example on dataproc

I would recommend reaching out to the Beam project instead, because this seems like an issue that is entirely on their side. As for the listed workaround, they are suggesting to modify the wordcount example to also bundle the listed jackson dependencies. Maybe you could use a different

Re: Flink 1.13 fails to load log4j2 yaml configuration file via jackson-dataformat-yaml

uot; I'm assuming that with improved classloader separation, you actually need to put your dependency into `lib/` instead of putting it into your user jar. But I'm pulling in @Chesnay Schepler <mailto:ches...@apache.org> who has much more insights. On Sun, Jul 4, 2021 at 9:45 PM Yuval Itzchako

Re: Presence of Jars in Flink reg security

One of these (plexus-utils) is afaik used by maven, so the scanner is potentially scanning the wrong thing. Or you are scanning all dependencies downloaded during the build of Flink, including everything used by various plugins of the build process & maven itself. On 5/5/2021 11:08 AM, Till

[NOTICE] Flink 1.12.3 artifacts for Scala 2.12 were built against Scala 2.11

To all Scala 2.12 users, Due to a mistake during the release process of Flink 1.12.3 jars intended to be built against Scala 2.12 were actually built against Scala 2.11 . This affects all jars published to maven central; the convenience binaries are not affected. Scala 2.12 users are

Re: ClassLoader leak when using s3a upload through DataSet.output

FileSystems must not be bundled in the user jar. You must place them in lib/ or plugins/, because by bundling it you break our assumption that they exist for the lifetime of the cluster (which in turn means we don't really have to worry about cleaning up). On 2/10/2021 4:01 PM, Vishal

Re: clarification on backpressure metrics in Apache Flink Dashboard

Yes. Unless operator 2 is also back-pressured of course, then you should take a look at the sink. On 2/11/2021 4:50 AM, Marco Villalobos wrote: given: [source] -> [operator 1] -> [operator 2] -> [sink]. If within the dashboard, operator 1 shows that it has backpressure, does that mean I

Re: Seeing Rocks Native Metrics in Data Dog

scopes must be remapped for your Flink metrics to be supported, otherwise they are submitted as custom metrics." Could we instead add host and ids as tags to our metrics? Thanks for your help! On Tue, Jan 26, 2021 at 2:49 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote:

Re: Datadog reporter timeout & OOM issue

Yes, I could see how the memory issue can occur. However, it should be limited to buffering 64 requests; this is the default limit that okhttp imposes on concurrent calls. Maybe lowering this value already does the trick. On 1/27/2021 5:52 AM, Xingcan Cui wrote: Hi all, Recently, I tried to

Re: Datadog reporter timeout & OOM issue

(setting this field is currently not possible from a Flink user perspective; it is something I will investigate) On 1/27/2021 10:30 AM, Chesnay Schepler wrote: Yes, I could see how the memory issue can occur. However, it should be limited to buffering 64 requests; this is the default limit

Re: Timers not firing until stream end

ckopen70.com/s2t/c/5/f18dQhb0S7kv8cpgQZVc6VPt59hl3kW7_k2842PjkFxW2R1KhZ7v4vclW2Rxbb82bzNKzf7GYHvr01?te=W3R5hFj4cm2zwW4fQ47l4fGCmnW3Fbt5S3H4THtF3F6jFSWsSg1=5987503666495488=adc3545f-9610-4164-fa4a-2bddbd615e33>  +44 7961 125282 See our latest features <https://t.sidekickopen70.com/s2t/c/5/f18dQh

Re: JobManager seems to be leaking temporary jar files

The problem of submitted jar files not being closed is a known one: https://issues.apache.org/jira/browse/FLINK-9844 IIRC it's not exactly trivial to fix since class-loading is involved. It's not strictly related to the REST API; it also occurs in the CLI but is less noticeable since jars are

Re: Job execution graph state - INITIALIZING

INITIALIZING is the very first state a job is in. It is the state of a job that has been accepted by the JobManager, but the processing of said job has not started yet. In other words, INITIALIZING = submitted job, CREATED = data-structures and components required for scheduling have been

Re: Could not find a suitable table factory for 'org.apache.flink.table.factories.CatalogFactory' in the classpath

Please try copying the connector jar into the lib/ or opt/ directory. On 3/26/2021 11:59 AM, Yik San Chan wrote: This question is cross-posted on Stack Overflow https://stackoverflow.com/questions/66815572/could-not-find-a-suitable-table-factory-for-org-apache-flink-table-factories-ca

Re: Hadoop is not in the classpath/dependencies

This looks related to HDFS-12920; where Hadoop 2.X tries to read a duration from hdfs-default.xml expecting plain numbers, but in 3.x they also contain time units. On 3/30/2021 9:37 AM, Matthias Seiler wrote: Thank you all for the replies! I did as @Maminspapin suggested and indeed the

Re: State size increasing exponentially in Flink v1.9

Could you show us how you interact with the map state (ideally the full code of your function that accesses the state)? On 3/25/2021 1:13 AM, Almeida, Julius wrote: Hey, Hope you all are doing well! I am using flink v1.9 with RocksDBStateBackend, but over time the state size is increasing

Re: Fail to cancel perJob for that deregisterApplication is not called

Where exactly did you add your own log message? WebMonitorEndpoint.closeAsync() already logs on it's own whether the shutdown future was completed; meaning that it shouldn't have been necessary to add a separate log message. If you now only see the one you added, chances are that it was added

Re: Flink Read S3 Intellij IDEA Error

-11-2-cant-find-implementation-for-s3-despite-correct-plugins-being> Thanks Sri On Wed, Mar 10, 2021 at 11:34 AM Chesnay Schepler mailto:ches...@apache.org>> wrote: Well, you could do this before running the job: // set the ConfigConstan

Re: Flink Read S3 Intellij IDEA Error

ll its not working. Thanks Sri On Fri, 12 Mar 2021 at 06:18, Chesnay Schepler mailto:ches...@apache.org>> wrote: The concept of plugins does not exist in 1.8.1. As a result it should be sufficient for your use-case to add a dependency on flink-s3-fs-hadoop to

Re: Evenly Spreading Out Source Tasks

Is this a brand-new job, with the cluster having all 18 TMs at the time of submission? (or did you add more TMs while the job was running) On 3/12/2021 5:47 PM, Aeden Jameson wrote: Hi Matthias, Yes, all the task managers have the same hardware/memory configuration. Aeden On Fri, Mar 12,

Re: Unit Testing for Custom Metrics in Flink

I've seen being emitted as expected via hitting the appropriately configured metrics endpoint (using a PrometheusReporter). However, while I was trying to write a few unit tests for this, I seemed to encounter an issue. I was following a Stack Overflow post that was answered by @Chesnay

Re: Unit Testing for Custom Metrics in Flink

however I wasn't sure if that was the most appropriate place to add this type of functionality or if it was needed at all. Thanks much, Rion On Tue, Mar 16, 2021 at 4:45 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Was there anything in the logs (ideally on debug)?

Re: Unit Testing for Custom Metrics in Flink

Actually you'd have to further subclass the operatorMetricGroup such that addGroup works as expected. This is admittedly a bit of a drag :/ On 3/16/2021 4:35 PM, Chesnay Schepler wrote: The test harness is fully independent of the MiniClusterResource; it isn't actually running a job. That's

Re: Unit Testing for Custom Metrics in Flink

world, I was trying to use the built-in constructs available (thus the dynamic groups / metrics being added). On Tue, Mar 16, 2021 at 9:36 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Are you actually running a job, or are you using a harness for testing your funct

Re: Flink Read S3 Intellij IDEA Error

Well, you could do this before running the job: // set the ConfigConstants.ENV_FLINK_PLUGINS_DIR environment variable, pointing to a directory containing the plugins PluginManager pluginManager = PluginUtils.createPluginManagerFromRootFolder(new Configuration()); Filesystem.initialize(new

Re: 1.12.2 docker image

they should be released in a day or two. On 3/3/2021 11:18 PM, Bohinski, Kevin wrote: Hi, Are there plans to provide a docker image for 1.12.2? Best kevin

Re: Flink Zookeeper leader change v 1.9.X

1) This could occur due to a number of reasons, like processes crashing, network issues between ZK and Flink, or the JobManager being stuck in some blocking operation for a long time. You will need to take a look at the ZK/Flink logs to narrow things down. 2) For FLINK-14091 the issue was not

Re: Compile Error

The flink-clients dependency is correct. We will need additional information to debug the Job execution failures, because these can happen due to all kind of reasons. Things like the full stacktrace, or exceptions from the logging output. Additionally, I would recommend to base your project

Re: Flink upgrade causes operator to lose state

It is currently not possible to upgrade table API / SQL applications via savepoints. This thread may provide some more insights: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-and-checkpoints-and-savepoints-td40749.html On 3/3/2021 6:53 PM, soumoks wrote: Hi,

Re: Unit Testing State Stores in KeyedProcessFunctions

I do not believe this to be possible. Given that the state will likely in some form affect the behavior of the function (usually in regards to what it outputs), it may be a better idea to test for that. (I suppose you'd want tests like that anyway) On 3/3/2021 8:10 PM, Rion Williams wrote:

Re: Stop vs Cancel with savepoint

Your understanding of cancel vs stop(-with-savepoint) is correct. I agree that we should update the REST API documentation and have a section outlining the problems with cancel-with-savepoint. Would you like to open a ticket yourself? On 3/3/2021 11:16 AM, Thomas Eckestad wrote: Hi! Cancel

Re: Running StreamingJob.java in apache flink workflow

The quickstarts provide you with a skeleton for building applications. It provides you with pre-defined entrypoints (like StreamingJob) and the recommended maven setup to package this application into a jar. However, they do not provide a running application. When you look at the StreamingJob

Re: Allocating tasks to specific TaskManagers

Are trying to do something along the lines of "deploy this subtask to a TM that has a GPU"? If so, then the community is currently working on supporting such use-cases; see FLIP-56 / FLIP-156

Re: Defining GlobalJobParameters in Flink Unit Testing Harnesses

Could you show us how you create test harness? On 3/4/2021 5:13 AM, Rion Williams wrote: Hi all, Early today I had asked a few questions regarding the use of the many testing constructs available within Flink and believe that I have things in a good direction at present. I did run into a

Re: Defining GlobalJobParameters in Flink Unit Testing Harnesses

ing::class.java)     ) } I've also tried a few variants of that with a separate declaration for the function itself, etc. On Thu, Mar 4, 2021 at 6:47 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Could you show us how you create test harness? On 3/4/2021 5:13 AM, Rion Wi

Re: Python Flink cluster: how to shut-down

run-application creates an application cluster that shuts down once the job is complete. As such, canceling the job in this case is equivalent to shutting down the cluster. AFAIK you also need to specify kubernetes arguments when using the list command. (without any argument it just assumes

Re: Defining GlobalJobParameters in Flink Unit Testing Harnesses

ot;) } Does anything look out of place here? I haven't gone spelunking into the source code for this yet, but I'm assuming that I'm setting the correct values on the execution configuration. Thanks again, Rion On Thu, Mar 4, 2021 at 7:57 AM Chesnay Schepler <mailto:ches...@apache.org&g

Re: reading file from s3

Can you show us the full exception stacktrace? Intuitively I would think your cluster configuration contains an invalid value for some memory configuration option. On 3/4/2021 4:45 PM, Avi Levi wrote: Hi , I am pretty new. I am keep on struggling to read a file from s3 but getting this weird 

Re: Compile Error

;count": so different from a streaming job. *Is something I missed due to not understanding those two lines?* * * *Thank you!* On Wed, Mar 3, 2021 at 2:49 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: The flink-clients dependency is correct. We will nee

Re: Compile Error

ence/display/MAVEN/MojoFailureException> On Thu, Mar 4, 2021 at 10:18 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: use mvn install then, it is the closest thing to package. On 3/4/2021 5:12 PM, Abdullah bin Omar wrote: I found this one (attached pic 1) when I ru

Re: Flink 1.11.2 test cases fail with Scala 2.12.12

Coud you check your dependency tree for the version of scala-library? On 2/24/2021 7:28 AM, soumoks wrote: Thank you for the response but this error continues to happen with Scala 2.12.7. The app itself continues to compile without errors but the test cases fail with the same error. Seems to

Re: Sharding of Operators

will flink runtime handle such a situation? *From:*Chesnay Schepler *Sent:* Friday, February 19, 2021 12:52 AM *To:* yidan zhao ; Tripathi,Vikash *Cc:* user *Subject:* Re: Sharding of Operators When you change the parallelism then keys are re-distributed across operators instances. /However

Re: Flink Metrics emitted from a Kubernetes Application Cluster

This is currently not possible. See also FLINK-8358 On 4/9/2021 4:47 AM, Claude M wrote: Hello, I've setup Flink as an Application Cluster in Kubernetes. Now I'm looking into monitoring the Flink cluster in Datadog. This is what is configured in the flink-conf.yaml to emit metrics:

Re: Flink InfluxDb connector not present in Maven

Please reach out to the bahir project (bahir.apache.org) for issues related to the release of Bahir connectors. On 4/17/2021 10:58 AM, Vinay Patil wrote: Hi Team, Flink influx db connector `flink-connector-influxdb_2.1` is not present in Maven , can you please upload the same

Re: Flink docker 1.11.3 actually runs 1.11.2

ssh # python USER flink WORKDIR /opt/flink COPY flink/db-libs/*   lib/ Thanks for the support, Flavio On Tue, Apr 13, 2021 at 7:43 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Please provide steps to reproduce the issue. I can't see anything wrong in th

Re: Flink docker 1.11.3 actually runs 1.11.2

Please provide steps to reproduce the issue. I can't see anything wrong in the dockerfiles (they reference the correct release url), and the referenced release correctly identifies itself as 1.11.3 . I also started a container with the image, started a jobmanager, and the logs show 1.11.3

Re: Max-parellelism limitation

@Olivier Could you clarify which limitation you are referring to? On 4/20/2021 5:23 AM, Guowei Ma wrote: Hi, Olivier Yes. The introduction of this concept is to solve the problem of rescaling the keystate. Best, Guowei On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier

Re: idleTimeMsPerSecond exceeds 1000

This ticket seems related; the issue was fixed in 1.13: https://issues.apache.org/jira/browse/FLINK-19174 On 4/21/2021 4:20 AM, Alexey Trenikhun wrote: Hello, When Flink job mostly idle, idleTimeMsPerSecond for given task_name and subtask_index sometimes exceeds 1000, I saw values up to 1350,

Re: How do I increase number of db connections of the Flink JDBC Connector?

Every works uses exactly 1 connection, so in order to increase the number of connections you must indeed increase the worker parallelism. On 2/19/2021 6:51 AM, Li Peng wrote: Hey folks, I'm trying to use flink to write high throughput incoming data to a SQL db using the JDBC Connector as

Re: How is proctime represented?

Could you check whether this answers your question? https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/timely-stream-processing.html#notions-of-time-event-time-and-processing-time On 2/19/2021 7:29 AM, Rex Fenley wrote: Hello, When using PROCTIME() in CREATE DDL for a

Re: Tag flink metrics to job name

jobname. I’m using the below reporter - || |metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter| Is there any way to tag job names to the task and job manager metrics. Thanks, Hemant On Fri, 19 Feb 2021 at 12:40 AM, Chesnay Schepler <mailto:ches...@apache.org>&

Re: Tag flink metrics to job name

multiple jobs are running on the same cluster. Also, the issue am trying to resolve is I’m seeing high memory usage for one of the containers I want isolate the issue with one job and then investigate further. Thanks, Hemant On Fri, 19 Feb 2021 at 12:18 PM, Chesnay Schepler <mailto:c

Re: How is proctime represented?

filled in when used by an operator, then again, that's still not exactly what it says so I only feel 70% confident like that's what is happening. On Thu, Feb 18, 2021 at 10:55 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Could you check whether this answers your

Re: Tag flink metrics to job name

When you mean "job_name", are you referring to the Prometheus concept of jobs, of the one of Flink? Which of Flink prometheus reporters are you using? On 2/17/2021 7:37 PM, bat man wrote: Hello there, I am using prometheus to push metrics to prometheus and then use grafana for

Re: Sharding of Operators

When you change the parallelism then keys are re-distributed across operators instances. /However/, this re-distribution is limited to the set /maxParallelism /(set via the ExecutionConfig), which by default is 128 if no operators exceeded the parallelism on the first submission. This

Re: latency related to the checkpointing mode EXACTLY ONCE

Yes, if you are only reading committed data than it will take least the checkpoint interval for the data to be available to downstream consumers. On 2/18/2021 6:17 PM, Tan, Min wrote: Hi, We use the checkpointing mode EXACTLY ONCE for some of our flink jobs. I wonder how the

Re: Best practices around checkpoint intervals and sizes?

A lower checkpoint interval (== more checkpoints / time) will consume more resources and hence can affect the job performance. It ultimately boils down to how much latency you are willing to accept when a failure occurs and data has to be re-processed (more checkpoints => less data). How long

Re: failures during job start

on? *From:*Chesnay Schepler *Sent:* Thursday, August 19, 2021 2:23 PM *To:* Colletta, Edward ; user@flink.apache.org *Subject:* Re: failures during job start *NOTICE: This email is from an external sender - **do not click on links or attachments unless you recognize the sender and know

Re: submit new job is not working

Is the problem that previously uploaded jars are no longer available (which would be expected behavior), or that you cannot upload new jars? If it is the latter, could you use the developer tools of you browser to check what response the UI receives when attempting to upload the jar? On

Re: Logging in flink 1.13.2 when running tests

Chesnay Schepler <mailto:ches...@apache.org>> wrote: It's not unusual for this warning to be logged. You could always disable logging from the WebMonitorUtils: logger.wmu.name <http://logger.wmu.name> = org.apache.flink.runtime.webmonitor.WebMonitorUtils logger.

Re: Logging in flink 1.13.2 when running tests

It's not unusual for this warning to be logged. You could always disable logging from the WebMonitorUtils: logger.wmu.name = org.apache.flink.runtime.webmonitor.WebMonitorUtils logger.wmu.level = off On 24/08/2021 18:10, Nikola Hrusov wrote: Hello, I am trying to manage logs better with

Re: Task Managers having trouble registering after restart

on identifying/confirming where the leak is coming from? On Tue, Aug 24, 2021 at 3:48 AM Arvid Heise <mailto:ar...@apache.org>> wrote: Hi Kevin, The metrics are exposed similarly, so I expect the same issues as they come from Kafka's Consumer API itself. I'll pull in

Re: web.timeout usage in the code

There is some internal magic going on. Have a look at the interface the handlers work against, you will see that the timeout parameter is annotated with @RpcTimeout. The AkkaInvocationHandler extracts the timeout based on this annotation and uses it internally. On 24/08/2021 12:05, Juha

<    5   6   7   8   9   10   11   12   13   >