[jira] [Comment Edited] (FLINK-35404) Report metrics of KafkaConsumer in new Kafka source
[ https://issues.apache.org/jira/browse/FLINK-35404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847993#comment-17847993 ] Harshit Mittal edited comment on FLINK-35404 at 5/20/24 8:55 PM: - A better way to solve this problem is to implement interface `MetricsReporter` from kafka metrics using a MetricGroup from apache-flink. was (Author: hmittal83): A better way to solve this problem is to implement KafkaMetrics reporter interface using a MetricGroup from apache-flink. > Report metrics of KafkaConsumer in new Kafka source > --- > > Key: FLINK-35404 > URL: https://issues.apache.org/jira/browse/FLINK-35404 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.13.0 >Reporter: Harshit Mittal >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > > Currently new Kafka source only registers metrics that are available on > KafkaPartitionSplitReader initialization. However, there are metrics added > later in the lifecycle like consumer lag metrics that are missing from the > KafkaConsumer metric group. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35404) Report metrics of KafkaConsumer in new Kafka source
[ https://issues.apache.org/jira/browse/FLINK-35404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshit Mittal updated FLINK-35404: --- Description: Currently new Kafka source only registers metrics that are available on KafkaPartitionSplitReader initialization. However, there are metrics added later in the lifecycle like consumer lag metrics that are missing from the KafkaConsumer metric group. was:Currently Kafka new source doesn't register metrics of KafkaConsumer in KafkaPartitionSplitReader. These metrics should be added for debugging and monitoring purpose. > Report metrics of KafkaConsumer in new Kafka source > --- > > Key: FLINK-35404 > URL: https://issues.apache.org/jira/browse/FLINK-35404 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.13.0 >Reporter: Harshit Mittal >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > > Currently new Kafka source only registers metrics that are available on > KafkaPartitionSplitReader initialization. However, there are metrics added > later in the lifecycle like consumer lag metrics that are missing from the > KafkaConsumer metric group. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35404) Report metrics of KafkaConsumer in new Kafka source
[ https://issues.apache.org/jira/browse/FLINK-35404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847993#comment-17847993 ] Harshit Mittal commented on FLINK-35404: A better way to solve this problem is to implement KafkaMetrics reporter interface using a MetricGroup from apache-flink. > Report metrics of KafkaConsumer in new Kafka source > --- > > Key: FLINK-35404 > URL: https://issues.apache.org/jira/browse/FLINK-35404 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.13.0 >Reporter: Harshit Mittal >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > > Currently new Kafka source only registers metrics that are available on > KafkaPartitionSplitReader initialization. However, there are metrics added > later in the lifecycle like consumer lag metrics that are missing from the > KafkaConsumer metric group. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35404) Report metrics of KafkaConsumer in new Kafka source
[ https://issues.apache.org/jira/browse/FLINK-35404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshit Mittal updated FLINK-35404: --- Fix Version/s: (was: 1.14.0) (was: 1.13.2) > Report metrics of KafkaConsumer in new Kafka source > --- > > Key: FLINK-35404 > URL: https://issues.apache.org/jira/browse/FLINK-35404 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.13.0 >Reporter: Harshit Mittal >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > > Currently Kafka new source doesn't register metrics of KafkaConsumer in > KafkaPartitionSplitReader. These metrics should be added for debugging and > monitoring purpose. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35404) Report metrics of KafkaConsumer in new Kafka source
Harshit Mittal created FLINK-35404: -- Summary: Report metrics of KafkaConsumer in new Kafka source Key: FLINK-35404 URL: https://issues.apache.org/jira/browse/FLINK-35404 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Affects Versions: 1.13.0 Reporter: Harshit Mittal Assignee: Qingsheng Ren Fix For: 1.14.0, 1.13.2 Currently Kafka new source doesn't register metrics of KafkaConsumer in KafkaPartitionSplitReader. These metrics should be added for debugging and monitoring purpose. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-19065) java.lang.IllegalStateException: Auto generated UIDs have been disabled on join
[
https://issues.apache.org/jira/browse/FLINK-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683749#comment-17683749
]
Harshit Mittal edited comment on FLINK-19065 at 2/10/23 12:08 AM:
--
We recently run into this issue and I realize that I have an easy fix that'd
also satisfy the uniqueness/consistency constraint. I'm going to take a stab at
it.
was (Author: hmittal83):
We recently run into this issue and I realize that I have an easy fix that'd
also satisfy the uniqueness constraint.
> java.lang.IllegalStateException: Auto generated UIDs have been disabled on
> join
> ---
>
> Key: FLINK-19065
> URL: https://issues.apache.org/jira/browse/FLINK-19065
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0, 1.11.1
>Reporter: Maris
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> Join operation with AutoGeneratedUID disabled leads to
> {code:java}
> java.lang.IllegalStateException: Auto generated UIDs have been disabled but
> no UID or hash has been assigned to operator Map
> {code}
> code to reproduce
> {code:java}
> class JoinSpec extends AnyFlatSpec with Matchers with Serializable {
> it should "be able to join streams" in {
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env.getConfig.disableAutoGeneratedUIDs()
> val a = env.fromCollection(List("1", "2",
> "3")).name("a").uid("source-uid")
> val b = env.fromCollection(List("1", "2",
> "3")).name("b").uid("source-uid2")
> val c = a
> .join(b)
> .where(identity)
> .equalTo(identity)
> .window(TumblingProcessingTimeWindows.of(Time.seconds(30)))((a, b) =>
> a+b)
> .uid("joined").name("joined")
> c.addSink(s => println(s))
> .name("ab")
> .uid("ab")
> println(env.getExecutionPlan)
> env.execute
> succeed
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-19065) java.lang.IllegalStateException: Auto generated UIDs have been disabled on join
[
https://issues.apache.org/jira/browse/FLINK-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683749#comment-17683749
]
Harshit Mittal commented on FLINK-19065:
We recently run into this issue and I realize that I have an easy fix that'd
also satisfy the uniqueness constraint.
> java.lang.IllegalStateException: Auto generated UIDs have been disabled on
> join
> ---
>
> Key: FLINK-19065
> URL: https://issues.apache.org/jira/browse/FLINK-19065
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0, 1.11.1
>Reporter: Maris
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> Join operation with AutoGeneratedUID disabled leads to
> {code:java}
> java.lang.IllegalStateException: Auto generated UIDs have been disabled but
> no UID or hash has been assigned to operator Map
> {code}
> code to reproduce
> {code:java}
> class JoinSpec extends AnyFlatSpec with Matchers with Serializable {
> it should "be able to join streams" in {
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env.getConfig.disableAutoGeneratedUIDs()
> val a = env.fromCollection(List("1", "2",
> "3")).name("a").uid("source-uid")
> val b = env.fromCollection(List("1", "2",
> "3")).name("b").uid("source-uid2")
> val c = a
> .join(b)
> .where(identity)
> .equalTo(identity)
> .window(TumblingProcessingTimeWindows.of(Time.seconds(30)))((a, b) =>
> a+b)
> .uid("joined").name("joined")
> c.addSink(s => println(s))
> .name("ab")
> .uid("ab")
> println(env.getExecutionPlan)
> env.execute
> succeed
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
