[jira] [Updated] (SPARK-27833) Structured Streaming Custom Sink -- java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark
[ https://issues.apache.org/jira/browse/SPARK-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raviteja updated SPARK-27833: - Summary: Structured Streaming Custom Sink -- java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark (was: Structured Streaming Custom Sink --) > Structured Streaming Custom Sink -- java.lang.AssertionError: assertion > failed: No plan for EventTimeWatermark > --- > > Key: SPARK-27833 > URL: https://issues.apache.org/jira/browse/SPARK-27833 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 > Environment: spark 2.3.0 > java 1.8 > kafka version 0.10. >Reporter: Raviteja >Priority: Blocker > Labels: spark-streaming-kafka > Attachments: kafka_consumer_code.java, kafka_custom_sink.java, > kafka_error_log.txt > > > Hi , > We have a requirement to read data from kafka, apply some transformation and > store data to database .For this we are implementing watermarking feature > along with aggregate function and for storing we are writing our own sink > (Structured streaming) .we are using spark 2.3.0, java 1.8 and kafka version > 0.10. > We are getting the below error. > "*java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark > timestamp#39: timestamp, interval 2 minutes*" > > works perfectly fine when we use Console as sink instead custom sink. For > Debugging the issue, we are performing "dataframe.show()" in our custom sink > and nothing else. > Please find the attachment for the Error log and the code. Please look into > this issue as this a blocker and we are not able to proceed further or find > any alternatives as we need watermarking feature. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27833) Structured Streaming Custom Sink --
[ https://issues.apache.org/jira/browse/SPARK-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raviteja updated SPARK-27833: - Attachment: kafka_custom_sink.java > Structured Streaming Custom Sink -- > --- > > Key: SPARK-27833 > URL: https://issues.apache.org/jira/browse/SPARK-27833 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 > Environment: spark 2.3.0 > java 1.8 > kafka version 0.10. >Reporter: Raviteja >Priority: Blocker > Labels: spark-streaming-kafka > Attachments: kafka_consumer_code.java, kafka_custom_sink.java, > kafka_error_log.txt > > > Hi , > We have a requirement to read data from kafka, apply some transformation and > store data to database .For this we are implementing watermarking feature > along with aggregate function and for storing we are writing our own sink > (Structured streaming) .we are using spark 2.3.0, java 1.8 and kafka version > 0.10. > We are getting the below error. > "*java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark > timestamp#39: timestamp, interval 2 minutes*" > > works perfectly fine when we use Console as sink instead custom sink. For > Debugging the issue, we are performing "dataframe.show()" in our custom sink > and nothing else. > Please find the attachment for the Error log and the code. Please look into > this issue as this a blocker and we are not able to proceed further or find > any alternatives as we need watermarking feature. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27833) Structured Streaming Custom Sink --
[ https://issues.apache.org/jira/browse/SPARK-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raviteja updated SPARK-27833: - Attachment: kafka_error_log.txt > Structured Streaming Custom Sink -- > --- > > Key: SPARK-27833 > URL: https://issues.apache.org/jira/browse/SPARK-27833 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 > Environment: spark 2.3.0 > java 1.8 > kafka version 0.10. >Reporter: Raviteja >Priority: Blocker > Labels: spark-streaming-kafka > Attachments: kafka_consumer_code.java, kafka_custom_sink.java, > kafka_error_log.txt > > > Hi , > We have a requirement to read data from kafka, apply some transformation and > store data to database .For this we are implementing watermarking feature > along with aggregate function and for storing we are writing our own sink > (Structured streaming) .we are using spark 2.3.0, java 1.8 and kafka version > 0.10. > We are getting the below error. > "*java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark > timestamp#39: timestamp, interval 2 minutes*" > > works perfectly fine when we use Console as sink instead custom sink. For > Debugging the issue, we are performing "dataframe.show()" in our custom sink > and nothing else. > Please find the attachment for the Error log and the code. Please look into > this issue as this a blocker and we are not able to proceed further or find > any alternatives as we need watermarking feature. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27833) Structured Streaming Custom Sink --
[ https://issues.apache.org/jira/browse/SPARK-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raviteja updated SPARK-27833: - Attachment: kafka_consumer_code.java > Structured Streaming Custom Sink -- > --- > > Key: SPARK-27833 > URL: https://issues.apache.org/jira/browse/SPARK-27833 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 > Environment: spark 2.3.0 > java 1.8 > kafka version 0.10. >Reporter: Raviteja >Priority: Blocker > Labels: spark-streaming-kafka > Attachments: kafka_consumer_code.java, kafka_custom_sink.java, > kafka_error_log.txt > > > Hi , > We have a requirement to read data from kafka, apply some transformation and > store data to database .For this we are implementing watermarking feature > along with aggregate function and for storing we are writing our own sink > (Structured streaming) .we are using spark 2.3.0, java 1.8 and kafka version > 0.10. > We are getting the below error. > "*java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark > timestamp#39: timestamp, interval 2 minutes*" > > works perfectly fine when we use Console as sink instead custom sink. For > Debugging the issue, we are performing "dataframe.show()" in our custom sink > and nothing else. > Please find the attachment for the Error log and the code. Please look into > this issue as this a blocker and we are not able to proceed further or find > any alternatives as we need watermarking feature. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27833) Structured Streaming Custom Sink --
[ https://issues.apache.org/jira/browse/SPARK-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raviteja updated SPARK-27833: - Docs Text: (was: 19/05/24 17:31:58 INFO ConsumerConfig: ConsumerConfig values: metric.reporters = [] metadata.max.age.ms = 30 partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor] reconnect.backoff.ms = 50 sasl.kerberos.ticket.renew.window.factor = 0.8 max.partition.fetch.bytes = 1048576 bootstrap.servers = [10.20.0.10:6667] ssl.keystore.type = JKS enable.auto.commit = false sasl.mechanism = GSSAPI interceptor.classes = null exclude.internal.topics = true ssl.truststore.password = null client.id = ssl.endpoint.identification.algorithm = null max.poll.records = 1 check.crcs = true request.timeout.ms = 4 heartbeat.interval.ms = 3000 auto.commit.interval.ms = 5000 receive.buffer.bytes = 65536 ssl.truststore.type = JKS ssl.truststore.location = null ssl.keystore.password = null fetch.min.bytes = 1 send.buffer.bytes = 131072 value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer group.id = spark-kafka-source-ec244f99-4ea8-467c-bdc5-b2489e2dbf98-1705936181-driver-0 retry.backoff.ms = 100 sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 ssl.trustmanager.algorithm = PKIX ssl.key.password = null fetch.max.wait.ms = 500 sasl.kerberos.min.time.before.relogin = 6 connections.max.idle.ms = 54 session.timeout.ms = 3 metrics.num.samples = 2 key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer ssl.protocol = TLS ssl.provider = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.keystore.location = null ssl.cipher.suites = null security.protocol = PLAINTEXT ssl.keymanager.algorithm = SunX509 metrics.sample.window.ms = 3 auto.offset.reset = earliest 19/05/24 17:31:58 INFO ConsumerConfig: ConsumerConfig values: metric.reporters = [] metadata.max.age.ms = 30 partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor] reconnect.backoff.ms = 50 sasl.kerberos.ticket.renew.window.factor = 0.8 max.partition.fetch.bytes = 1048576 bootstrap.servers = [10.20.0.10:6667] ssl.keystore.type = JKS enable.auto.commit = false sasl.mechanism = GSSAPI interceptor.classes = null exclude.internal.topics = true ssl.truststore.password = null client.id = consumer-1 ssl.endpoint.identification.algorithm = null max.poll.records = 1 check.crcs = true request.timeout.ms = 4 heartbeat.interval.ms = 3000 auto.commit.interval.ms = 5000 receive.buffer.bytes = 65536 ssl.truststore.type = JKS ssl.truststore.location = null ssl.keystore.password = null fetch.min.bytes = 1 send.buffer.bytes = 131072 value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer group.id = spark-kafka-source-ec244f99-4ea8-467c-bdc5-b2489e2dbf98-1705936181-driver-0 retry.backoff.ms = 100 sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 ssl.trustmanager.algorithm = PKIX ssl.key.password = null fetch.max.wait.ms = 500 sasl.kerberos.min.time.before.relogin = 6 connections.max.idle.ms = 54 session.timeout.ms = 3 metrics.num.samples = 2 key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer ssl.protocol = TLS ssl.provider = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.keystore.location = null ssl.cipher.suites = null security.protocol = PLAINTEXT ssl.keymanager.algorithm = SunX509 metrics.sample.window.ms = 3 auto.offset.reset = earliest 19/05/24 17:31:58 INFO AppInfoParser: Kafka version : 0.10.0.1 19/05/24 17:31:58 INFO AppInfoParser: Kafka commitId : a7a17cdec9eaa6c5 19/05/24 17:32:00 INFO MicroBatchExecution: Starting [id = 9fab90c9-b1f8-469e-bc9c-71b3d1e5272c, runId = 56b3c327-d0fb-45b5-a820-4fe16f85f60d]. Use hdfs://nn6.htrunk.com:8020/user/tester/agg_fun_30 to store the query checkpoint. 19/05/24 17:32:00 INFO ConsumerConfig: ConsumerConfig values: metric.reporters = [] metadata.max.age.ms = 30
