[
https://issues.apache.org/jira/browse/SPARK-25731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandan updated SPARK-25731:
----------------------------
Description:
[https://github.com/apache/spark/tree/master/external]
As far as I can see,
This doesn't have support for newly release *kafka2.0,*
support is available only till *kafka-0-10.*
If we use the
"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0"
for kafka2.0, below is the error I get
11:46:18.061 [stream execution thread for [id =
e393ea37-8009-4ce0-b996-94f767994fb8, runId =
bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2,
groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
*Completed connection to node -1. Fetching API versions.*
11:46:18.061 [stream execution thread for [id =
e393ea37-8009-4ce0-b996-94f767994fb8, runId =
bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2,
groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
*Initiating API versions fetch from node -1.*
11:46:18.452 [stream execution thread for [id =
e393ea37-8009-4ce0-b996-94f767994fb8, runId =
bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2,
groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
Connection with *kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected*
*java.io.EOFException: null*
at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)
I might be wrong, but this is the best option I thought to open an issue.
was:
[https://github.com/apache/spark/tree/master/external]
As far as I can see,
This doesn't have support for newly release *kafka2.0,*
support is available only till *kafka-0-10.*
If we use the
"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0"
for kafka2.0, below is the error I get
11:46:18.061 [stream execution thread for [id =
e393ea37-8009-4ce0-b996-94f767994fb8, runId =
bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2,
groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
*Completed connection to node -1. Fetching API versions.*
11:46:18.061 [stream execution thread for [id =
e393ea37-8009-4ce0-b996-94f767994fb8, runId =
bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2,
groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
*Initiating API versions fetch from node -1.*
11:46:18.452 [stream execution thread for [id =
e393ea37-8009-4ce0-b996-94f767994fb8, runId =
bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2,
groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
Connection with *kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected*
*java.io.EOFException: null*
at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)
> Spark Structured Streaming Support for Kafka 2.0
> ------------------------------------------------
>
> Key: SPARK-25731
> URL: https://issues.apache.org/jira/browse/SPARK-25731
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.3.2
> Reporter: Chandan
> Priority: Major
>
> [https://github.com/apache/spark/tree/master/external]
> As far as I can see,
> This doesn't have support for newly release *kafka2.0,*
> support is available only till *kafka-0-10.*
> If we use the
> "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0"
> for kafka2.0, below is the error I get
> 11:46:18.061 [stream execution thread for [id =
> e393ea37-8009-4ce0-b996-94f767994fb8, runId =
> bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
> org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2,
> groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
> *Completed connection to node -1. Fetching API versions.*
> 11:46:18.061 [stream execution thread for [id =
> e393ea37-8009-4ce0-b996-94f767994fb8, runId =
> bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
> org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2,
> groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
> *Initiating API versions fetch from node -1.*
> 11:46:18.452 [stream execution thread for [id =
> e393ea37-8009-4ce0-b996-94f767994fb8, runId =
> bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG
> org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2,
> groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0]
> Connection with *kafka-muhammad-45e0.aivencloud.com/18.203.67.147
> disconnected*
> *java.io.EOFException: null*
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
> at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)
>
>
> I might be wrong, but this is the best option I thought to open an issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
