[jira] [Commented] (BAHIR-164) Spark Streaming with MQTT fails with Spark 2.3.0
[ https://issues.apache.org/jira/browse/BAHIR-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679250#comment-16679250 ] ASF subversion and git services commented on BAHIR-164: --- Commit b3902bac67edc2134bcc2c755fadc5c60c8ae01c in bahir's branch refs/heads/master from [~prashant.x90] [ https://git-wip-us.apache.org/repos/asf?p=bahir.git;h=b3902ba ] [BAHIR-164][BAHIR-165] Port Mqtt sql source to datasource v2 API Migrating Mqtt spark structured streaming connector to DatasourceV2 API. Closes #65 > Spark Streaming with MQTT fails with Spark 2.3.0 > > > Key: BAHIR-164 > URL: https://issues.apache.org/jira/browse/BAHIR-164 > Project: Bahir > Issue Type: Sub-task > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.2.0, Spark-2.3 >Reporter: Jussi-Pekka Partanen >Priority: Blocker > > Currently I’m not able to use latest Spark 2.3.0 release together with Bahir, > but getting the following error: > > _java.lang.AssertionError: assertion failed: DataFrame returned by getBatch > from org.apache.bahir.sql.streaming.mqtt.MQTTTextStreamSource@66847d45 did > not have isStreaming=true_ > > org/apache/bahir/sql/streaming/mqtt/MQTTStreamSource.scala line 164 seems to > be causing the issue, i.e. it’s returning a normal dataframe instead of > streaming dataframe. > > There seems to be discussion about the same topic, but on stream source for > Kafka: > https://lists.apache.org/thread.html/015f2fbe9fbeafb86214a969a2d883a4898621189a86f21a4c6fbb75@%3Cdev.spark.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-164) Spark Streaming with MQTT fails with Spark 2.3.0
[ https://issues.apache.org/jira/browse/BAHIR-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494653#comment-16494653 ] ASF GitHub Bot commented on BAHIR-164: -- Github user ScrapCodes commented on the issue: https://github.com/apache/bahir/pull/65 Yes, this is step one in supporting spark 2.3. The other connectors need to be updated before bahir fully supports spark 2.3+ > Spark Streaming with MQTT fails with Spark 2.3.0 > > > Key: BAHIR-164 > URL: https://issues.apache.org/jira/browse/BAHIR-164 > Project: Bahir > Issue Type: Sub-task > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.2.0, Spark-2.3 >Reporter: Jussi-Pekka Partanen >Priority: Blocker > > Currently I’m not able to use latest Spark 2.3.0 release together with Bahir, > but getting the following error: > > _java.lang.AssertionError: assertion failed: DataFrame returned by getBatch > from org.apache.bahir.sql.streaming.mqtt.MQTTTextStreamSource@66847d45 did > not have isStreaming=true_ > > org/apache/bahir/sql/streaming/mqtt/MQTTStreamSource.scala line 164 seems to > be causing the issue, i.e. it’s returning a normal dataframe instead of > streaming dataframe. > > There seems to be discussion about the same topic, but on stream source for > Kafka: > https://lists.apache.org/thread.html/015f2fbe9fbeafb86214a969a2d883a4898621189a86f21a4c6fbb75@%3Cdev.spark.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-164) Spark Streaming with MQTT fails with Spark 2.3.0
[ https://issues.apache.org/jira/browse/BAHIR-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494651#comment-16494651 ] ASF GitHub Bot commented on BAHIR-164: -- Github user lresende commented on the issue: https://github.com/apache/bahir/pull/65 Thank you for these updates @ScrapCodes. Just clarifying, this should not be applied until we do the Bahir release that supports Spark 2.3, correct? > Spark Streaming with MQTT fails with Spark 2.3.0 > > > Key: BAHIR-164 > URL: https://issues.apache.org/jira/browse/BAHIR-164 > Project: Bahir > Issue Type: Sub-task > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.2.0, Spark-2.3 >Reporter: Jussi-Pekka Partanen >Priority: Blocker > > Currently I’m not able to use latest Spark 2.3.0 release together with Bahir, > but getting the following error: > > _java.lang.AssertionError: assertion failed: DataFrame returned by getBatch > from org.apache.bahir.sql.streaming.mqtt.MQTTTextStreamSource@66847d45 did > not have isStreaming=true_ > > org/apache/bahir/sql/streaming/mqtt/MQTTStreamSource.scala line 164 seems to > be causing the issue, i.e. it’s returning a normal dataframe instead of > streaming dataframe. > > There seems to be discussion about the same topic, but on stream source for > Kafka: > https://lists.apache.org/thread.html/015f2fbe9fbeafb86214a969a2d883a4898621189a86f21a4c6fbb75@%3Cdev.spark.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-164) Spark Streaming with MQTT fails with Spark 2.3.0
[ https://issues.apache.org/jira/browse/BAHIR-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471645#comment-16471645 ] ASF GitHub Bot commented on BAHIR-164: -- Github user ScrapCodes commented on the issue: https://github.com/apache/bahir/pull/65 ok to test > Spark Streaming with MQTT fails with Spark 2.3.0 > > > Key: BAHIR-164 > URL: https://issues.apache.org/jira/browse/BAHIR-164 > Project: Bahir > Issue Type: Sub-task > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.2.0, Spark-2.3 >Reporter: Jussi-Pekka Partanen >Priority: Blocker > > Currently I’m not able to use latest Spark 2.3.0 release together with Bahir, > but getting the following error: > > _java.lang.AssertionError: assertion failed: DataFrame returned by getBatch > from org.apache.bahir.sql.streaming.mqtt.MQTTTextStreamSource@66847d45 did > not have isStreaming=true_ > > org/apache/bahir/sql/streaming/mqtt/MQTTStreamSource.scala line 164 seems to > be causing the issue, i.e. it’s returning a normal dataframe instead of > streaming dataframe. > > There seems to be discussion about the same topic, but on stream source for > Kafka: > https://lists.apache.org/thread.html/015f2fbe9fbeafb86214a969a2d883a4898621189a86f21a4c6fbb75@%3Cdev.spark.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-164) Spark Streaming with MQTT fails with Spark 2.3.0
[ https://issues.apache.org/jira/browse/BAHIR-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456281#comment-16456281 ] Prashant Sharma commented on BAHIR-164: --- MQTT source has to be migrated to new DataSourceV2 API. > Spark Streaming with MQTT fails with Spark 2.3.0 > > > Key: BAHIR-164 > URL: https://issues.apache.org/jira/browse/BAHIR-164 > Project: Bahir > Issue Type: Bug > Components: Spark Streaming Connectors >Affects Versions: Spark-2.2.0, Spark-2.3 >Reporter: Jussi-Pekka Partanen >Priority: Blocker > > Currently I’m not able to use latest Spark 2.3.0 release together with Bahir, > but getting the following error: > > _java.lang.AssertionError: assertion failed: DataFrame returned by getBatch > from org.apache.bahir.sql.streaming.mqtt.MQTTTextStreamSource@66847d45 did > not have isStreaming=true_ > > org/apache/bahir/sql/streaming/mqtt/MQTTStreamSource.scala line 164 seems to > be causing the issue, i.e. it’s returning a normal dataframe instead of > streaming dataframe. > > There seems to be discussion about the same topic, but on stream source for > Kafka: > https://lists.apache.org/thread.html/015f2fbe9fbeafb86214a969a2d883a4898621189a86f21a4c6fbb75@%3Cdev.spark.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
