[jira] [Commented] (SPARK-22824) Spark Structured Streaming Source trait breaking change

2017-12-18 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-22824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295745#comment-16295745
 ] 

Apache Spark commented on SPARK-22824:
--

User 'joseph-torres' has created a pull request for this issue:
https://github.com/apache/spark/pull/20012

> Spark Structured Streaming Source trait breaking change
> ---
>
> Key: SPARK-22824
> URL: https://issues.apache.org/jira/browse/SPARK-22824
> Project: Spark
>  Issue Type: Bug
>  Components: Structured Streaming
>Affects Versions: 2.3.0
> Environment: Azure Databricks 3.4
>Reporter: Matthieu Maitre
>Assignee: Jose Torres
>
> org.apache.spark.sql.execution.streaming.Offset was moved to 
> org.apache.spark.sql.sources.v2.reader.Offset on the Source trait:
> https://github.com/apache/spark/commit/f8c7c1f21aa9d1fd38b584ca8c4adf397966e9f7#diff-56453fdb4dc2d7666c7ab237cb102189
> This broke our custom sources that are called in Spark jobs running on Azure 
> Databricks 3.4 since the Maven package does not match the hosted Spark bits. 
> We use the following Maven version to be able to deploy on Azure Databricks:
> 
> org.apache.spark
> spark-sql_2.11
> 2.3.0-SNAPSHOT
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-22824) Spark Structured Streaming Source trait breaking change

2017-12-18 Thread Michael Armbrust (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-22824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295676#comment-16295676
 ] 

Michael Armbrust commented on SPARK-22824:
--

This is technically an internal API (as is all of 
[org.apache.spark.sql.execution|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/package.scala]).
  That said, I think we should do our best to preserve compatibility until 
there is a {{@Stable}} public alternative (which is the goal of the V2 project).

We'll open a PR to try and fix this.

> Spark Structured Streaming Source trait breaking change
> ---
>
> Key: SPARK-22824
> URL: https://issues.apache.org/jira/browse/SPARK-22824
> Project: Spark
>  Issue Type: Bug
>  Components: Structured Streaming
>Affects Versions: 2.3.0
> Environment: Azure Databricks 3.4
>Reporter: Matthieu Maitre
>
> org.apache.spark.sql.execution.streaming.Offset was moved to 
> org.apache.spark.sql.sources.v2.reader.Offset on the Source trait:
> https://github.com/apache/spark/commit/f8c7c1f21aa9d1fd38b584ca8c4adf397966e9f7#diff-56453fdb4dc2d7666c7ab237cb102189
> This broke our custom sources that are called in Spark jobs running on Azure 
> Databricks 3.4 since the Maven package does not match the hosted Spark bits. 
> We use the following Maven version to be able to deploy on Azure Databricks:
> 
> org.apache.spark
> spark-sql_2.11
> 2.3.0-SNAPSHOT
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-22824) Spark Structured Streaming Source trait breaking change

2017-12-18 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-22824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295260#comment-16295260
 ] 

Sean Owen commented on SPARK-22824:
---

[~zsxwing] [~joseph.torres] was that meant to be an internal API?

> Spark Structured Streaming Source trait breaking change
> ---
>
> Key: SPARK-22824
> URL: https://issues.apache.org/jira/browse/SPARK-22824
> Project: Spark
>  Issue Type: Bug
>  Components: Structured Streaming
>Affects Versions: 2.3.0
> Environment: Azure Databricks 3.4
>Reporter: Matthieu Maitre
>
> org.apache.spark.sql.execution.streaming.Offset was moved to 
> org.apache.spark.sql.sources.v2.reader.Offset on the Source trait:
> https://github.com/apache/spark/commit/f8c7c1f21aa9d1fd38b584ca8c4adf397966e9f7#diff-56453fdb4dc2d7666c7ab237cb102189
> This broke our custom sources that are called in Spark jobs running on Azure 
> Databricks 3.4 since the Maven package does not match the hosted Spark bits. 
> We use the following Maven version to be able to deploy on Azure Databricks:
> 
> org.apache.spark
> spark-sql_2.11
> 2.3.0-SNAPSHOT
> 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org