[jira] [Commented] (SPARK-23203) DataSourceV2 should use immutable trees.
[ https://issues.apache.org/jira/browse/SPARK-23203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346253#comment-16346253 ] Apache Spark commented on SPARK-23203: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/20448 > DataSourceV2 should use immutable trees. > > > Key: SPARK-23203 > URL: https://issues.apache.org/jira/browse/SPARK-23203 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ryan Blue >Priority: Major > > The DataSourceV2 integration doesn't use [immutable > trees|https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html], > which is a basic requirement of Catalyst. The v2 relation should not wrap a > mutable reader and change the logical plan by pushing projections and filters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23203) DataSourceV2 should use immutable trees.
[ https://issues.apache.org/jira/browse/SPARK-23203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338120#comment-16338120 ] Marcelo Vanzin commented on SPARK-23203: Ah, cool, wasn't aware that it was still experimental. Was just wondering since I saw this before I meant to reply to the rc1 e-mail. (I'm also concerned about the amount of patches going in after rcs started, but I generally count that as standard operation procedure during initial rcs...) > DataSourceV2 should use immutable trees. > > > Key: SPARK-23203 > URL: https://issues.apache.org/jira/browse/SPARK-23203 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ryan Blue >Priority: Major > > The DataSourceV2 integration doesn't use [immutable > trees|https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html], > which is a basic requirement of Catalyst. The v2 relation should not wrap a > mutable reader and change the logical plan by pushing projections and filters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23203) DataSourceV2 should use immutable trees.
[ https://issues.apache.org/jira/browse/SPARK-23203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338112#comment-16338112 ] Ryan Blue commented on SPARK-23203: --- [~vanzin], given that DataSourceV2 is experimental, I don't think it should block the release. But I also don't think it was a great idea to rush patches lately to get them into 2.3.0. We are shipping an experimental implementation that will no doubt change a lot as we add basics, like checking a data source table's schema against the data frame that is being written (which is completely by-passed right now). > DataSourceV2 should use immutable trees. > > > Key: SPARK-23203 > URL: https://issues.apache.org/jira/browse/SPARK-23203 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ryan Blue >Priority: Major > > The DataSourceV2 integration doesn't use [immutable > trees|https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html], > which is a basic requirement of Catalyst. The v2 relation should not wrap a > mutable reader and change the logical plan by pushing projections and filters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23203) DataSourceV2 should use immutable trees.
[ https://issues.apache.org/jira/browse/SPARK-23203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338088#comment-16338088 ] Apache Spark commented on SPARK-23203: -- User 'rdblue' has created a pull request for this issue: https://github.com/apache/spark/pull/20387 > DataSourceV2 should use immutable trees. > > > Key: SPARK-23203 > URL: https://issues.apache.org/jira/browse/SPARK-23203 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ryan Blue >Priority: Major > > The DataSourceV2 integration doesn't use [immutable > trees|https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html], > which is a basic requirement of Catalyst. The v2 relation should not wrap a > mutable reader and change the logical plan by pushing projections and filters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23203) DataSourceV2 should use immutable trees.
[ https://issues.apache.org/jira/browse/SPARK-23203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338063#comment-16338063 ] Marcelo Vanzin commented on SPARK-23203: [~rdblue] is this something that affects the API or is it just an internal change? (Or, in other words, "should it block 2.3"?) > DataSourceV2 should use immutable trees. > > > Key: SPARK-23203 > URL: https://issues.apache.org/jira/browse/SPARK-23203 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ryan Blue >Priority: Major > > The DataSourceV2 integration doesn't use [immutable > trees|https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html], > which is a basic requirement of Catalyst. The v2 relation should not wrap a > mutable reader and change the logical plan by pushing projections and filters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
