[
https://issues.apache.org/jira/browse/SPARK-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342963#comment-15342963
]
Ryan Blue commented on SPARK-16032:
-----------------------------------
I'm referring to disabling the use of {{partitionBy}} with {{insertInto}} and
removing support for {{saveAsTable}} (from the doc: "I think it's fine we drop
the support in 2.0"). In 1.6, {{partitionBy}} can be used to set up partition
columns as they are expected by {{insertInto}}.
What change caused the confusing and inconsistent behavior? Before this set of
changes, {{partitionBy}} was validated against the table's partitioning (at
least for Hive) like this change set suggests doing when it is used with
{{saveAsTable}}. The insert SQL case is inconsistent, but there are other ways
to solve that problem.
bq. I do not think we should ship 2.0 without fixing these behaviors and try to
fix them in future releases (the fix will possible change the behaviors again).
I know that we want to get this out, but I don't think it is a good idea to put
it in 2.0 before it's ready. This codifies that the "correct" way to write to a
Hive table is to put the partition columns at the end rather than explicitly
marking them, and it disallows marking those columns as you would using
"PARTITION" in SQL. That's going to break jobs and I'm not confident that it's
the right choice.
> Audit semantics of various insertion operations related to partitioned tables
> -----------------------------------------------------------------------------
>
> Key: SPARK-16032
> URL: https://issues.apache.org/jira/browse/SPARK-16032
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Cheng Lian
> Assignee: Wenchen Fan
> Priority: Critical
> Attachments: [SPARK-16032] Spark SQL table insertion auditing -
> Google Docs.pdf
>
>
> We found that semantics of various insertion operations related to partition
> tables can be inconsistent. This is an umbrella ticket for all related
> tickets.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
