[ https://issues.apache.org/jira/browse/SPARK-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341658#comment-15341658 ]
Cheng Lian commented on SPARK-16032: ------------------------------------ Hey [~rdblue], [~yhuai] and [~cloud_fan] had already covered a lot of details. I think one problem here is that we made all the changes during a relatively short time with quite a few PRs, thus it can be quite hard to grasp the main idea as well as to track all the details. I agree that by name resolution can be more intuitive for users, but the most important issue we would like to address here is *inconsistency*. From the attached report we may see that table insertion behaviors can be inconsistent across the following dimensions: # Table type: Hive SerDe table or data source table # Partition type: static or dynamic # Write mode: overwriting or appending # API type: SQL or DataFrame DSL # Resolution manner: by name or by position All the subtasks listed here mostly aim to ensure consistent and predictable behaviors. However, unfortunately, we noticed all these issues too late, and it can be hard to cover all the dimensions all at once at this stage. From my perspective, the reason why we couldn't cover by name resolution here is that: # Hive and conventional SQL databases have clearly defined semantics of by position resolution # By name resolution can be useful, but it takes more time to achieve reasonable and consistent syntax and semantics for both SQL and DataFrame API. # [PR #12313|https://github.com/apache/spark/pull/12313] only covers the code path of Hive SerDe tables, and it could be hard to cover all the cases if the inconsistencies across the first 3 dimensions are not properly addressed first. Due to the above reasons, we decided to follow the conventional and standard SQL semantics, i.e. by position resolution, to ensure consistent semantics first. You may argue that we dropped existing by name resolution feature in some cases. However, this feature wasn't well defined at the first place, and can be error prone and misleading. On the other hand, by fixing all existing inconsistencies, it would be a lot easier to introduce by name resolution in 2.1 after we come up with a better design. > Audit semantics of various insertion operations related to partitioned tables > ----------------------------------------------------------------------------- > > Key: SPARK-16032 > URL: https://issues.apache.org/jira/browse/SPARK-16032 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.0 > Reporter: Cheng Lian > Assignee: Wenchen Fan > Priority: Blocker > Attachments: [SPARK-16032] Spark SQL table insertion auditing - > Google Docs.pdf > > > We found that semantics of various insertion operations related to partition > tables can be inconsistent. This is an umbrella ticket for all related > tickets. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org