HeartSaVioR edited a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531535795 Actually the other intention was let test fail when "includeHeaders" is set to "true", as we suspected from https://github.com/apache/spark/pull/22282#issuecomment-463008868, but turns out it doesn't break even the option is set to true. It's because of the mechanism how `dropDuplicate` works - row is stored as key, and getting value via key doesn't leverage type while comparing as both hashCode and equals of row don't use type. Luckily it doesn't let query crash, but it would be ideal to do type check and avoid the possibility of crashing in runtime - SPARK-27237 (#24173) has been proposed to address this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
