[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/13419 I will prefer refreshing the dataset every time a dataset is reloaded but keeping existing ones unchanged. ~~~scala val df1 = sqlContext.read.parquet(dir).cache() df1.count()

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user sameeragarwal commented on the pull request: https://github.com/apache/spark/pull/13419 @dongjoon-hyun no reason; old habits. I'll fix this. Thanks! :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13419 Hi, @sameeragarwal . Is there any reason to use `SQLContext` instead of `SparkSession` in this PR? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13419 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59668/ Test

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13419 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13419 **[Test build #59668 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59668/consoleFull)** for PR 13419 at commit

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13419#discussion_r65251560 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala --- @@ -67,6 +67,28 @@ class

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13419#discussion_r65251574 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala --- @@ -67,6 +67,28 @@ class

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user sameeragarwal commented on the pull request: https://github.com/apache/spark/pull/13419 @yhuai @mengxr what are your thoughts on this approach? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13419 **[Test build #59668 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59668/consoleFull)** for PR 13419 at commit

[GitHub] spark pull request: [SPARK-15678][SQL] Drop cache on appends and overwrites

GitHub user sameeragarwal opened a pull request: https://github.com/apache/spark/pull/13419 [SPARK-15678][SQL] Drop cache on appends and overwrites ## What changes were proposed in this pull request? SparkSQL currently doesn't drop caches if the underlying data is