[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...
Github user erfangc commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r73814190 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -298,6 +298,7 @@ case class InsertIntoHiveTable( // Invalidate the cache. sqlContext.sharedState.cacheManager.invalidateCache(table) + sqlContext.sessionState.catalog.invalidateTable(table.catalogTable.identifier) --- End diff -- Ok, will do - thought so too as this relates to `InsertIntoHive` was a hail mary --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...
Github user erfangc commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r73807925 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -298,6 +298,7 @@ case class InsertIntoHiveTable( // Invalidate the cache. sqlContext.sharedState.cacheManager.invalidateCache(table) + sqlContext.sessionState.catalog.invalidateTable(table.catalogTable.identifier) --- End diff -- Hi @mallman does this change cause new sessions (ex: external App to the ThriftServer via JDBC) to not see the cached tables? I noticed this in the released version 2.0.0 whereby `CACHE TABLE` in one session has no effect on new sessions. Future SQL statements are still reading the underlying Parquet files from Disk (as evidenced by tasks being `NODE_LOCAL` and `RACK_LOCAL` instead of `PROCESS_LOCAL`). Sorry if this question is unrelated to your patch, but this became a major issue in 2.0.0 for us, where as in 1.6.2 we do not have an issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3586][streaming]Support nested director...
Github user erfangc commented on the pull request: https://github.com/apache/spark/pull/2765#issuecomment-107060825 This feature would definitely be helpful. Thanks to @wangxiaojing and whoever continuing to work on PR! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
