[GitHub] spark pull request #14802: [SPARK-17235][SQL] Support purging of old logs in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14802


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14802: [SPARK-17235][SQL] Support purging of old logs in...

[petermaxlee]

Github user petermaxlee commented on a diff in the pull request:

https://github.com/apache/spark/pull/14802#discussion_r76191571
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala
 ---
@@ -155,8 +174,8 @@ class HDFSMetadataLogSuite extends SparkFunSuite with 
SharedSQLContext {
 }
   }
 
-
-  def testManager(basePath: Path, fm: FileManager): Unit = {
+  /** Basic test case for [[FileManager]] implementation. */
+  private def testFileManager(basePath: Path, fm: FileManager): Unit = {
--- End diff --

I renamed this because initially I thought it's a noun meaning "manager for 
testing", rather than "to test the file manager".



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14802: [SPARK-17235][SQL] Support purging of old logs in...

[petermaxlee]

GitHub user petermaxlee opened a pull request:

https://github.com/apache/spark/pull/14802

[SPARK-17235][SQL] Support purging of old logs in MetadataLog

## What changes were proposed in this pull request?
This patch adds a purge interface to MetadataLog, and an implementation in 
HDFSMetadataLog. The purge function is currently unused, but I will use it to 
purge old execution and file source logs in follow-up patches. These changes 
are required in a production structured streaming job that runs for a long 
period of time.

## How was this patch tested?
Added a unit test case in HDFSMetadataLogSuite.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/petermaxlee/spark SPARK-17235

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14802.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14802


commit 0d9d1e6d59fb68996bf96b5238835a0718a8da1a
Author: petermaxlee 
Date:   2016-08-25T07:11:47Z

[SPARK-17235][SQL] Support purging of old logs in MetadataLog




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org