[
https://issues.apache.org/jira/browse/FALCON-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karishma Gulati updated FALCON-321:
-----------------------------------
Attachment: oozie-logs-test-2.txt
oozie-logs-test-1.txt
results.txt
PFA a summary of the results, as well as the oozie logs of 2 separate retention
tests.
Used the same feed definition given in the chain above.
Worked fine for us.
Please update if this doesn't answer the problem, and there's some other test
case that needs to be run.
> Feed evictor deleting more stuff than it should
> -----------------------------------------------
>
> Key: FALCON-321
> URL: https://issues.apache.org/jira/browse/FALCON-321
> Project: Falcon
> Issue Type: Bug
> Reporter: Raghav Kumar Gautam
> Priority: Blocker
> Labels: system-tests
> Attachments: oozie-logs-test-1.txt, oozie-logs-test-2.txt, results.txt
>
>
> In FeedEvictor.java we have:
> {code:java}
> private void deleteParentIfEmpty(FileSystem fs, Path parent, Path
> feedBasePath) throws IOException {
> if (feedBasePath.equals(parent)) {
> LOG.info("Not deleting feed base path:" + parent);
> } else {
> if (fs.getContentSummary(parent).getFileCount() == 0) {
> LOG.info("Parent path: " + parent + " is empty, deleting
> path");
> if (fs.delete(parent, true)) {
> LOG.info("Deleted empty dir: " + parent);
> } else {
> throw new IOException("Unable to delete parent path:" +
> parent);
> }
> deleteParentIfEmpty(fs, parent.getParent(), feedBasePath);
> }
> }
> }
> {code}
> In the fs.getContentSummary(parent).getFileCount() call if the parent has no
> files but has directories then we delete the parent directory. Which is
> incorrect.
> Here is log from falcon-regression's RetentionTest.testRetention(parameters:
> hours, 24, true, daily) :
> {noformat}
> 2014-02-24 15:09:45,034 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Applying retention on
> DATA=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}#META=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksMetaData#STATS=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksStats#TMP=/tmp
> type: instance, Limit: hours(24), timezone: UTC, frequency: hours,
> storageFILESYSTEM
> 2014-02-24 15:09:45,051 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Normalized path : /retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}
> 2014-02-24 15:09:45,123 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Searching for /retention/testFolders/*/*/*/*
> 2014-02-24 15:09:45,486 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Deleted instance :/retention/testFolders/2014/01/21/00
> 2014-02-24 15:09:45,500 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Parent path: /retention/testFolders/2014/01/21 is empty, deleting path
> 2014-02-24 15:09:45,509 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Deleted empty dir: /retention/testFolders/2014/01/21
> 2014-02-24 15:09:45,511 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Parent path: /retention/testFolders/2014/01 is empty, deleting path
> 2014-02-24 15:09:45,517 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Deleted empty dir: /retention/testFolders/2014/01
> 2014-02-24 15:09:45,518 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Parent path: /retention/testFolders/2014 is empty, deleting path
> 2014-02-24 15:09:45,525 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Deleted empty dir: /retention/testFolders/2014
> 2014-02-24 15:09:45,526 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Not deleting feed base path:/retention/testFolders
> {noformat}
> Stacktrace:
> {noformat}
> Failing Oozie Launcher, Main class [org.apache.falcon.retention.FeedEvictor],
> main() threw exception, Unable to delete instance:
> /retention/testFolders/2014/01/21/03
> java.io.IOException: Unable to delete instance:
> /retention/testFolders/2014/01/21/03
> at
> org.apache.falcon.retention.FeedEvictor.deleteInstance(FeedEvictor.java:321)
> at
> org.apache.falcon.retention.FeedEvictor.fileSystemEvictor(FeedEvictor.java:174)
> at org.apache.falcon.retention.FeedEvictor.evictFS(FeedEvictor.java:149)
> at org.apache.falcon.retention.FeedEvictor.evict(FeedEvictor.java:139)
> at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:93)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
