[
https://issues.apache.org/jira/browse/HDFS-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031762#comment-14031762
]
Allen Wittenauer commented on HDFS-6382:
----------------------------------------
Let's take a step back and give a more concrete example: /tmp
/tmp is the plague of Hadoop operations teams everywhere. Between Hive and Pig
leaving bits hanging around because the client can't properly handle signals to
crunch that by default just leaves (seemingly) as much crap laying around that
it possibly can because someone somewhere might want to debug it someday, /tmp
is the cesspool of HDFS. Every competent admin ends up writing some sort of
auto-/tmp-cleaner because of these issues. At scale, /tmp can have hundreds of
TB and millions of objects in it in less than 24 hours. It sucks.
One eventually gets to the point that the auto cleaner job is now running
hourly just so /tmp doesn't overrun the rest of HDFS. Because these run
outside of HDFS, they are slow and tedious and generally fall in the lap of
teams that don't do Java so end up doing all sorts of squirrely things to make
these jobs work. This also sucks.
Now, I can see why using an MR job is appealing (easy!), but it isn't very
effective. For one, we've already been here once and the result was distch.
Hell, there was a big fight just to get distch written and that--years and
years later!--still isn't documented because of how slow it works. Throw in
directories like /tmp that simply have WAY too much churn and one can see that
depending upon MR to work here just isn't viable. Plus, in the places that
need this the most, one has to deal with getting what essentially becomes a
critical part of uptime getting scheduled, competing with all of the other
things running.... and, to remind you, to just delete files. It's sort of
ridiculous to require YARN running for what is fundamentally a file system
problem. It simply doesn't work in the real world.
While at Hadoop Summit, a bunch of us sat around a table and were talking about
this issue with regards specifically to /tmp. (We didn't know about this JIRA,
BTW.) The solution we came up with was basically a service that would bootstrap
by reading fsimage and then reading the edits stream by sending the audit
information to Kafka. One of the big advantages of this is that we can get
near real-time updates of the parts of the file system we care need to operate
on. Since we only care about a subsection of the file system, the memory
requirements are significantly lower and it might be possible to coalesce
deletes in a smart way to cut back on RPCs. I suspect it wouldn't be hard to
generalize this type of solution to handle multiple user cases. But for me,
this is critical admin functionality that HDFS needs desperately and throwing
the the problem to MR just isn't workable.
> HDFS File/Directory TTL
> -----------------------
>
> Key: HDFS-6382
> URL: https://issues.apache.org/jira/browse/HDFS-6382
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
> Affects Versions: 2.4.0
> Reporter: Zesheng Wu
> Assignee: Zesheng Wu
> Attachments: HDFS-TTL-Design -2.pdf, HDFS-TTL-Design.pdf
>
>
> In production environment, we always have scenario like this, we want to
> backup files on hdfs for some time and then hope to delete these files
> automatically. For example, we keep only 1 day's logs on local disk due to
> limited disk space, but we need to keep about 1 month's logs in order to
> debug program bugs, so we keep all the logs on hdfs and delete logs which are
> older than 1 month. This is a typical scenario of HDFS TTL. So here we
> propose that hdfs can support TTL.
> Following are some details of this proposal:
> 1. HDFS can support TTL on a specified file or directory
> 2. If a TTL is set on a file, the file will be deleted automatically after
> the TTL is expired
> 3. If a TTL is set on a directory, the child files and directories will be
> deleted automatically after the TTL is expired
> 4. The child file/directory's TTL configuration should override its parent
> directory's
> 5. A global configuration is needed to configure that whether the deleted
> files/directories should go to the trash or not
> 6. A global configuration is needed to configure that whether a directory
> with TTL should be deleted when it is emptied by TTL mechanism or not.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
