[
https://issues.apache.org/jira/browse/FALCON-580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119339#comment-14119339
]
Srikanth Sundarrajan commented on FALCON-580:
---------------------------------------------
{quote}
How will eviction ever work? It will be broken or at best will not evict
anything.
{quote}
When data is not time based, eviction based on time isn't supposed to do
anything, if and when we implement age based eviction (based on directory
creation /modification time stamp), this would evict as appropriate.
{quote}
DistCp becomes very inefficient as time and data grows since it now needs to
check every file on each side for copy.
{quote}
True, but can't be reason disallowing non-time based pattern on feed paths, but
what if this is a snapshot and needs to be constantly synced up (which is
essentially non-time based). In either case this is a feature to be used
ideally for reasonably sized snapshots which are constantly mutated.
A flag would help to formalize this, but flag can be inferred with a schema
change simply by evaluating if time pattern is present in the path or not. But
dont know what the equivalent would be for hcat table based feed.
> Mandate date pattern for the feed path in the xsd
> -------------------------------------------------
>
> Key: FALCON-580
> URL: https://issues.apache.org/jira/browse/FALCON-580
> Project: Falcon
> Issue Type: Bug
> Reporter: Sowmya Ramesh
>
> The granularity of date pattern in the feed path should be atleast that of a
> frequency of a feed. This should be mandated in the feed xsd.
> e.g.:
> {noformat}
> Valid format: <location type="data"
> path="/hdfsDataLocation/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> Invalid format: <location type="data" path="/hdfsDataLocation"/>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
