Moving discussion to datafu dev mailing list. ---------- Forwarded message ---------- From: Matthew Hayes <matthew.terence.ha...@gmail.com> Date: Wed, Feb 5, 2014 at 9:18 AM Subject: Re: Hourglass Input paths To: "dat...@googlegroups.com" <dat...@googlegroups.com>
Hi Abhishek, At the moment the input directory structure is fixed to the yyyy/mm/dd format. This is one area where I'd like to make the system more customizable. I'd like to create an interface that represents the concept of a set of input paths that each are associated with a particular date range, but without being tied to a particular format like yyyy/mm/dd. Then you would have the option of creating a class that implements this interface and providing this class to Hourglass. You could create a class like WeeklyInput for example that assumes yyyy/mm/dd structure but dates only on week boundaries. Or you could create a class like MonthlyInput that uses yyyy/mm. Or one that uses yyyymmdd. The same could be done for the output path. You could even have an input structure that has overlapping date ranges, like "daily/yyyy/mm/dd" and "monthly/yyyy/mm". Hourglass could determine the appropriate files to read based on the desired date range to consume. These are just some ideas. I haven't had time to work on this yet. If you're interested in working on it we would appreciate the help ;) -Matt On Wed, Feb 5, 2014 at 3:06 AM, Abhishek Gayakwad <a.gayak...@gmail.com>wrote: > Hi, > > Hourglass requires input directory structure in yyyy/mm/dd format, Is it > possible to provide custom logic for reading from different directory > structure e.g. yyyymmdd or some other date key. Is it possible to change > the frequency form daily to weekly ? > > Thanks > Abhishek > > -- > You received this message because you are subscribed to the Google Groups > "DataFu" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to datafu+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. >
