[Discuss] datasets input file in user's repository

2019-03-18 Thread Nadav Har Tzvi
Hi, Just wanna open this up for discussion as it seems we somehow skipped this point. Basically, by now we pretty much have the new datasets APIs in place in the Python SDK and in implementing frameworks. (amaterasu-pyspark, amaterasu-pandas, amaterasu-python) The only question left is regarding t

Re: [Discuss] datasets input file in user's repository

2019-03-18 Thread Yaniv Rodenski
Hi Nadav, I think datasets should be per environment, (for example, it is very common to use different databases for dev/test/prod), so I think that datasets as configurations in Amaterasu should sit under env). Cheers, Yaniv On Tue, Mar 19, 2019 at 5:13 AM Nadav Har Tzvi wrote: > Hi, > > Just

Re: [Discuss] datasets input file in user's repository

2019-03-18 Thread Nadav Har Tzvi
I assumed as much. I will implement a change to the leader to load the data from the env, it will become a part of the PR for Amaterasu-46. Cheers, Nadav On Tue, 19 Mar 2019 at 01:36, Yaniv Rodenski wrote: > Hi Nadav, > > I think datasets should be per environment, (for example, it is very co