I assumed as much.
I will implement a change to the leader to load the data from the env, it
will become a part of the PR for Amaterasu-46.
Cheers,
Nadav
On Tue, 19 Mar 2019 at 01:36, Yaniv Rodenski wrote:
> Hi Nadav,
>
> I think datasets should be per environment, (for example, it is very common
> to use different databases for dev/test/prod), so I think that datasets as
> configurations in Amaterasu should sit under env).
>
> Cheers,
> Yaniv
>
> On Tue, Mar 19, 2019 at 5:13 AM Nadav Har Tzvi
> wrote:
>
> > Hi,
> >
> > Just wanna open this up for discussion as it seems we somehow skipped
> this
> > point.
> > Basically, by now we pretty much have the new datasets APIs in place in
> the
> > Python SDK and in implementing frameworks. (amaterasu-pyspark,
> > amaterasu-pandas, amaterasu-python)
> > The only question left is regarding the way we get the datasets
> > definitions.
> > Currently, we still look up the datasets definitions in the maki file,
> > under the action's exports.
> > Do we intend to keep it that way? I assume not as I think that every
> action
> > needs access to all defined datasets.
> > In that case, how will the user submit datasets configuration? Is it
> > another file next to the maki.yaml? Is it a file that resides in the
> > environment, e.g. next to the env.yaml? Is it not even a file on its own
> > but a part of the env.yaml?
> > Ideas, anyone?
> >
> > Let's discuss this please!
> >
> > Cheers,
> > Nadav
> >
>
>
> --
> Yaniv Rodenski
>
> +61 477 778 405
> ya...@shinto.io
>
