Hi Nadav, I think datasets should be per environment, (for example, it is very common to use different databases for dev/test/prod), so I think that datasets as configurations in Amaterasu should sit under env).
Cheers, Yaniv On Tue, Mar 19, 2019 at 5:13 AM Nadav Har Tzvi <[email protected]> wrote: > Hi, > > Just wanna open this up for discussion as it seems we somehow skipped this > point. > Basically, by now we pretty much have the new datasets APIs in place in the > Python SDK and in implementing frameworks. (amaterasu-pyspark, > amaterasu-pandas, amaterasu-python) > The only question left is regarding the way we get the datasets > definitions. > Currently, we still look up the datasets definitions in the maki file, > under the action's exports. > Do we intend to keep it that way? I assume not as I think that every action > needs access to all defined datasets. > In that case, how will the user submit datasets configuration? Is it > another file next to the maki.yaml? Is it a file that resides in the > environment, e.g. next to the env.yaml? Is it not even a file on its own > but a part of the env.yaml? > Ideas, anyone? > > Let's discuss this please! > > Cheers, > Nadav > -- Yaniv Rodenski +61 477 778 405 [email protected]
