Hey Arun,
I kinda feel like the datastores yaml is somewhat obscure. I propose the
following structure.
Instead of
datasets:
hive:
- key: transactions
uri: /user/somepath
format: parquet
database: transations_daily
table: transx
- key: second_transactions
uri: /seconduser/somepath
format: avro
database: transations_monthly
table: avro_table
file:
- key: users
uri: s3://filestore
format: parquet
mode: overwrite
I would have
datasets:
- key: transactions
uri: /user/somepath
format: parquet
database: transations_daily
table: transx
type: hive
- key: second_transactions
uri: /seconduser/somepath
format: avro
database: transations_monthly
table: avro_table
type: hive
- key: users
uri: s3://filestore
format: parquet
mode: overwrite
type: file
In my opinion it is more straightforward and uniform. I think it is also
more straightforward code-wise.
What do you think?
Cheers,
Nadav
On Mon, 14 Jan 2019 at 00:57, Yaniv Rodenski <[email protected]> wrote:
> Hi Arun,
>
> I've added my comments to the PR, but good call, I agree @Nadav Har Tzvi
> <[email protected]> should at least review as you both need to
> maintain compatible APIs.
>
> Cheers,
> Yaniv
>
> On Sun, Jan 13, 2019 at 10:21 PM Arun Manivannan <[email protected]> wrote:
>
>> Hi Guy, Yaniv and Nadiv,
>>
>> This PR <https://github.com/apache/incubator-amaterasu/pull/39> just
>> captures part of the issue - the datasets.yaml, ConfigManager and the
>> testcases. The Integration with the AmaContext is yet to be done but I
>> would like to get your thoughts on the implementation.
>>
>> Guy - Would it be okay if you could help throw some light on the syntax
>> and
>> the idiomatic part of Kotlin itself. Newbie here.
>>
>> Cheers,
>> Arun
>>
>> On Fri, Oct 12, 2018 at 7:15 PM Yaniv Rodenski (JIRA) <[email protected]>
>> wrote:
>>
>> > Yaniv Rodenski created AMATERASU-52:
>> > ---------------------------------------
>> >
>> > Summary: Implement AmaContext.datastores
>> > Key: AMATERASU-52
>> > URL:
>> https://issues.apache.org/jira/browse/AMATERASU-52
>> > Project: AMATERASU
>> > Issue Type: Task
>> > Reporter: Yaniv Rodenski
>> > Assignee: Arun Manivannan
>> > Fix For: 0.2.1-incubating
>> >
>> >
>> > AmaContext.datastores should contain the data from datastores.yaml
>> >
>> >
>> >
>> > --
>> > This message was sent by Atlassian JIRA
>> > (v7.6.3#76005)
>> >
>>
>
>
> --
> Yaniv Rodenski
>
> +61 477 778 405
> [email protected]
>
>
