Re: [DISCUSSION] Presto+Carbon transactional and Non-transactional Write Support
Hi Ajantha, Thanks for the inputs, please check the comments. a) you mentioned, currently creating a table form presto and inserting data will be a non-transactional table. so, to create a transactional table, we still depend on spark? > currently it's dependent on spark, but I'm planning to support create table from presto and remove the dependency I will start once I finish this requirement. b) If spark has created a global sort table and presto inserts data. what will happen? Do we ignore those table properties in presto write? > Currenlty we are ignoring the table properties and considering only the system-level properties. Once we support create table, we can consider these. c) Partition and complex type can be handled as immediate follow up of this as it is a very common feature now a days. > This is already planned, so I will create subtasks in Jira to follow up with these. Thanks, Regards, Akash R -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [DISCUSSION] Presto+Carbon transactional and Non-transactional Write Support
+1 Good to enhance Carbon with Presto engine support. Regards, Venu -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [DISCUSSION] Presto+Carbon transactional and Non-transactional Write Support
+1, I have some suggestions and questions, a) you mentioned, currently creating a table form presto and inserting data will be a non-transactional table. so, to create a transactional table, we still depend on spark ? *I feel we should support transactional table creation with all table properties also from presto (to remove dependency on spark). * b) If spark has created global sort table and presto inserts data. what will happen ? Do we ignore those table properties in presto write ? c) Partition and complex type can be handled as immediate follow up of this as it is a very common feature now a days. Thanks, Ajantha On Mon, 27 Jul, 2020, 1:07 pm Kunal Kapoor, wrote: > +1, > It would be great to have write support from presto > > Thanks > Kunal Kapoor > > On Tue, Jul 14, 2020 at 6:08 PM Akash Nilugal > wrote: > > > Hi Community, > > > > As we know the CarbonDataisan indexed columnar data format for fast > > analytics on big data platforms. So > > we have already integrated with the query engines like spark and even > > presto. Currently with presto we > > only support the querying of carbon data files. But we don’t yet support > > the writing of carbon data files > > through presto engine. > > > > Currentlypresto is integrated with carbondata for reading the carbondata > > files via presto. > > For this, we should be having the store already ready which may be > written > > carbon in spark and the table > > should be hive metastore. So using carbondata connector we are able to > read > > the carbondata files. But we > > cannot create a table or load the data to the table in presto. So it will > > somewhat hectic job to read the > > carbon files, by writing first with other engines. > > > > So here I will be trying to support the transactional load support in > > presto integration for carbon. > > > > I have attached the design document in the Jira, please refer and any > > suggestions or input is most welcome. > > > > https://issues.apache.org/jira/browse/CARBONDATA-3831 > > > > > > Regards, > > Akash R. > > >
[DISCUSSION] Presto+Carbon transactional and Non-transactional Write Support
Hi Community, As we know the CarbonDataisan indexed columnar data format for fast analytics on big data platforms. So we have already integrated with the query engines like spark and even presto. Currently with presto we only support the querying of carbon data files. But we don’t yet support the writing of carbon data files through presto engine. Currentlypresto is integrated with carbondata for reading the carbondata files via presto. For this, we should be having the store already ready which may be written carbon in spark and the table should be hive metastore. So using carbondata connector we are able to read the carbondata files. But we cannot create a table or load the data to the table in presto. So it will somewhat hectic job to read the carbon files, by writing first with other engines. So here I will be trying to support the transactional load support in presto integration for carbon. I have attached the design document in the Jira, please refer and any suggestions or input is most welcome. https://issues.apache.org/jira/browse/CARBONDATA-3831 Regards, Akash R.
Re: [DISCUSSION] Presto+Carbon transactional and Non-transactional Write Support
+ 1 for the proposal, I didn't see design doc in JIRA. Please check. Also once we provide write support, it is better to have carbondata as a separate plugin instead of extending hive. As presto-hive was not meant to have write support and it is mainly meant for query at where it exist. Also hope you cover what all table properties supported in write in the design doc. Thanks, Ajantha On Thu, May 21, 2020 at 8:57 PM Akash Nilugal wrote: > Hi Community, > > As we know the CarbonDataisan indexed columnar data format for fast > analytics on big data platforms. So > we have already integrated with the query engines like spark and even > presto. Currently with presto we > only support the querying of carbon data files. But we don’t yet support > the writing of carbon data files > through presto engine. > > Currentlypresto is integrated with carbondata for reading the carbondata > files via presto. > For this, we should be having the store already ready which may be written > carbon in spark and the table > should be hive metastore. So using carbondata connector we are able to read > the carbondata files. But we > cannot create a table or load the data to the table in presto. So it will > somewhat hectic job to read the > carbon files, by writing first with other engines. > > So here I will be trying to support the transactional load support in > presto integration for carbon. > > I have attached the design document in the Jira, please refer and any > suggestions or input is most welcome. > > https://issues.apache.org/jira/browse/CARBONDATA-3831 > > > Regards, > Akash R. >
