Re: [VOTE] Apache CarbonData 1.4.1(RC2) release

[Venkata Gollamudi]

+1

Regards,
Venkata Ramana Gollamudi

On Tue, Aug 14, 2018, 11:05 Kunal Kapoor  wrote:

> +1
>
> Regards
> Kunal Kapoor
>
> On Fri, Aug 10, 2018, 8:14 AM Ravindra Pesala 
> wrote:
>
> > Hi
> >
> >
> > I submit the Apache CarbonData 1.4.1 (RC2) for your vote.
> >
> >
> > 1.Release Notes:
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320220&version=12343148
> >
> > Some key features and improvements in this release:
> >
> >1. Supported Local dictionary to improve IO and query performance.
> >2. Improved and stabilized Bloom filter datamap.
> >3. Supported left outer join MV datamap(Alpha feature)
> >4. Supported driver min max caching for specified columns and
> >segregate block and blocklet cache.
> >5. Support Flat folder structure in carbon to maintain the same folder
> >structure as Hive.
> >6. Supported S3 read and write on carbondata files
> >7. Support projection push down for struct data type.
> >8. Improved complex datatypes compression and performance through
> >adaptive encoding.
> >9. Many Bug fixes and stabilized carbondata.
> >
> >
> >  2. The tag to be voted upon : apache-carbondata-1.4.1.rc2(commit:
> > a17db2439aa51f6db7da293215f9732ffb200bd9)
> >
> >
> >
> https://github.com/apache/carbondata/releases/tag/apache-carbondata-1.4.1-rc2
> >
> >
> > 3. The artifacts to be voted on are located here:
> >
> > https://dist.apache.org/repos/dist/dev/carbondata/1.4.1-rc2/
> >
> >
> > 4. A staged Maven repository is available for review at:
> >
> >
> https://repository.apache.org/content/repositories/orgapachecarbondata-1032
> >
> >
> > 5. Release artifacts are signed with the following key:
> >
> > *https://people.apache.org/keys/committer/ravipesala.asc
> > <
> >
> https://link.getmailspring.com/link/1524823736.local-38e60b2f-d8f4-v1.2.1-7e744...@getmailspring.com/9?redirect=https%3A%2F%2Fpeople.apache.org%2Fkeys%2Fcommitter%2Fravipesala.asc&recipient=ZGV2QGNhcmJvbmRhdGEuYXBhY2hlLm9yZw%3D%3D
> > >*
> >
> >
> > Please vote on releasing this package as Apache CarbonData 1.4.1,  The
> vote
> >
> > will be open for the next 72 hours and passes if a majority of
> >
> > at least three +1 PMC votes are cast.
> >
> >
> > [ ] +1 Release this package as Apache CarbonData 1.4.1
> >
> > [ ] 0 I don't feel strongly about it, but I'm okay with the release
> >
> > [ ] -1 Do not release this package because...
> >
> >
> > Regards,
> > Ravindra.
> >
>

Re: [VOTE] Apache CarbonData 1.4.1(RC2) release

[Kunal Kapoor]

+1

Regards
Kunal Kapoor

On Fri, Aug 10, 2018, 8:14 AM Ravindra Pesala  wrote:

> Hi
>
>
> I submit the Apache CarbonData 1.4.1 (RC2) for your vote.
>
>
> 1.Release Notes:
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320220&version=12343148
>
> Some key features and improvements in this release:
>
>1. Supported Local dictionary to improve IO and query performance.
>2. Improved and stabilized Bloom filter datamap.
>3. Supported left outer join MV datamap(Alpha feature)
>4. Supported driver min max caching for specified columns and
>segregate block and blocklet cache.
>5. Support Flat folder structure in carbon to maintain the same folder
>structure as Hive.
>6. Supported S3 read and write on carbondata files
>7. Support projection push down for struct data type.
>8. Improved complex datatypes compression and performance through
>adaptive encoding.
>9. Many Bug fixes and stabilized carbondata.
>
>
>  2. The tag to be voted upon : apache-carbondata-1.4.1.rc2(commit:
> a17db2439aa51f6db7da293215f9732ffb200bd9)
>
>
> https://github.com/apache/carbondata/releases/tag/apache-carbondata-1.4.1-rc2
>
>
> 3. The artifacts to be voted on are located here:
>
> https://dist.apache.org/repos/dist/dev/carbondata/1.4.1-rc2/
>
>
> 4. A staged Maven repository is available for review at:
>
> https://repository.apache.org/content/repositories/orgapachecarbondata-1032
>
>
> 5. Release artifacts are signed with the following key:
>
> *https://people.apache.org/keys/committer/ravipesala.asc
> <
> https://link.getmailspring.com/link/1524823736.local-38e60b2f-d8f4-v1.2.1-7e744...@getmailspring.com/9?redirect=https%3A%2F%2Fpeople.apache.org%2Fkeys%2Fcommitter%2Fravipesala.asc&recipient=ZGV2QGNhcmJvbmRhdGEuYXBhY2hlLm9yZw%3D%3D
> >*
>
>
> Please vote on releasing this package as Apache CarbonData 1.4.1,  The vote
>
> will be open for the next 72 hours and passes if a majority of
>
> at least three +1 PMC votes are cast.
>
>
> [ ] +1 Release this package as Apache CarbonData 1.4.1
>
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
>
> [ ] -1 Do not release this package because...
>
>
> Regards,
> Ravindra.
>

Re: [VOTE] Apache CarbonData 1.4.1(RC2) release

[manish gupta]

+1

Regards
Manish Gupta

On Mon, 13 Aug 2018 at 8:29 PM, Kumar Vishal 
wrote:

> +1
> Regards
> Kumar Vishal
>
> On Fri, 10 Aug 2018 at 08:14, Ravindra Pesala 
> wrote:
>
> > Hi
> >
> >
> > I submit the Apache CarbonData 1.4.1 (RC2) for your vote.
> >
> >
> > 1.Release Notes:
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320220&version=12343148
> >
> > Some key features and improvements in this release:
> >
> >1. Supported Local dictionary to improve IO and query performance.
> >2. Improved and stabilized Bloom filter datamap.
> >3. Supported left outer join MV datamap(Alpha feature)
> >4. Supported driver min max caching for specified columns and
> >segregate block and blocklet cache.
> >5. Support Flat folder structure in carbon to maintain the same folder
> >structure as Hive.
> >6. Supported S3 read and write on carbondata files
> >7. Support projection push down for struct data type.
> >8. Improved complex datatypes compression and performance through
> >adaptive encoding.
> >9. Many Bug fixes and stabilized carbondata.
> >
> >
> >  2. The tag to be voted upon : apache-carbondata-1.4.1.rc2(commit:
> > a17db2439aa51f6db7da293215f9732ffb200bd9)
> >
> >
> >
> https://github.com/apache/carbondata/releases/tag/apache-carbondata-1.4.1-rc2
> >
> >
> > 3. The artifacts to be voted on are located here:
> >
> > https://dist.apache.org/repos/dist/dev/carbondata/1.4.1-rc2/
> >
> >
> > 4. A staged Maven repository is available for review at:
> >
> >
> https://repository.apache.org/content/repositories/orgapachecarbondata-1032
> >
> >
> > 5. Release artifacts are signed with the following key:
> >
> > *https://people.apache.org/keys/committer/ravipesala.asc
> > <
> >
> https://link.getmailspring.com/link/1524823736.local-38e60b2f-d8f4-v1.2.1-7e744...@getmailspring.com/9?redirect=https%3A%2F%2Fpeople.apache.org%2Fkeys%2Fcommitter%2Fravipesala.asc&recipient=ZGV2QGNhcmJvbmRhdGEuYXBhY2hlLm9yZw%3D%3D
> > >*
> >
> >
> > Please vote on releasing this package as Apache CarbonData 1.4.1,  The
> vote
> >
> > will be open for the next 72 hours and passes if a majority of
> >
> > at least three +1 PMC votes are cast.
> >
> >
> > [ ] +1 Release this package as Apache CarbonData 1.4.1
> >
> > [ ] 0 I don't feel strongly about it, but I'm okay with the release
> >
> > [ ] -1 Do not release this package because...
> >
> >
> > Regards,
> > Ravindra.
> >
>

?????? can we add partition or split partition on range partitioned tables

[??????]

ok, it works. thanks




--  --
??: "Lu Cao";
: 2018??8??14??(??) 10:27
??: "dev";

: Re: can we add partition or split partition on range partitioned tables



Just add the new boundary '400' like the list partition. CarbonData will
auto merge the range list and split the data file(if exist...)

ALTER TABLE [db_name].table_name ADD PARTITION('400')

If you want to add a value less than the max boundary(e.g.  150), then you
should use 'alter table split partition' statement.
You can check 'TestAlterPartitionTable.scala' for more examples.

Regards,
Lionel

On Tue, Aug 14, 2018 at 10:15 AM, ?? <251922...@qq.com> wrote:

> Thanks for your reply.
> I have refered this document before, But I can not find how to add
> partition for range table, I don't know how to assign
> new range info.
> for example, a table have had a pange info "'RANGE_INFO'='1, 100, 200,
> 300'", then I want to add a partition with range between 300 and 400, how
> to do it?
>
>
> I checked the carbondata code, I can only find add partition to List table.
>
>
>
>
>
>
>
>
> --  --
> ??: "Lu Cao";
> : 2018??8??14??(??) 9:34
> ??: "dev";
>
> : Re: can we add partition or split partition on range partitioned tables
>
>
>
> Hi,
> By default the value larger than max boundary you defined will fall into
> the default partition. You can also use ALTER TABLE statement to add a
> partition.
> Pls refer to the documents in carbondata official site
> http://carbondata.apache.org/data-management-on-carbondata.html
>
> Regards,
> Lionel
>
> On Mon, Aug 13, 2018 at 2:43 PM, ?? <251922...@qq.com> wrote:
>
> > hi community,
> >  carbon have a range partition feature, for example??
> > CREATE TABLE test_range (
> > _col_a int)
> > partitioned by (productid int)
> > STORED BY 'carbondata'
> > TBLPROPERTIES ('partition_type'='RANGE', 'RANGE_INFO'='1, 100, 200, 300')
> >
> >
> >
> > can we add a partition which store productid between  300 and 400,
> > if we can, how to write this add partition sql?
>

Re: can we add partition or split partition on range partitioned tables

[Lu Cao]

Just add the new boundary '400' like the list partition. CarbonData will
auto merge the range list and split the data file(if exist...)

ALTER TABLE [db_name].table_name ADD PARTITION('400')

If you want to add a value less than the max boundary(e.g.  150), then you
should use 'alter table split partition' statement.
You can check 'TestAlterPartitionTable.scala' for more examples.

Regards,
Lionel

On Tue, Aug 14, 2018 at 10:15 AM, 喜之郎 <251922...@qq.com> wrote:

> Thanks for your reply.
> I have refered this document before, But I can not find how to add
> partition for range table, I don't know how to assign
> new range info.
> for example, a table have had a pange info "'RANGE_INFO'='1, 100, 200,
> 300'", then I want to add a partition with range between 300 and 400, how
> to do it?
>
>
> I checked the carbondata code, I can only find add partition to List table.
>
>
>
>
>
>
>
>
> -- 原始邮件 --
> 发件人: "Lu Cao";
> 发送时间: 2018年8月14日(星期二) 上午9:34
> 收件人: "dev";
>
> 主题: Re: can we add partition or split partition on range partitioned tables
>
>
>
> Hi,
> By default the value larger than max boundary you defined will fall into
> the default partition. You can also use ALTER TABLE statement to add a
> partition.
> Pls refer to the documents in carbondata official site
> http://carbondata.apache.org/data-management-on-carbondata.html
>
> Regards,
> Lionel
>
> On Mon, Aug 13, 2018 at 2:43 PM, 喜之郎 <251922...@qq.com> wrote:
>
> > hi community,
> >  carbon have a range partition feature, for example：
> > CREATE TABLE test_range (
> > _col_a int)
> > partitioned by (productid int)
> > STORED BY 'carbondata'
> > TBLPROPERTIES ('partition_type'='RANGE', 'RANGE_INFO'='1, 100, 200, 300')
> >
> >
> >
> > can we add a partition which store productid between  300 and 400,
> > if we can, how to write this add partition sql?
>

?????? can we add partition or split partition on range partitioned tables

[??????]

Thanks for your reply.
I have refered this document before, But I can not find how to add partition 
for range table, I don't know how to assign 
new range info.
for example, a table have had a pange info "'RANGE_INFO'='1, 100, 200, 300'", 
then I want to add a partition with range between 300 and 400, how to do it?


I checked the carbondata code, I can only find add partition to List table.








--  --
??: "Lu Cao";
: 2018??8??14??(??) 9:34
??: "dev";

: Re: can we add partition or split partition on range partitioned tables



Hi,
By default the value larger than max boundary you defined will fall into
the default partition. You can also use ALTER TABLE statement to add a
partition.
Pls refer to the documents in carbondata official site
http://carbondata.apache.org/data-management-on-carbondata.html

Regards,
Lionel

On Mon, Aug 13, 2018 at 2:43 PM, ?? <251922...@qq.com> wrote:

> hi community,
>  carbon have a range partition feature, for example??
> CREATE TABLE test_range (
> _col_a int)
> partitioned by (productid int)
> STORED BY 'carbondata'
> TBLPROPERTIES ('partition_type'='RANGE', 'RANGE_INFO'='1, 100, 200, 300')
>
>
>
> can we add a partition which store productid between  300 and 400,
> if we can, how to write this add partition sql?

Re: can we add partition or split partition on range partitioned tables

[Lu Cao]

Hi,
By default the value larger than max boundary you defined will fall into
the default partition. You can also use ALTER TABLE statement to add a
partition.
Pls refer to the documents in carbondata official site
http://carbondata.apache.org/data-management-on-carbondata.html

Regards,
Lionel

On Mon, Aug 13, 2018 at 2:43 PM, 喜之郎 <251922...@qq.com> wrote:

> hi community,
>  carbon have a range partition feature, for example：
> CREATE TABLE test_range (
> _col_a int)
> partitioned by (productid int)
> STORED BY 'carbondata'
> TBLPROPERTIES ('partition_type'='RANGE', 'RANGE_INFO'='1, 100, 200, 300')
>
>
>
> can we add a partition which store productid between  300 and 400,
> if we can, how to write this add partition sql?

Re: [VOTE] Apache CarbonData 1.4.1(RC2) release

[Kumar Vishal]

+1
Regards
Kumar Vishal

On Fri, 10 Aug 2018 at 08:14, Ravindra Pesala  wrote:

> Hi
>
>
> I submit the Apache CarbonData 1.4.1 (RC2) for your vote.
>
>
> 1.Release Notes:
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320220&version=12343148
>
> Some key features and improvements in this release:
>
>1. Supported Local dictionary to improve IO and query performance.
>2. Improved and stabilized Bloom filter datamap.
>3. Supported left outer join MV datamap(Alpha feature)
>4. Supported driver min max caching for specified columns and
>segregate block and blocklet cache.
>5. Support Flat folder structure in carbon to maintain the same folder
>structure as Hive.
>6. Supported S3 read and write on carbondata files
>7. Support projection push down for struct data type.
>8. Improved complex datatypes compression and performance through
>adaptive encoding.
>9. Many Bug fixes and stabilized carbondata.
>
>
>  2. The tag to be voted upon : apache-carbondata-1.4.1.rc2(commit:
> a17db2439aa51f6db7da293215f9732ffb200bd9)
>
>
> https://github.com/apache/carbondata/releases/tag/apache-carbondata-1.4.1-rc2
>
>
> 3. The artifacts to be voted on are located here:
>
> https://dist.apache.org/repos/dist/dev/carbondata/1.4.1-rc2/
>
>
> 4. A staged Maven repository is available for review at:
>
> https://repository.apache.org/content/repositories/orgapachecarbondata-1032
>
>
> 5. Release artifacts are signed with the following key:
>
> *https://people.apache.org/keys/committer/ravipesala.asc
> <
> https://link.getmailspring.com/link/1524823736.local-38e60b2f-d8f4-v1.2.1-7e744...@getmailspring.com/9?redirect=https%3A%2F%2Fpeople.apache.org%2Fkeys%2Fcommitter%2Fravipesala.asc&recipient=ZGV2QGNhcmJvbmRhdGEuYXBhY2hlLm9yZw%3D%3D
> >*
>
>
> Please vote on releasing this package as Apache CarbonData 1.4.1,  The vote
>
> will be open for the next 72 hours and passes if a majority of
>
> at least three +1 PMC votes are cast.
>
>
> [ ] +1 Release this package as Apache CarbonData 1.4.1
>
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
>
> [ ] -1 Do not release this package because...
>
>
> Regards,
> Ravindra.
>

Re: How to look up date segment details in carbon without partition.

[Liang Chen]

Hi 

In Carbondata system, the segment concept may be different with other
system.
One data load is one segment for carbondata.

Actually, carbondata currently support partition with global sort also, you
can use date as partition column to check data size for under each partition
folder.

Regards
Liang

carbondata-newuser wrote
> It seems carbondata not recommend use partitionby and partitionby is not
> supported in global sort scope.
> It is very conveniently to look up how many date partition(along with the
> partition size every day) already exists in hive(save as parquet).
> In carbondata I add the date column to first sort columns in order to
> using
> global sort scope.
> But how can I look up segment corresponding date and size in carbondata.
>  
> 
> 
> 
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/





--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [VOTE] Apache CarbonData 1.4.1(RC2) release

[Liang Chen]

Hi 

+1. many good improvements and bug fixs.

Regards
Liang 


ravipesala wrote
> Hi
> 
> 
> I submit the Apache CarbonData 1.4.1 (RC2) for your vote.
> 
> 
> 1.Release Notes:
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320220&version=12343148
> 
> Some key features and improvements in this release:
> 
>1. Supported Local dictionary to improve IO and query performance.
>2. Improved and stabilized Bloom filter datamap.
>3. Supported left outer join MV datamap(Alpha feature)
>4. Supported driver min max caching for specified columns and
>segregate block and blocklet cache.
>5. Support Flat folder structure in carbon to maintain the same folder
>structure as Hive.
>6. Supported S3 read and write on carbondata files
>7. Support projection push down for struct data type.
>8. Improved complex datatypes compression and performance through
>adaptive encoding.
>9. Many Bug fixes and stabilized carbondata.
> 
> 
>  2. The tag to be voted upon : apache-carbondata-1.4.1.rc2(commit:
> a17db2439aa51f6db7da293215f9732ffb200bd9)
> 
> https://github.com/apache/carbondata/releases/tag/apache-carbondata-1.4.1-rc2
> 
> 
> 3. The artifacts to be voted on are located here:
> 
> https://dist.apache.org/repos/dist/dev/carbondata/1.4.1-rc2/
> 
> 
> 4. A staged Maven repository is available for review at:
> 
> https://repository.apache.org/content/repositories/orgapachecarbondata-1032
> 
> 
> 5. Release artifacts are signed with the following key:
> 
> *https://people.apache.org/keys/committer/ravipesala.asc
>  1524823736.local-38e60b2f-d8f4-v1.2.1-7e7447b6@

> /9?redirect=https%3A%2F%2Fpeople.apache.org%2Fkeys%2Fcommitter%2Fravipesala.asc&recipient=ZGV2QGNhcmJvbmRhdGEuYXBhY2hlLm9yZw%3D%3D>*
> 
> 
> Please vote on releasing this package as Apache CarbonData 1.4.1,  The
> vote
> 
> will be open for the next 72 hours and passes if a majority of
> 
> at least three +1 PMC votes are cast.
> 
> 
> [ ] +1 Release this package as Apache CarbonData 1.4.1
> 
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
> 
> [ ] -1 Do not release this package because...
> 
> 
> Regards,
> Ravindra.





--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Discussion] Propose to upgrade the version of integration/presto from 0.187 to 0.206

[sujith chacko]

+1 for presto 0.207/0.208 upgradation.
 As liang mentioned there are some memory issues related patches as well
there  are some security fixes in 0.207 version
Hope the patches are present in presto 0.208 version also.

Thanks,
Sujith

On Mon, 13 Aug 2018 at 2:38 PM, Liang Chen  wrote:

> Hi
>
> Just i checked 0.207 and 0.208, there are fixed many memory issues, so
> propose to upgrade to 0.208 for Apache CarbonData 1.5.0 integration.
>
> Regards
> Liang
>
>
> bhavya411 wrote
> > Hi Dev,
> >
> > Yes, we should definitely go for the 0.206 upgrade for Presto as we are
> > now
> > using the dictionary_aggregation feature for optimization. The other bug
> > fixes are also important for carbondata integration.
> > However, they have changed the connector interface as well, so we might
> > need to change our interface accordingly.
> >
> > Thanks and regards
> > Bhavya
> >
> > On Tue, Jul 24, 2018 at 2:11 PM, Liang Chen <
>
> > chenliang6136@
>
> > > wrote:
> >
> >> Hi Dev
> >>
> >> The presto community already released 0.206 last week (refer the detail
> >> at
> >> https://prestodb.io/docs/current/release/release-0.206.html),  this
> >> release
> >> fixed many issues, so propose Apache CarbonData community to upgrade to
> >> the
> >> latest presto version for carbondata integration.
> >>
> >> please provide your opinion.
> >>
> >> Regards
> >> Liang
> >>
> >
> >
> >
> > --
> > *Bhavya Aggarwal*
> > CTO & Partner
> > Knoldus Inc. ;
> > +91-9910483067
> > Canada - USA - India - Singapore
> > ;
> > ;
> > ;
> > ;
>
>
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

Re: [Discussion] Propose to upgrade the version of integration/presto from 0.187 to 0.206

[Liang Chen]

Hi

Just i checked 0.207 and 0.208, there are fixed many memory issues, so
propose to upgrade to 0.208 for Apache CarbonData 1.5.0 integration.

Regards
Liang


bhavya411 wrote
> Hi Dev,
> 
> Yes, we should definitely go for the 0.206 upgrade for Presto as we are
> now
> using the dictionary_aggregation feature for optimization. The other bug
> fixes are also important for carbondata integration.
> However, they have changed the connector interface as well, so we might
> need to change our interface accordingly.
> 
> Thanks and regards
> Bhavya
> 
> On Tue, Jul 24, 2018 at 2:11 PM, Liang Chen <

> chenliang6136@

> > wrote:
> 
>> Hi Dev
>>
>> The presto community already released 0.206 last week (refer the detail
>> at
>> https://prestodb.io/docs/current/release/release-0.206.html),  this
>> release
>> fixed many issues, so propose Apache CarbonData community to upgrade to
>> the
>> latest presto version for carbondata integration.
>>
>> please provide your opinion.
>>
>> Regards
>> Liang
>>
> 
> 
> 
> -- 
> *Bhavya Aggarwal*
> CTO & Partner
> Knoldus Inc. ;
> +91-9910483067
> Canada - USA - India - Singapore
> ;
> ;
> ;
> ;





--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/