GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/2379
[CARBONDATA-2420][32K] Support string longer than 32000 characters
Add a property in creating table 'long_string_columns' to support string
columns that will contains more than 32000 characters.
Inside carbondata, it use an integer instead of short to store the length
of bytes content.
Internally in Carbondata,
1. add a new datatype called `varchar` to represent the long string column
2. add a new encoding called `DIRECT_COMPRESS_VARCHAR` to the archer column
page meta
3. use an integer (previously short) to store the length of bytes content.
4. add 2GB constraint for one column page
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [x] Any interfaces changed?
`Only internal interfaces have been changed`
- [x] Any backward compatibility impacted?
`NO`
- [x] Document update required?
`YES`
- [x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
`Added tests`
- How it is tested? Please attach test report.
`Tested in local machine`
- Is it a performance related change? Please attach the performance
test report.
`NO`
- Any additional information to help reviewers in testing this
change.
`NA`
- [x] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
`NA`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuchuanyin/carbondata 0602_support_long_string
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2379.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2379
----
commit b689d66493521452ff9938415e0d0aa66b56c2c5
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-02T07:17:04Z
Support string longer than 32000 characters
Add a table property 'long_string_columns' in create table DDL that
indicate those columns will contain more than 32000 characters.
Internally in Carbondata,
1. add a new datatype called `text` to represent the long string column
2. add a new encoding called `DIRECT_COMPRESS_TEXT` to the text column
page meta
3. Use an integer (previously short) to store the length of bytes
content.
commit f145c6c60238c400b5db6a6bf2696246b698154a
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-05T12:46:26Z
rename datatype name from text to varchar
commit 4180f8118d1ff90205b0f1567bef2cdfee3a1b62
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-12T12:35:58Z
Add 2GB constraint for one column page
commit 710845b155ed5b7a611a900c70b0d766d80ae48d
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-06-14T12:11:40Z
update tests
----
---
