Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2379#discussion_r196636059
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java
---
@@ -289,6 +289,12 @@ public void putDouble(int rowId, double value) {
@Override
public void putBytes(int rowId, byte[] bytes) {
+ // rowId * 4 represents the length of L in LV
+ if (bytes.length > (Integer.MAX_VALUE - totalLength - rowId * 4)) {
--- End diff --
I come across a new idea:
During parsing/converting, we can calculate #numberOfRowsPerPage *
#currentCharacterLength, if it is larger than 2GB, dataload will fail.
Notice that the #numberOfRowsPerPage is specified by user through
configurations.
If this is OK, I'll implementation in the future not this PR.
---
