[kudu-CR](branch-1.17.x) KUDU-3483 Fix flushing data in batch when table schema changed
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20622 ) Change subject: KUDU-3483 Fix flushing data in batch when table schema changed .. KUDU-3483 Fix flushing data in batch when table schema changed In auto_flush_background or manual_flush mode, applying an operation firstly inserts the row into the buffer. When the buffer is full or function flush() is called, it tries to flush multiple rows into Kudu server. Firstly, it groups this data according to the tablet id as a batch. A batch may contains multiple rows which belong to the same tablet. Then a batch will encode into bytes. At this time, it reads the table schema of the first row and decides the format of the data. If two rows have different schema but belongs to the same tablet, which maybe because of altering the table between inserting two rows, it causes array index outof index bound exception. This patch will validate the schema of multiple rows which belong to the same tablet. If the schema is different, it puts them into the different groups as different batches. Change-Id: Ie6501962b32814d121f180b2942999c402d927db Reviewed-on: http://gerrit.cloudera.org:8080/19949 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin (cherry picked from commit c9e6e36a742c1164bf20a1913a50b9bd03807ed7) Reviewed-on: http://gerrit.cloudera.org:8080/20622 Tested-by: Alexey Serbin Reviewed-by: Wang Xixu <1450306...@qq.com> Reviewed-by: Yifan Zhang --- M java/kudu-client/src/main/java/org/apache/kudu/Schema.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java A java/kudu-client/src/test/java/org/TestSchema.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAlterTable.java 4 files changed, 366 insertions(+), 19 deletions(-) Approvals: Alexey Serbin: Verified Wang Xixu: Looks good to me, but someone else must approve Yifan Zhang: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.17.x Gerrit-MessageType: merged Gerrit-Change-Id: Ie6501962b32814d121f180b2942999c402d927db Gerrit-Change-Number: 20622 Gerrit-PatchSet: 2 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Yingchun Lai
[kudu-CR](branch-1.17.x) KUDU-3483 Fix flushing data in batch when table schema changed
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20622 ) Change subject: KUDU-3483 Fix flushing data in batch when table schema changed .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.17.x Gerrit-MessageType: comment Gerrit-Change-Id: Ie6501962b32814d121f180b2942999c402d927db Gerrit-Change-Number: 20622 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Thu, 26 Oct 2023 06:16:46 + Gerrit-HasComments: No
[kudu-CR](branch-1.17.x) KUDU-3483 Fix flushing data in batch when table schema changed
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/20622 ) Change subject: KUDU-3483 Fix flushing data in batch when table schema changed .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.17.x Gerrit-MessageType: comment Gerrit-Change-Id: Ie6501962b32814d121f180b2942999c402d927db Gerrit-Change-Number: 20622 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Thu, 26 Oct 2023 02:02:12 + Gerrit-HasComments: No
[kudu-CR](branch-1.17.x) KUDU-3483 Fix flushing data in batch when table schema changed
Alexey Serbin has removed a vote on this change. Change subject: KUDU-3483 Fix flushing data in batch when table schema changed .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/20622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.17.x Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ie6501962b32814d121f180b2942999c402d927db Gerrit-Change-Number: 20622 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yingchun Lai
[kudu-CR](branch-1.17.x) KUDU-3483 Fix flushing data in batch when table schema changed
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20622 ) Change subject: KUDU-3483 Fix flushing data in batch when table schema changed .. Patch Set 1: Verified+1 unrelated test failure in RaftConsensusITest.TestSlowFollower (RELEASE) -- To view, visit http://gerrit.cloudera.org:8080/20622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.17.x Gerrit-MessageType: comment Gerrit-Change-Id: Ie6501962b32814d121f180b2942999c402d927db Gerrit-Change-Number: 20622 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Wed, 25 Oct 2023 21:05:43 + Gerrit-HasComments: No
[kudu-CR](branch-1.17.x) KUDU-3483 Fix flushing data in batch when table schema changed
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20622 Change subject: KUDU-3483 Fix flushing data in batch when table schema changed .. KUDU-3483 Fix flushing data in batch when table schema changed In auto_flush_background or manual_flush mode, applying an operation firstly inserts the row into the buffer. When the buffer is full or function flush() is called, it tries to flush multiple rows into Kudu server. Firstly, it groups this data according to the tablet id as a batch. A batch may contains multiple rows which belong to the same tablet. Then a batch will encode into bytes. At this time, it reads the table schema of the first row and decides the format of the data. If two rows have different schema but belongs to the same tablet, which maybe because of altering the table between inserting two rows, it causes array index outof index bound exception. This patch will validate the schema of multiple rows which belong to the same tablet. If the schema is different, it puts them into the different groups as different batches. Change-Id: Ie6501962b32814d121f180b2942999c402d927db Reviewed-on: http://gerrit.cloudera.org:8080/19949 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin (cherry picked from commit c9e6e36a742c1164bf20a1913a50b9bd03807ed7) --- M java/kudu-client/src/main/java/org/apache/kudu/Schema.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java A java/kudu-client/src/test/java/org/TestSchema.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAlterTable.java 4 files changed, 366 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/22/20622/1 -- To view, visit http://gerrit.cloudera.org:8080/20622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.17.x Gerrit-MessageType: newchange Gerrit-Change-Id: Ie6501962b32814d121f180b2942999c402d927db Gerrit-Change-Number: 20622 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Wang Xixu <1450306...@qq.com>
