[kudu-CR] [HIVE-12971] add a hive serde for kudu
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13230 to look at the new patch set (#2). Change subject: [HIVE-12971] add a hive serde for kudu .. [HIVE-12971] add a hive serde for kudu Change-Id: I72daf3c3f06ccf67e3fa3039c4afa20f6c075cba --- M java/gradle/dependencies.gradle A java/kudu-hive-serde/build.gradle A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduBridgeUtils.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduConstants.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduSerDe.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduPredicateAnalyzer.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStorageHandler.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStructField.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStructObjectInspector.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/PartialRowWritable.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/TimestampConverter.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduRecordReader.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduTableInputFormat.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduTableSplit.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/output/HiveKuduOutputFormat.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/output/KuduRecordUpserter.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/utils/objectinspectorutils/OIUtils.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/utils/objectinspectorutils/inpectors/RenameStructFieldsStructObjectInspector.java A java/kudu-hive-serde/src/test/java/org/apache/kudu/hive/serde/kudu/KuduStructObjectInspectorTest.java M java/settings.gradle 20 files changed, 2,565 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/13230/2 -- To view, visit http://gerrit.cloudera.org:8080/13230 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I72daf3c3f06ccf67e3fa3039c4afa20f6c075cba Gerrit-Change-Number: 13230 Gerrit-PatchSet: 2 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] HIVE-12971 Add a Hive serde for Kudu
Clemens Valiente has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13231 Change subject: HIVE-12971 Add a Hive serde for Kudu .. HIVE-12971 Add a Hive serde for Kudu Change-Id: Ibdc5bdf671a29b566cfb8bc790a71e436ec31693 --- M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduBridgeUtils.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduConstants.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStorageHandler.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduTableInputFormat.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/output/KuduRecordUpserter.java 5 files changed, 61 insertions(+), 87 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/13231/1 -- To view, visit http://gerrit.cloudera.org:8080/13231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibdc5bdf671a29b566cfb8bc790a71e436ec31693 Gerrit-Change-Number: 13231 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente
[kudu-CR] HIVE-12971 Add a Hive serde for Kudu
Clemens Valiente has abandoned this change. ( http://gerrit.cloudera.org:8080/13231 ) Change subject: HIVE-12971 Add a Hive serde for Kudu .. Abandoned Duplicate, see https://gerrit.cloudera.org/#/c/13230/ -- To view, visit http://gerrit.cloudera.org:8080/13231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ibdc5bdf671a29b566cfb8bc790a71e436ec31693 Gerrit-Change-Number: 13231 Gerrit-PatchSet: 2 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] HIVE-12971 Add a Hive serde for Kudu
Clemens Valiente has abandoned this change. ( http://gerrit.cloudera.org:8080/13228 ) Change subject: HIVE-12971 Add a Hive serde for Kudu .. Abandoned not fixing kudu-mapreduce further -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 8 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] HIVE-12971 Add a Hive serde for Kudu
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13231 to look at the new patch set (#2). Change subject: HIVE-12971 Add a Hive serde for Kudu .. HIVE-12971 Add a Hive serde for Kudu Change-Id: Ibdc5bdf671a29b566cfb8bc790a71e436ec31693 --- M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduBridgeUtils.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduConstants.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStorageHandler.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduTableInputFormat.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/output/KuduRecordUpserter.java 5 files changed, 61 insertions(+), 87 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/13231/2 -- To view, visit http://gerrit.cloudera.org:8080/13231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibdc5bdf671a29b566cfb8bc790a71e436ec31693 Gerrit-Change-Number: 13231 Gerrit-PatchSet: 2 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] HIVE-12971 Add a Hive serde for Kudu
Hello Kudu Jenkins, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13228 to look at the new patch set (#8). Change subject: HIVE-12971 Add a Hive serde for Kudu .. HIVE-12971 Add a Hive serde for Kudu Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 --- M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduBridgeUtils.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduConstants.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStorageHandler.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduTableInputFormat.java M java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/output/KuduRecordUpserter.java 5 files changed, 61 insertions(+), 87 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/28/13228/8 -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 8 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [HIVE-12971] add a hive serde for kudu
Clemens Valiente has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13230 Change subject: [HIVE-12971] add a hive serde for kudu .. [HIVE-12971] add a hive serde for kudu Change-Id: I72daf3c3f06ccf67e3fa3039c4afa20f6c075cba --- M java/gradle/dependencies.gradle A java/kudu-hive-serde/build.gradle A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduBridgeUtils.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduConstants.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/HiveKuduSerDe.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduPredicateAnalyzer.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStorageHandler.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStructField.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/KuduStructObjectInspector.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/PartialRowWritable.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/TimestampConverter.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduRecordReader.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduTableInputFormat.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/input/KuduTableSplit.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/output/HiveKuduOutputFormat.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/output/KuduRecordUpserter.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/utils/objectinspectorutils/OIUtils.java A java/kudu-hive-serde/src/main/java/org/apache/kudu/hive/serde/utils/objectinspectorutils/inpectors/RenameStructFieldsStructObjectInspector.java A java/kudu-hive-serde/src/test/java/org/apache/kudu/hive/serde/kudu/KuduStructObjectInspectorTest.java M java/settings.gradle 20 files changed, 2,591 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/13230/1 -- To view, visit http://gerrit.cloudera.org:8080/13230 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I72daf3c3f06ccf67e3fa3039c4afa20f6c075cba Gerrit-Change-Number: 13230 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente
[kudu-CR] Throw exception on write error with kudu-mapreduce
Clemens Valiente has posted comments on this change. ( http://gerrit.cloudera.org:8080/13228 ) Change subject: Throw exception on write error with kudu-mapreduce .. Patch Set 7: Yes, I had a look at it and it turns out kudu-mapreduce doesn't support running two unit tests on the KuduOutputFormat... it would need quite some work refactoring it and I am not sure mapreduce is still used enough to be worth it. I decided to not use this package for the hive serde. > Looks like there are two Java test failures to fix here: > -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 7 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Fri, 03 May 2019 16:14:54 + Gerrit-HasComments: No
[kudu-CR] Throw exception on write error with kudu-mapreduce
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13228 to look at the new patch set (#6). Change subject: Throw exception on write error with kudu-mapreduce .. Throw exception on write error with kudu-mapreduce The kudu-mapreduce package only logs write errors but doesn't act on them. This can cause jobs appear to be successful despite not writing a single successful row. Since records are written asynchronously, we need to check for errors during the write process, and once again when we close the kudu session. Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableOutputFormat.java 2 files changed, 53 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/28/13228/6 -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 6 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] Throw exception on write error with kudu-mapreduce
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13228 to look at the new patch set (#5). Change subject: Throw exception on write error with kudu-mapreduce .. Throw exception on write error with kudu-mapreduce The kudu-mapreduce package only logs write errors but doesn't act on them. This can cause jobs appear to be successful despite not writing a single successful row. Since records are written asynchronously, we need to check for errors during the write process, and once again when we close the kudu session. Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableOutputFormat.java 2 files changed, 47 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/28/13228/5 -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 5 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] Throw exception on write errors with the kudu-mapreduce client
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13228 to look at the new patch set (#2). Change subject: Throw exception on write errors with the kudu-mapreduce client .. Throw exception on write errors with the kudu-mapreduce client The kudu-mapreduce package only logs write errors but doesn't act on them. This can cause jobs appear to be successful despite not writing a single successful row. Since records are written asynchronously, we need to check for errors during the write process, and once again when we close the kudu session. Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableOutputFormat.java 2 files changed, 47 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/28/13228/2 -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 2 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] Throw exception on write errors with kudu-mapreduce
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13228 to look at the new patch set (#3). Change subject: Throw exception on write errors with kudu-mapreduce .. Throw exception on write errors with kudu-mapreduce The kudu-mapreduce package only logs write errors but doesn't act on them. This can cause jobs appear to be successful despite not writing a single successful row. Since records are written asynchronously, we need to check for errors during the write process, and once again when we close the kudu session. Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableOutputFormat.java 2 files changed, 47 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/28/13228/3 -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 3 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] Throw exception on write error with kudu-mapreduce
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13228 to look at the new patch set (#4). Change subject: Throw exception on write error with kudu-mapreduce .. Throw exception on write error with kudu-mapreduce The kudu-mapreduce package only logs write errors but doesn't act on them. This can cause jobs appear to be successful despite not writing a single successful row. Since records are written asynchronously, we need to check for errors during the write process, and once again when we close the kudu session. Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableOutputFormat.java 2 files changed, 47 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/28/13228/4 -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 4 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] throw error on write errors
Clemens Valiente has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13228 Change subject: throw error on write errors .. throw error on write errors Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableOutputFormat.java 2 files changed, 49 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/28/13228/1 -- To view, visit http://gerrit.cloudera.org:8080/13228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6db6a59104425c396fbbeb2105bf2c777cc348c7 Gerrit-Change-Number: 13228 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente
[kudu-CR] Move configuration keys into its own class and make public
Clemens Valiente has posted comments on this change. ( http://gerrit.cloudera.org:8080/8920 ) Change subject: Move configuration keys into its own class and make public .. Patch Set 2: Todd, David, as promised I fixed the merge conflict and added motive and description. please also refer to my previous comment -- To view, visit http://gerrit.cloudera.org:8080/8920 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64f9ae5c2bc21ddaf757530a33642bd18cd774bf Gerrit-Change-Number: 8920 Gerrit-PatchSet: 2 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 11 Jan 2018 09:17:53 + Gerrit-HasComments: No
[kudu-CR] Move configuration keys into its own class and make public
Hello David Ribeiro Alves, Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8920 to look at the new patch set (#2). Change subject: Move configuration keys into its own class and make public .. Move configuration keys into its own class and make public External applications using the kudu-mapreduce package need to set these configuration properties correctly. Usually they are set in the JobConfiguration entity of a mapreduce job. For this, it is highly preferable to have access to the config keys as they were defined within the kudu-mapreduce package instead of redefining them within the application (reduces duplication and potential for errors) Since InputFormat and OutputFormat use some similar keys (and actually used to use slightly different property names for the same thing in the past) this commit moves all configuration keys into a common class and makes them public for access. Change-Id: I64f9ae5c2bc21ddaf757530a33642bd18cd774bf --- A java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduMapReduceConstants.java M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableInputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableOutputFormat.java 6 files changed, 106 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/20/8920/2 -- To view, visit http://gerrit.cloudera.org:8080/8920 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I64f9ae5c2bc21ddaf757530a33642bd18cd774bf Gerrit-Change-Number: 8920 Gerrit-PatchSet: 2 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR] KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader
Clemens Valiente has posted comments on this change. ( http://gerrit.cloudera.org:8080/8921 ) Change subject: KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader .. Patch Set 4: > Patch Set 4: > > (2 comments) I added the clarification. The lines should below 80 cols already if I am not mistaken? -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 4 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Mon, 08 Jan 2018 10:39:59 + Gerrit-HasComments: No
[kudu-CR] KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader
Hello David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8921 to look at the new patch set (#5). Change subject: KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader .. KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader This commit prevents a possible race condition between getSplits() method and TableRecordReader in the KuduTableInputFormat, when both try to access and shutdown the KuduClient. Both are sharing the same client and shut it down after use. In some scenarios the client might still be accessed after that and throwing an error. So the TableRecordReader gets its own client with this commit. This increases the number of opened Kudu clients by a MR application at most by one (The one that was shared by getSplits() with a TableRecordReader) Also clarified the behaviour of MR applications and how many open Kudu clients one might have to expect in total. Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 45 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/8921/5 -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 5 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] move configuration constants into separate class and make public
Clemens Valiente has posted comments on this change. ( http://gerrit.cloudera.org:8080/8920 ) Change subject: move configuration constants into separate class and make public .. Patch Set 1: Sorry, this is definitely my fault for making the commit message very sparse. First, my plan is to first have PR 8921 approved and then to fix the arising merge conflicts and commit message here. My motivation: External applications using the kudu-mapreduce package need to set these configuration properties correctly. Making the keys public means we can use them directly and avoid duplication, inconsistencies, mistakes through typos, and version conflicts when these properties are changed (as already happened in kudu-mapreduce). Concrete example is the hive-kudu SerDe I am working on. I need to copy the table name and master addresses from the table properties to the MR configuration for the KuduTableInput/OutputFormats. I would feel much more comfortable if I could refer to these Constants in the kudu-mapreduce package directly instead of redefining them within my project again. In my experience it is also a common practice to expose Configuration Keys that other applications need to set. e.g. Hive: https://github.com/apache/hive/blob/master/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java Gobblin: https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java -- To view, visit http://gerrit.cloudera.org:8080/8920 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64f9ae5c2bc21ddaf757530a33642bd18cd774bf Gerrit-Change-Number: 8920 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 05 Jan 2018 10:18:44 + Gerrit-HasComments: No
[kudu-CR] KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader
Clemens Valiente has posted comments on this change. ( http://gerrit.cloudera.org:8080/8921 ) Change subject: KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader .. Patch Set 4: All done, thanks for the review! -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 4 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Fri, 05 Jan 2018 10:10:22 + Gerrit-HasComments: No
[kudu-CR] KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader
Hello David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8921 to look at the new patch set (#4). Change subject: KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader .. KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader This commit prevents a possible race condition between getSplits() method and TableRecordReader in the KuduTableInputFormat Both are sharing the same client and shut it down after use. In some scenarios the client might still be accessed after that and throwing an error. So the TableRecordReader gets its own client with this commit. This increases the number of opened Kudu clients by a MR application at most by one (The one that was shared by getSplits() with a TableRecordReader) Also clarified the behaviour of MR applications and how many open Kudu clients one might have to expect in total. Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 45 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/8921/4 -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 4 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-2249 Prevent race condition between getSplits() method and TableRecordReader
Clemens Valiente has posted comments on this change. ( http://gerrit.cloudera.org:8080/8921 ) Change subject: KUDU-2249 Prevent race condition between getSplits() method and TableRecordReader .. Patch Set 2: > Patch Set 1: > > (1 comment) Hi David, I clarified the behaviour of MR applications and number of open clients in the class documentation and motivation for the changes in the commit message. -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 2 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Thu, 04 Jan 2018 09:41:51 + Gerrit-HasComments: No
[kudu-CR] KUDU-2249 Prevent race condition between getSplits() method and TableRecordReader
Hello David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8921 to look at the new patch set (#3). Change subject: KUDU-2249 Prevent race condition between getSplits() method and TableRecordReader .. KUDU-2249 Prevent race condition between getSplits() method and TableRecordReader Both are sharing the same client and shut it down after use. In some scenarios the client might still be accessed after that and throwing an error. So the TableRecordReader gets its own client with this commit. This increases the number of opened Kud clients by a MR application at most by one (The one that was shared by getSplits() with a TableRecordReader) Also clarified the behaviour of MR applications and how many open Kudu clients one might have to expect in total. Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 45 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/8921/3 -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 3 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader
Clemens Valiente has abandoned this change. ( http://gerrit.cloudera.org:8080/8938 ) Change subject: KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader .. Abandoned duplicate -- To view, visit http://gerrit.cloudera.org:8080/8938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I833902b7ae813c1b4ae0505e20f206edc9875719 Gerrit-Change-Number: 8938 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-2249 Prevent race condition between getSplits() method and TableRecordReader
Hello David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8921 to look at the new patch set (#2). Change subject: KUDU-2249 Prevent race condition between getSplits() method and TableRecordReader .. KUDU-2249 Prevent race condition between getSplits() method and TableRecordReader Both are sharing the same client and shut it down after use. In some scenarios the client might still be accessed after that and throwing an error. So the TableRecordReader gets its own client with this commit. Also clarified the behaviour of MR applications and how many open Kudu clients one might have to expect. Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 45 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/8921/2 -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 2 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader class
Clemens Valiente has abandoned this change. ( http://gerrit.cloudera.org:8080/8937 ) Change subject: KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader class .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/8937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I6508a02edaba76f3234e64e2a44b4846dc7e818b Gerrit-Change-Number: 8937 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader
Clemens Valiente has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8938 Change subject: KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader .. KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader Both are sharing the same client and shut it down after use. In some scenarios the client might still be accessed after that and throwing an error. Also clarified the behaviour of MR applications and how many open Kudu clients one might have to expect. Change-Id: I833902b7ae813c1b4ae0505e20f206edc9875719 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 18 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/38/8938/1 -- To view, visit http://gerrit.cloudera.org:8080/8938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I833902b7ae813c1b4ae0505e20f206edc9875719 Gerrit-Change-Number: 8938 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente
[kudu-CR] KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader class
Clemens Valiente has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8937 Change subject: KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader class .. KUDU-2249 Prevent a possible race conditions between the getSplits() method and TableRecordReader class Both are sharing the same client and shut it down after use. In some scenarios the client might still be accessed after that and throwing an error. Also clarified the behaviour of MR applications and how many open Kudu clients one might have to expect. Change-Id: I6508a02edaba76f3234e64e2a44b4846dc7e818b --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 18 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/37/8937/1 -- To view, visit http://gerrit.cloudera.org:8080/8937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6508a02edaba76f3234e64e2a44b4846dc7e818b Gerrit-Change-Number: 8937 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente
[kudu-CR] KUDU-2249 give the TableRecordReader their own KuduClient to use.
Clemens Valiente has posted comments on this change. ( http://gerrit.cloudera.org:8080/8921 ) Change subject: KUDU-2249 give the TableRecordReader their own KuduClient to use. .. Patch Set 1: (1 comment) Hi David, I discussed this on the dev mailing list here: http://mail-archives.apache.org/mod_mbox/kudu-dev/201712.mbox/%3CAM0PR0502MB405140DD23048A17522BB55C9C070%40AM0PR0502MB4051.eurprd05.prod.outlook.com%3E The problem is the getInputSplit() closing the client that the Reader want to use. There's no real clean way of sharing the client and closing it properly. Due to the mapreduce architecture, each tablet usually will read from a separate Map Container and thus need its own client anyway. This just fixes a bug in the rare scenario that the getinputsplit and record reader are executed in one container. http://gerrit.cloudera.org:8080/#/c/8921/1/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java File java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java: http://gerrit.cloudera.org:8080/#/c/8921/1/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java@253 PS1, Line 253: private KuduClient getKuduClient() { : : String masterAddresses = conf.get(MASTER_ADDRESSES_KEY); : this.operationTimeoutMs = conf.getLong(OPERATION_TIMEOUT_MS_KEY, : AsyncKuduClient.DEFAULT_OPERATION_TIMEOUT_MS); : KuduClient kuduClient = new KuduClient.KuduClientBuilder(masterAddresses) : .defaultOperationTimeoutMs(operationTimeoutMs) : .build(); : KuduTableMapReduceUtil.importCredentialsFromCurrentSubject(kuduClient); : return kuduClient; : } > Kudu clients cache state internally, like tablet locations. Changing this t Hi David, I discussed this on the dev mailing list here: http://mail-archives.apache.org/mod_mbox/kudu-dev/201712.mbox/%3CAM0PR0502MB405140DD23048A17522BB55C9C070%40AM0PR0502MB4051.eurprd05.prod.outlook.com%3E The problem is the getInputSplit() closing the client that the Reader want to use. There's no real clean way of sharing the client and closing it properly. Due to the mapreduce architecture, each tablet usually will read from a separate Map Container and thus need its own client anyway. This just fixes a bug in the rare scenario that the getinputsplit and record reader are executed in one container. -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Wed, 03 Jan 2018 08:39:45 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2249 give the TableRecordReader their own KuduClient to use.
Clemens Valiente has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8921 Change subject: KUDU-2249 give the TableRecordReader their own KuduClient to use. .. KUDU-2249 give the TableRecordReader their own KuduClient to use. Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 27 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/8921/1 -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente
[kudu-CR] move configuration constants into separate class and make public
Clemens Valiente has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8920 Change subject: move configuration constants into separate class and make public .. move configuration constants into separate class and make public Change-Id: I64f9ae5c2bc21ddaf757530a33642bd18cd774bf --- A java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduMapReduceConstants.java M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableOutputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableInputFormat.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITKuduTableOutputFormat.java 6 files changed, 77 insertions(+), 84 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/20/8920/1 -- To view, visit http://gerrit.cloudera.org:8080/8920 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I64f9ae5c2bc21ddaf757530a33642bd18cd774bf Gerrit-Change-Number: 8920 Gerrit-PatchSet: 1 Gerrit-Owner: Clemens Valiente