[kudu-CR] Add AvroKuduEventProducer to Kudu-Flume integration
Will Berkeley has posted comments on this change. Change subject: Add AvroKuduEventProducer to Kudu-Flume integration .. Patch Set 5: > BTW this producer is starting to look great. :D -- To view, visit http://gerrit.cloudera.org:8080/4034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6715df72e447e72f4801a2e026f6840d09b401e1 Gerrit-PatchSet: 5 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-HasComments: No
[kudu-CR] Add AvroKuduEventProducer to Kudu-Flume integration
Kudu Jenkins has posted comments on this change. Change subject: Add AvroKuduEventProducer to Kudu-Flume integration .. Patch Set 6: Build Started http://104.196.14.100/job/kudu-gerrit/3230/ -- To view, visit http://gerrit.cloudera.org:8080/4034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6715df72e447e72f4801a2e026f6840d09b401e1 Gerrit-PatchSet: 6 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-HasComments: No
[kudu-CR] Add AvroKuduEventProducer to Kudu-Flume integration
Will Berkeley has posted comments on this change.
Change subject: Add AvroKuduEventProducer to Kudu-Flume integration
..
Patch Set 5:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/4034/5/java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/AvroKuduEventProducer.java
File
java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/AvroKuduEventProducer.java:
Line 57: *
> nit: i don't these extra s do anything. Do they?
Done
Line 259: private Schema schema(Event event) throws FlumeException {
> mind naming this getSchema() for consistency with Java style?
Done
http://gerrit.cloudera.org:8080/#/c/4034/5/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/AvroKuduEventProducerTest.java
File
java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/AvroKuduEventProducerTest.java:
Line 61: private static final String schemaLiteral =
> you can use Guava and do:
Done
Line 111: private void testEvents(int eventCount, String whereIsSchema)
> style nit: let's create a SchemaLocation enum for whereIsSchema instead of
Done
Line 118: : new Context(ImmutableMap.of("producer.schemaPath",
schemaURI));
> style nit: Use constants for config values:
Done
Line 175: GenericRecord record = new GenericData.Record(schema);
> Would you mind using SpecificRecord instead of GenericRecord when writing t
I think I did this right.
--
To view, visit http://gerrit.cloudera.org:8080/4034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6715df72e447e72f4801a2e026f6840d09b401e1
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Will Berkeley
Gerrit-HasComments: Yes
[kudu-CR] Add AvroKuduEventProducer to Kudu-Flume integration
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4034 to look at the new patch set (#6). Change subject: Add AvroKuduEventProducer to Kudu-Flume integration .. Add AvroKuduEventProducer to Kudu-Flume integration This adds a KuduEventProducer that accepts events with Avro-encoded bodies and allows the KuduSink to push them to Kudu. It only accepts schemas of Record type that hold values of types compatible with Kudu's types. The schema is provided to the EventProducer ahead of time or in the event header, as a URL or as a literal. Change-Id: I6715df72e447e72f4801a2e026f6840d09b401e1 --- M java/kudu-flume-sink/pom.xml A java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/AvroKuduEventProducer.java A java/kudu-flume-sink/src/test/avro/testAvroEventProducer.avsc A java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/AvroKuduEventProducerTest.java 4 files changed, 563 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/34/4034/6 -- To view, visit http://gerrit.cloudera.org:8080/4034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6715df72e447e72f4801a2e026f6840d09b401e1 Gerrit-PatchSet: 6 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley
[kudu-CR] [WIP] KUDU-861 Support changing default, storage attributes
Will Berkeley has posted comments on this change. Change subject: [WIP] KUDU-861 Support changing default, storage attributes .. Patch Set 3: > Build Started http://104.196.14.100/job/kudu-gerrit/3229/ Fixed compilation whoopsie that doesn't show up on OS X. Should fail b/c Java client not changed yet. -- To view, visit http://gerrit.cloudera.org:8080/4310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-HasComments: No
[kudu-CR] [WIP] KUDU-861 Support changing default, storage attributes
Kudu Jenkins has posted comments on this change. Change subject: [WIP] KUDU-861 Support changing default, storage attributes .. Patch Set 3: Build Started http://104.196.14.100/job/kudu-gerrit/3229/ -- To view, visit http://gerrit.cloudera.org:8080/4310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
[kudu-CR] [WIP] KUDU-861 Support changing default, storage attributes
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4310 to look at the new patch set (#3). Change subject: [WIP] KUDU-861 Support changing default, storage attributes .. [WIP] KUDU-861 Support changing default, storage attributes This patch adds support for adding, changing, or removing column defaults. It adds support for these operations to the wire protocol, master, and C++ client, and folds column renaming into the same PB message as alter column more generally. It is not backwards-compatible. Some thorny things remain. The column's type isn't checked (or necessarily known) client-side, the server just receives some bytes for the default value, and, since the client stores 64-bit integer values as defaults for all integer-backed types, the value might be too large. We'll be ok, we just truncate. This is actually how it works for regular add column and create table, but there the type of the column is known, so the truncation is backed up by column type-checking, and it can't be the case that there's too few bytes. Possible solutions: 1. Leave it as it is. It works and shouldn't break, but users may have a weird experience like setting an integer column to have default value "", getting no error, and ending up with a default of 24929 = 0x61616161. 2. Put a discriminated union type structure in the PB message so some type checking can be done, i.e. INT vs FLOAT not INT32 vs. INT16. 3. Change KuduValue to require more type information + 2 above. Changing type and nullability of a column is still unsupported (doing so is much more complicated than this). Java client support is upcoming. Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c --- M src/kudu/client/client-test.cc M src/kudu/client/schema.cc M src/kudu/client/schema.h M src/kudu/client/table_alterer-internal.cc M src/kudu/client/value-internal.h M src/kudu/client/value.cc M src/kudu/common/schema.cc M src/kudu/common/schema.h M src/kudu/integration-tests/alter_table-test.cc M src/kudu/master/catalog_manager.cc M src/kudu/master/master.proto 11 files changed, 476 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/10/4310/3 -- To view, visit http://gerrit.cloudera.org:8080/4310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [WIP] KUDU-861 Support changing default, storage attributes
Kudu Jenkins has posted comments on this change. Change subject: [WIP] KUDU-861 Support changing default, storage attributes .. Patch Set 2: Build Started http://104.196.14.100/job/kudu-gerrit/3228/ -- To view, visit http://gerrit.cloudera.org:8080/4310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
[kudu-CR] [WIP] KUDU-861 Support changing default, storage attributes
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4310 to look at the new patch set (#2). Change subject: [WIP] KUDU-861 Support changing default, storage attributes .. [WIP] KUDU-861 Support changing default, storage attributes This patch adds support for adding, changing, or removing column defaults. It adds support for these operations to the wire protocol, master, and C++ client, and folds column renaming into the same PB message as alter column more generally. It is not backwards-compatible. Some thorny things remain. The column's type isn't checked (or necessarily known) client-side, the server just receives some bytes for the default value, and, since the client stores 64-bit integer values as defaults for all integer-backed types, the value might be too large. We'll be ok, we just truncate. This is actually how it works for regular add column and create table, but there the type of the column is known, so the truncation is backed up by column type-checking, and it can't be the case that there's too few bytes. Possible solutions: 1. Leave it as it is. It works and shouldn't break, but users may have a weird experience like setting an integer column to have default value "", getting no error, and ending up with a default of 24929 = 0x61616161. 2. Put a discriminated union type structure in the PB message so some type checking can be done, i.e. INT vs FLOAT not INT32 vs. INT16. 3. Change KuduValue to require more type information + 2 above. Changing type and nullability of a column is still unsupported (doing so is much more complicated than this). Java client support is upcoming. Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c --- M src/kudu/client/client-test.cc M src/kudu/client/schema.cc M src/kudu/client/schema.h M src/kudu/client/table_alterer-internal.cc M src/kudu/client/value-internal.h M src/kudu/client/value.cc M src/kudu/common/schema.cc M src/kudu/common/schema.h M src/kudu/integration-tests/alter_table-test.cc M src/kudu/master/catalog_manager.cc M src/kudu/master/master.proto 11 files changed, 476 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/10/4310/2 -- To view, visit http://gerrit.cloudera.org:8080/4310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [WIP] KUDU-861 Support changing default, storage attributes
Will Berkeley has uploaded a new change for review. http://gerrit.cloudera.org:8080/4310 Change subject: [WIP] KUDU-861 Support changing default, storage attributes .. [WIP] KUDU-861 Support changing default, storage attributes This patch adds support for adding, changing, or removing column defaults. It adds support for these operations to the wire protocol, master, and C++ client, and folds column renaming into the same PB message as alter column more generally. It is not backwards-compatible. Some thorny things remain. The column's type isn't checked (or necessarily known) client-side, the server just receives some bytes for the default value, and, since the client stores 64-bit integer values as defaults for all integer-backed types, the value might be too large. We'll be ok, we just truncate. This is actually how it works for regular add column and create table, but there the type of the column is known, so the truncation is backed up by column type-checking, and it can't be the case that there's too few bytes. Possible solutions: 1. Leave it as it is. It works and shouldn't break, but users may have a weird experience like setting an integer column to have default value "", getting no error, and ending up with a default of 24929 = 0x61616161. 2. Put a discriminated union type structure in the PB message so some type checking can be done, i.e. INT vs FLOAT not INT32 vs. INT16. 3. Change KuduValue to require more type information + 2 above. Changing type and nullability of a column is still unsupported (doing so is much more complicated than this). Java client support is upcoming. Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c --- M src/kudu/client/client-test.cc M src/kudu/client/schema.cc M src/kudu/client/schema.h M src/kudu/client/table_alterer-internal.cc M src/kudu/client/value-internal.h M src/kudu/client/value.cc M src/kudu/common/schema.cc M src/kudu/common/schema.h M src/kudu/integration-tests/alter_table-test.cc M src/kudu/master/catalog_manager.cc M src/kudu/master/master.proto 11 files changed, 476 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/10/4310/1 -- To view, visit http://gerrit.cloudera.org:8080/4310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley
[kudu-CR] [WIP] KUDU-861 Support changing default, storage attributes
Kudu Jenkins has posted comments on this change. Change subject: [WIP] KUDU-861 Support changing default, storage attributes .. Patch Set 1: Build Started http://104.196.14.100/job/kudu-gerrit/3227/ -- To view, visit http://gerrit.cloudera.org:8080/4310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I457d99ba2188ef6e439df47c0d94f2dc1a62ea6c Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
