[GitHub] flink pull request: [FLINK-1391] Add support for using Avro-POJOs ...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/323 [FLINK-1391] Add support for using Avro-POJOs and Avro types with Kryo With this change, users can use POJOs generated from Avro Schemas with Flink. Either the entire POJO as a GenericType or with the POJO serializer. I also added support for using `GenericData.Record`, which is a container that checks the types using a schema. Its extremely inefficient to use that. I guess the only realistic usecase for that is local prototyping. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink1391-rebased Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/323.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #323 commit 0e003c3c08537f03698ca122b4096d9512bfcad6 Author: Robert Metzger Date: 2015-01-12T20:11:09Z [FLINK-1391] Add support for using Avro-POJOs and Avro types with Kryo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1391] Add support for using Avro-POJOs ...
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/323#discussion_r23435527
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/KryoSerializer.java
---
@@ -237,6 +244,25 @@ private void checkKryoInitialized() {
// Throwable and all subclasses should be serialized
via java serialization
kryo.addDefaultSerializer(Throwable.class, new
JavaSerializer());
+ // If the type we have to serialize as a GenricType is
implementing SpecificRecordBase,
+ // we have to register the avro serializer
+ // This rule only applies if users explicitly use the
GenericTypeInformation for the avro types
+ // usually, we are able to handle Avro POJOs with the
POJO serializer.
+ if(SpecificRecordBase.class.isAssignableFrom(type)) {
+ ClassTag tag =
scala.reflect.ClassTag$.MODULE$.apply(type);
+ this.kryo.register(type,
com.twitter.chill.avro.AvroSerializer.SpecificRecordSerializer(tag));
+
+ }
+ // Avro POJOs contain java.util.List which have
GenericData.Array as their runtime type
+ // because Kryo is not able to serialize them properly,
we use this serializer for them
+ this.kryo.register(GenericData.Array.class, new
SpecificInstanceCollectionSerializer(ArrayList.class));
--- End diff --
Should we make this registration conditional so that it only happens when
we have encountered an Avro type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1391] Add support for using Avro-POJOs ...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/323#issuecomment-71157859 Looks like a good fix to me. One comment/question inline, otherwise good to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1391] Add support for using Avro-POJOs ...
Github user rmetzger closed the pull request at: https://github.com/apache/flink/pull/323 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1391] Add support for using Avro-POJOs ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/323#issuecomment-71483920 I will make this change part of a new pull request for https://issues.apache.org/jira/browse/FLINK-1417. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
