Re: How to join two records?
In case this helps anyone, I created a simple class to do the join and it
works nicely for my use case. If there's already a library function for
this, let me know.
Example (
https://github.com/Quantiply/rico/blob/master/avro-serde/src/test/java/com/quantiply/avro/JoinTest.java#L44-L52
):
GenericRecord in1 = getIn1();
GenericRecord in2 = getIn2();
GenericRecord joined = new Join(getJoinedSchema())
.merge(in1)
.merge(in2)
.getBuilder()
.set(charlie, blah blah)
.build();
Class is here:
https://github.com/Quantiply/rico/blob/master/avro-serde/src/main/java/com/quantiply/avro/Join.java
Cheers,
Roger
On Thu, Apr 9, 2015 at 8:29 AM, Roger Hoover roger.hoo...@gmail.com wrote:
Hi,
If I have a two records of type A and B as below and want to join them
similar to SELECT * in SQL to produce a record of type AB, is there an
simple way to do this with Avro without writing code to copy each field
individually? It's like a reverse projection. I want to take to disjoint
projections of AB and merge them to create an AB record.
I appreciate any help.
Thanks,
Roger
{
name: A,
type: record,
namespace: fubar,
fields: [{name: a, type : int}]
}
{
name: B,
type: record,
namespace: fubar,
fields: [{name: b, type : int}]
}
{
name: AB,
type: record,
namespace: fubar,
fields: [{name: a, type : int}, {name: b, type : int}]
}
Encoding Avro schema as binary
Hi, I wanted to see how much overhead would be involved if we ship an avro schema along with message in a messaging context. Seems like it might simplify things to not always need a schema registry available with all schema versions. I found an old thread ( http://search-hadoop.com/m/zmrzAWDkbt1/noble+paul/v=threaded) referencing this issue. (https://issues.apache.org/jira/browse/AVRO-251) A couple questions? 1) Any particular reason the patch was never merged? Can anyone foresee major issues with this approach? 2) I applied this patch to the 1.5 branch thinking that I'd have the best luck there (both are from 2011). I'm having trouble building anything that depends on the avro-maven-plugin (see errors below). Any help would be appreciated testing this out. Thanks! I'm getting this error trying to use the avro-maven-plugin to compile the ipc module. Any ideas? [ERROR] Failed to execute goal org.apache.avro:avro-maven-plugin:1.5.5-SNAPSHOT:schema (schemas) on project avro-ipc: Execution schemas of goal org.apache.avro:avro-maven-plugin:1.5.5-SNAPSHOT:schema failed: An API incompatibility was encountered while executing org.apache.avro:avro-maven-plugin:1.5.5-SNAPSHOT:schema: java.lang.ExceptionInInitializerError: null [ERROR] - [ERROR] realm = pluginorg.apache.avro:avro-maven-plugin:1.5.5-SNAPSHOT [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy [ERROR] urls[0] = file:/Users/rhoover/.m2/repository/org/apache/avro/avro-maven-plugin/1.5.5-SNAPSHOT/avro-maven-plugin-1.5.5-SNAPSHOT.jar file:///Users/rhoover/.m2/repository/org/apache/avro/avro-maven-plugin/1.5.5-SNAPSHOT/avro-maven-plugin-1.5.5-SNAPSHOT.jar [ERROR] urls[1] = file:/Users/rhoover/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.jar file:///Users/rhoover/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.1/plexus-interpolation-1.1.jar [ERROR] urls[2] = file:/Users/rhoover/.m2/repository/org/codehaus/plexus/plexus-utils/1.5.5/plexus-utils-1.5.5.jar file:///Users/rhoover/.m2/repository/org/codehaus/plexus/plexus-utils/1.5.5/plexus-utils-1.5.5.jar [ERROR] urls[3] = file:/Users/rhoover/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar file:///Users/rhoover/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar [ERROR] urls[4] = file:/Users/rhoover/.m2/repository/org/apache/maven/shared/file-management/1.2.1/file-management-1.2.1.jar file:///Users/rhoover/.m2/repository/org/apache/maven/shared/file-management/1.2.1/file-management-1.2.1.jar [ERROR] urls[5] = file:/Users/rhoover/.m2/repository/org/apache/maven/shared/maven-shared-io/1.1/maven-shared-io-1.1.jar file:///Users/rhoover/.m2/repository/org/apache/maven/shared/maven-shared-io/1.1/maven-shared-io-1.1.jar [ERROR] urls[6] = file:/Users/rhoover/.m2/repository/org/apache/avro/avro-compiler/1.5.5-SNAPSHOT/avro-compiler-1.5.5-SNAPSHOT.jar file:///Users/rhoover/.m2/repository/org/apache/avro/avro-compiler/1.5.5-SNAPSHOT/avro-compiler-1.5.5-SNAPSHOT.jar [ERROR] urls[7] = file:/Users/rhoover/.m2/repository/org/apache/avro/avro/1.5.5-SNAPSHOT/avro-1.5.5-SNAPSHOT.jar file:///Users/rhoover/.m2/repository/org/apache/avro/avro/1.5.5-SNAPSHOT/avro-1.5.5-SNAPSHOT.jar [ERROR] urls[8] = file:/Users/rhoover/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar file:///Users/rhoover/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar [ERROR] urls[9] = file:/Users/rhoover/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.7.3/jackson-core-asl-1.7.3.jar file:///Users/rhoover/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.7.3/jackson-core-asl-1.7.3.jar [ERROR] urls[10] = file:/Users/rhoover/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar file:///Users/rhoover/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar [ERROR] urls[11] = file:/Users/rhoover/.m2/repository/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar file:///Users/rhoover/.m2/repository/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar [ERROR] urls[12] = file:/Users/rhoover/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar file:///Users/rhoover/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar [ERROR] urls[13] = file:/Users/rhoover/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar file:///Users/rhoover/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar [ERROR] urls[14] = file:/Users/rhoover/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar file:///Users/rhoover/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar [ERROR] urls[15] = file:/Users/rhoover/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar file:///Users/rhoover/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar [ERROR] urls[16] =
Re: Direct conversion from Generic Record to Specific Record
Great. Thanks, Christophe. On Thu, Feb 6, 2014 at 8:12 AM, Christophe Taton christophe.ta...@gmail.com wrote: I believe we could eventually make a generic function that can rewrite any record or record builder, whether specific or generic, into another record or record builder, whether specific or generic. Until this happens, Mika's suggestion is probably the best short-term solution. C. On Wed, Feb 5, 2014 at 2:59 PM, Roger Hoover roger.hoo...@gmail.comwrote: Hi Christophe, Thanks for your reply. When you say that we could write a generic conversion function, do you mean we can write one that works for all schemas? That would be great! I'd like an API something like this: MyCustomRecord record = MyCustomRecord.newBuilder(GenericRecord record).build() Thanks, Roger On Wed, Feb 5, 2014 at 10:59 AM, Christophe Taton christophe.ta...@gmail.com wrote: Hi Roger, With the change proposed in https://issues.apache.org/jira/browse/AVRO-1443, you would be able to create a specific record using a generic builder API. That means we could write a generic conversion function that creates specific records. Would that work for you? C. On Wed, Feb 5, 2014 at 10:23 AM, Roger Hoover roger.hoo...@gmail.comwrote: Hi, I'm working with an existing API that hands me a GenericRecord. Is there an easy way I can covert it into a SpecificRecord. Really I want to get it into the code-generated object. Thanks, Roger
How to convert a generic record to a specific record?
Hi, Sorry if this is a noob question. Is there an easy way to convert a generic record into a specific record? Thanks, Roger
