For more details on my question http://apache-spark-user-list.1001560.n3.nabble.com/How-to-generate-Java-bean-class-for-avro-files-using-spark-avro-project-tp22413.html
Thanks, Yamini On Tue, Apr 7, 2015 at 2:23 PM, Yamini Maddirala <yamini.m...@gmail.com> wrote: > Hi Michael, > > Yes, I did try spark-avro 0.2.0 databricks project. I am using CHD5.3 > which is based on spark 1.2. Hence I'm bound to use spark-avro 0.2.0 > instead of the latest. > > I'm not sure how spark-avro project can help me in this scenario. > > 1. I have JavaDStream of type avro generic record > :JavaDStream<GenericRecord> [This is the data being read from kafka topics] > 2. I'm able to get JavaSchemaRDD using the avro file like this > final JavaSchemaRDD schemaRDD2 = AvroUtils.avroFile(sqlContext, > "/xyz-Project/trunk/src/main/resources/xyz.avro"); > 3. I don't know how I can apply schema in step 2 to data in step 1. > I chose to do something like this > JavaSchemaRDD schemaRDD = sqlContext.applySchema(genericRecordJavaRDD, > xyz.class); > > Used avro maven plugin to generate xyz class in Java. But this is not > good because avro maven plugin creates a field SCHEMA which is not > supported in applySchema method. > > Please let me know how to deal with this. > > Appreciate your help > > Thanks, > Yamini > > > > > > > > > > > > > On Tue, Apr 7, 2015 at 1:57 PM, Michael Armbrust <mich...@databricks.com> > wrote: > >> Have you looked at spark-avro? >> >> https://github.com/databricks/spark-avro >> >> On Tue, Apr 7, 2015 at 3:57 AM, Yamini <yamini.m...@gmail.com> wrote: >> >>> Using spark(1.2) streaming to read avro schema based topics flowing in >>> kafka >>> and then using spark sql context to register data as temp table. Avro >>> maven >>> plugin(1.7.7 version) generates the java bean class for the avro file but >>> includes a field named SCHEMA$ of type org.apache.avro.Schema which is >>> not >>> supported in the JavaSQLContext class[Method : applySchema]. >>> How to auto generate java bean class for the avro file and over come the >>> above mentioned problem. >>> >>> Thanks. >>> >>> >>> >>> >>> ----- >>> Thanks, >>> Yamini >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
