Re: scala.MatchError: class org.apache.avro.Schema (of class java.lang.Class)
For more details on my question http://apache-spark-user-list.1001560.n3.nabble.com/How-to-generate-Java-bean-class-for-avro-files-using-spark-avro-project-tp22413.html Thanks, Yamini On Tue, Apr 7, 2015 at 2:23 PM, Yamini Maddirala wrote: > Hi Michael, > > Yes, I did try spark-avro 0.2.0 databricks project. I am using CHD5.3 > which is based on spark 1.2. Hence I'm bound to use spark-avro 0.2.0 > instead of the latest. > > I'm not sure how spark-avro project can help me in this scenario. > > 1. I have JavaDStream of type avro generic record > :JavaDStream [This is the data being read from kafka topics] > 2. I'm able to get JavaSchemaRDD using the avro file like this > final JavaSchemaRDD schemaRDD2 = AvroUtils.avroFile(sqlContext, > "/xyz-Project/trunk/src/main/resources/xyz.avro"); > 3. I don't know how I can apply schema in step 2 to data in step 1. > I chose to do something like this >JavaSchemaRDD schemaRDD = sqlContext.applySchema(genericRecordJavaRDD, > xyz.class); > >Used avro maven plugin to generate xyz class in Java. But this is not > good because avro maven plugin creates a field SCHEMA which is not > supported in applySchema method. > > Please let me know how to deal with this. > > Appreciate your help > > Thanks, > Yamini > > > > > > > > > > > > > On Tue, Apr 7, 2015 at 1:57 PM, Michael Armbrust > wrote: > >> Have you looked at spark-avro? >> >> https://github.com/databricks/spark-avro >> >> On Tue, Apr 7, 2015 at 3:57 AM, Yamini wrote: >> >>> Using spark(1.2) streaming to read avro schema based topics flowing in >>> kafka >>> and then using spark sql context to register data as temp table. Avro >>> maven >>> plugin(1.7.7 version) generates the java bean class for the avro file but >>> includes a field named SCHEMA$ of type org.apache.avro.Schema which is >>> not >>> supported in the JavaSQLContext class[Method : applySchema]. >>> How to auto generate java bean class for the avro file and over come the >>> above mentioned problem. >>> >>> Thanks. >>> >>> >>> >>> >>> - >>> Thanks, >>> Yamini >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
How to generate Java bean class for avro files using spark avro project
Is there a way to generate Java bean for a given avro schema file in spark 1.2 using spark-avro project 0.2.0 for following use case? 1. Topics from kafka read and stored in the form of avro generic records :JavaDStream 2. Using spark avro project able to get the schema in the following way JavaSchemaRDD schemaRDD2 = AvroUtils.avroFile(sqlContext, PathTofile.avro) 3. For each record in the above mentioned JavaDStream, need to apply schema retrieved in step 2. Chose to do this JavaSchemaRDD schemaRDD = sqlContext.applySchema(genericRecordJavaRDD, PathTofile.class) To generate Java bean class(PathTofile.class) chose to use avro maven plugin. But the generated java bean using plugin includes field named SCHEMA which is not supported my the applySchema method mentioned above. Please let me know if there is a better solution for this. - Thanks, Yamini -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-generate-Java-bean-class-for-avro-files-using-spark-avro-project-tp22413.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: scala.MatchError: class org.apache.avro.Schema (of class java.lang.Class)
Hi Michael, Yes, I did try spark-avro 0.2.0 databricks project. I am using CHD5.3 which is based on spark 1.2. Hence I'm bound to use spark-avro 0.2.0 instead of the latest. I'm not sure how spark-avro project can help me in this scenario. 1. I have JavaDStream of type avro generic record :JavaDStream [This is the data being read from kafka topics] 2. I'm able to get JavaSchemaRDD using the avro file like this final JavaSchemaRDD schemaRDD2 = AvroUtils.avroFile(sqlContext, "/xyz-Project/trunk/src/main/resources/xyz.avro"); 3. I don't know how I can apply schema in step 2 to data in step 1. I chose to do something like this JavaSchemaRDD schemaRDD = sqlContext.applySchema(genericRecordJavaRDD, xyz.class); Used avro maven plugin to generate xyz class in Java. But this is not good because avro maven plugin creates a field SCHEMA which is not supported in applySchema method. Please let me know how to deal with this. Appreciate your help Thanks, Yamini On Tue, Apr 7, 2015 at 1:57 PM, Michael Armbrust wrote: > Have you looked at spark-avro? > > https://github.com/databricks/spark-avro > > On Tue, Apr 7, 2015 at 3:57 AM, Yamini wrote: > >> Using spark(1.2) streaming to read avro schema based topics flowing in >> kafka >> and then using spark sql context to register data as temp table. Avro >> maven >> plugin(1.7.7 version) generates the java bean class for the avro file but >> includes a field named SCHEMA$ of type org.apache.avro.Schema which is not >> supported in the JavaSQLContext class[Method : applySchema]. >> How to auto generate java bean class for the avro file and over come the >> above mentioned problem. >> >> Thanks. >> >> >> >> >> - >> Thanks, >> Yamini >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
scala.MatchError: class org.apache.avro.Schema (of class java.lang.Class)
Using spark(1.2) streaming to read avro schema based topics flowing in kafka and then using spark sql context to register data as temp table. Avro maven plugin(1.7.7 version) generates the java bean class for the avro file but includes a field named SCHEMA$ of type org.apache.avro.Schema which is not supported in the JavaSQLContext class[Method : applySchema]. How to auto generate java bean class for the avro file and over come the above mentioned problem. Thanks. - Thanks, Yamini -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: NoSuchMethodException KafkaUtils.
Customized spark-streaming-kafka_2.10-1.1.0.jar. Included a new method in kafkaUtils class to handle byte array format. That helped. - Thanks, Yamini -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodException-KafkaUtils-tp17142p22384.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
