scala.MatchError: class org.apache.avro.Schema (of class java.lang.Class)
Using spark(1.2) streaming to read avro schema based topics flowing in kafka and then using spark sql context to register data as temp table. Avro maven plugin(1.7.7 version) generates the java bean class for the avro file but includes a field named SCHEMA$ of type org.apache.avro.Schema which is not supported in the JavaSQLContext class[Method : applySchema]. How to auto generate java bean class for the avro file and over come the above mentioned problem. Thanks. - Thanks, Yamini -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: scala.MatchError: class org.apache.avro.Schema (of class java.lang.Class)
For more details on my question http://apache-spark-user-list.1001560.n3.nabble.com/How-to-generate-Java-bean-class-for-avro-files-using-spark-avro-project-tp22413.html Thanks, Yamini On Tue, Apr 7, 2015 at 2:23 PM, Yamini Maddirala yamini.m...@gmail.com wrote: Hi Michael, Yes, I did try spark-avro 0.2.0 databricks project. I am using CHD5.3 which is based on spark 1.2. Hence I'm bound to use spark-avro 0.2.0 instead of the latest. I'm not sure how spark-avro project can help me in this scenario. 1. I have JavaDStream of type avro generic record :JavaDStreamGenericRecord [This is the data being read from kafka topics] 2. I'm able to get JavaSchemaRDD using the avro file like this final JavaSchemaRDD schemaRDD2 = AvroUtils.avroFile(sqlContext, /xyz-Project/trunk/src/main/resources/xyz.avro); 3. I don't know how I can apply schema in step 2 to data in step 1. I chose to do something like this JavaSchemaRDD schemaRDD = sqlContext.applySchema(genericRecordJavaRDD, xyz.class); Used avro maven plugin to generate xyz class in Java. But this is not good because avro maven plugin creates a field SCHEMA which is not supported in applySchema method. Please let me know how to deal with this. Appreciate your help Thanks, Yamini On Tue, Apr 7, 2015 at 1:57 PM, Michael Armbrust mich...@databricks.com wrote: Have you looked at spark-avro? https://github.com/databricks/spark-avro On Tue, Apr 7, 2015 at 3:57 AM, Yamini yamini.m...@gmail.com wrote: Using spark(1.2) streaming to read avro schema based topics flowing in kafka and then using spark sql context to register data as temp table. Avro maven plugin(1.7.7 version) generates the java bean class for the avro file but includes a field named SCHEMA$ of type org.apache.avro.Schema which is not supported in the JavaSQLContext class[Method : applySchema]. How to auto generate java bean class for the avro file and over come the above mentioned problem. Thanks. - Thanks, Yamini -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: scala.MatchError: class org.apache.avro.Schema (of class java.lang.Class)
Have you looked at spark-avro? https://github.com/databricks/spark-avro On Tue, Apr 7, 2015 at 3:57 AM, Yamini yamini.m...@gmail.com wrote: Using spark(1.2) streaming to read avro schema based topics flowing in kafka and then using spark sql context to register data as temp table. Avro maven plugin(1.7.7 version) generates the java bean class for the avro file but includes a field named SCHEMA$ of type org.apache.avro.Schema which is not supported in the JavaSQLContext class[Method : applySchema]. How to auto generate java bean class for the avro file and over come the above mentioned problem. Thanks. - Thanks, Yamini -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: scala.MatchError: class org.apache.avro.Schema (of class java.lang.Class)
Hi Michael, Yes, I did try spark-avro 0.2.0 databricks project. I am using CHD5.3 which is based on spark 1.2. Hence I'm bound to use spark-avro 0.2.0 instead of the latest. I'm not sure how spark-avro project can help me in this scenario. 1. I have JavaDStream of type avro generic record :JavaDStreamGenericRecord [This is the data being read from kafka topics] 2. I'm able to get JavaSchemaRDD using the avro file like this final JavaSchemaRDD schemaRDD2 = AvroUtils.avroFile(sqlContext, /xyz-Project/trunk/src/main/resources/xyz.avro); 3. I don't know how I can apply schema in step 2 to data in step 1. I chose to do something like this JavaSchemaRDD schemaRDD = sqlContext.applySchema(genericRecordJavaRDD, xyz.class); Used avro maven plugin to generate xyz class in Java. But this is not good because avro maven plugin creates a field SCHEMA which is not supported in applySchema method. Please let me know how to deal with this. Appreciate your help Thanks, Yamini On Tue, Apr 7, 2015 at 1:57 PM, Michael Armbrust mich...@databricks.com wrote: Have you looked at spark-avro? https://github.com/databricks/spark-avro On Tue, Apr 7, 2015 at 3:57 AM, Yamini yamini.m...@gmail.com wrote: Using spark(1.2) streaming to read avro schema based topics flowing in kafka and then using spark sql context to register data as temp table. Avro maven plugin(1.7.7 version) generates the java bean class for the avro file but includes a field named SCHEMA$ of type org.apache.avro.Schema which is not supported in the JavaSQLContext class[Method : applySchema]. How to auto generate java bean class for the avro file and over come the above mentioned problem. Thanks. - Thanks, Yamini -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org