Hi, this is OK if there is a fixed structure being used as schema in dataframe. But in my case, the schema used dataframe is build dynamically from metadata provided by REST service (and is thus unknown during compilation) so I have no case class to be used as schema.
Thanks, R.Krist From: Wang, Daoyuan [mailto:daoyuan.w...@intel.com] Sent: Friday, April 17, 2015 10:23 AM To: Krist Rastislav; Michael Armbrust Cc: user Subject: RE: ClassCastException processing date fields using spark SQL since 1.3.0 Normally I use like the following in scala: >case calss datetest (x: Int, y:java.sql.Date) >val dt = sc.parallelize(1 to 3).map(p => datetest(p, new >java.sql.Date(p*1000*60*60*24))) >sqlContext.createDataFrame(dt).registerTempTable(“t1”) >sql(“select * from t1”).collect.foreach(println) If you still meets exceptions, please let me know about your query. The implicit conversion should be driven when you call createDataFrame Thanks, Daoyuan From: Krist Rastislav [mailto:rkr...@vub.sk] Sent: Friday, April 17, 2015 3:52 PM To: Wang, Daoyuan; Michael Armbrust Cc: user Subject: RE: ClassCastException processing date fields using spark SQL since 1.3.0 Hello, thank You for Your answer – I am creating the DataFrames manually using org.apache.spark.sql.SQLContext#createDataFrame. RDD is my custom implementation encapsulating invocation of a remote REST-based web service and schema is created programatically upon metadata (obtained from the same WS). So in other words, the creation of Rows in DataFrame is fully under my control and the implicit conversion thus cannot occur. Is there any best practice (ideally a utility method) of creating Row instance from a set of values of types represented by DataFrame schema? I will try to take a deeper look into Your source code to locate the definition of the implicit conversion, but maybe some hint from Your side could help deliver a better implementation. Thank You very much for Your help (and for the great work you are doing there). Regards R.Krist From: Wang, Daoyuan [mailto:daoyuan.w...@intel.com] Sent: Friday, April 17, 2015 5:08 AM To: Michael Armbrust; Krist Rastislav Cc: user Subject: RE: ClassCastException processing date fields using spark SQL since 1.3.0 The conversion between date and int should be automatically handled by Implicit conversion. So we are accepting date types externally, and represented as integer internally. From: Wang, Daoyuan Sent: Friday, April 17, 2015 11:00 AM To: 'Michael Armbrust'; rkrist Cc: user Subject: RE: ClassCastException processing date fields using spark SQL since 1.3.0 Can you tell us how did you create the dataframe? From: Michael Armbrust [mailto:mich...@databricks.com] Sent: Friday, April 17, 2015 2:52 AM To: rkrist Cc: user Subject: Re: ClassCastException processing date fields using spark SQL since 1.3.0 Filed: https://issues.apache.org/jira/browse/SPARK-6967 Shouldn't they be null? Statistics are only used to eliminate partitions that can't possibly hold matching values. So while you are right this might result in a false positive, that will not result in a wrong answer. ________________________________ Informacie, ktore su obsahom tejto spravy elektronickej posty a vsetky pripojene subory a prilohy su doverne a su/mozu byt obchodnym a/alebo bankovym tajomstvom alebo su/mozu byt pravne chranene podla inych pravnych predpisov. Pre blizsie informacie navstivte, prosim, www.vub.sk/legalcaution<http://www.vub.sk/legalcaution>. The information contained in this electronic mail message and any files and attachments transmitted are confidential and are/may be a trade and/or bank secret or are/may be legally privileged under other legal regulations. For further information, please, visit www.vub.sk/legalcaution<http://www.vub.sk/legalcaution>. VUB, a.s., Mlynske nivy 1, 829 90 Bratislava 25, Slovenska republika ________________________________ Pred vytlacenim e-mailu prosim zvazte dopad na zivotne prostredie. Before printing this e-mail, think about the impact on the environment. ________________________________ Informacie, ktore su obsahom tejto spravy elektronickej posty a vsetky pripojene subory a prilohy su doverne a su/mozu byt obchodnym a/alebo bankovym tajomstvom alebo su/mozu byt pravne chranene podla inych pravnych predpisov. Pre blizsie informacie navstivte, prosim, www.vub.sk/legalcaution. The information contained in this electronic mail message and any files and attachments transmitted are confidential and are/may be a trade and/or bank secret or are/may be legally privileged under other legal regulations. For further information, please, visit www.vub.sk/legalcaution. VUB, a.s., Mlynske nivy 1, 829 90 Bratislava 25, Slovenska republika ________________________________ Pred vytlacenim e-mailu prosim zvazte dopad na zivotne prostredie. Before printing this e-mail, think about the impact on the environment.