)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
From: Krist Rastislav
Sent: Friday, April 17, 2015 10:31 AM
To: 'Wang, Daoyuan'; Michael Armbrust
Cc: user
Subject: RE: ClassCastException processing date fields using spark SQL since
1.3.0
there).
Regards
R.Krist
From: Wang, Daoyuan [mailto:daoyuan.w...@intel.com]
Sent: Friday, April 17, 2015 5:08 AM
To: Michael Armbrust; Krist Rastislav
Cc: user
Subject: RE: ClassCastException processing date fields using spark SQL since
1.3.0
The conversion between date and int should
Thank you for the explanation! I’ll check what can be done here.
From: Krist Rastislav [mailto:rkr...@vub.sk]
Sent: Friday, April 17, 2015 9:03 PM
To: Wang, Daoyuan; Michael Armbrust
Cc: user
Subject: RE: ClassCastException processing date fields using spark SQL since
1.3.0
So finally
...one additional note:
implementation of org.apache.spark.sql.columnar.IntColumnStats is IMHO
wrong. Small hint - what will be the resulting upper and lower values for
column containing no data (empty RDD or null values in Int column across the
whole RDD)?
Shouldn't they be null?
--
View
Filed: https://issues.apache.org/jira/browse/SPARK-6967
Shouldn't they be null?
Statistics are only used to eliminate partitions that can't possibly hold
matching values. So while you are right this might result in a false
positive, that will not result in a wrong answer.
Can you tell us how did you create the dataframe?
From: Michael Armbrust [mailto:mich...@databricks.com]
Sent: Friday, April 17, 2015 2:52 AM
To: rkrist
Cc: user
Subject: Re: ClassCastException processing date fields using spark SQL since
1.3.0
Filed: https://issues.apache.org/jira/browse/SPARK
: ClassCastException processing date fields using spark SQL since
1.3.0
Can you tell us how did you create the dataframe?
From: Michael Armbrust [mailto:mich...@databricks.com]
Sent: Friday, April 17, 2015 2:52 AM
To: rkrist
Cc: user
Subject: Re: ClassCastException processing date fields using spark SQL
