Hi, Akhil,
In your definition of sdp_d
<http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi>,
all your fields are of type Option[X]. In Scala, a value of type Option[X]
can hold one of two things:
1. None
2. Some(x), where x is of type X
So to fix your immediate problem, wrap all your parameters to the sdp_d
constructor in Some(...), as follows:
new sdp_d(Some(r(0).trim.toInt), Some(r(1).trim.toInt), Some(r(2).trim),
...
Your earlier question of why writing sdp_d(...) for a case class works but
you need to write new sdp_d(...) for an explicit class, there's a simple
answer. When you create a case class X in scala, Scala also makes a
companion object X behind the scenes with an apply method that calls new
(see below). Scala's rules will call this apply method automatically. So,
when you write "X(...)", you're really calling "X.apply(...)" which in turn
calls "new X(...)". (This is the same trick behind writing things like
List(1,2,3)) If you don't use a case class, you'd have to make the
companion object yourself explicitly.
For reference, this statement:
case class X(a: A, b: B)
is conceptually equivalent to
class X(val a: A, val b: B) extends ... {
override def toString: String = // Auto-generated
override def hashCode: Int = // Auto-generated
override def equals(that: Any): Boolean = // Auto-generated
... more convenience methods ...
}
object X {
def apply(a: A, b: B) = new X(a, b)
... more convenience methods ...
}
If you want to peek under the hood, try compiling a simple X.scala file
with the line "case class X(a: Int, b: Double)", then taking apart the
generated X.class and X$.class (e.g., "javap X.class").
More info here <http://docs.scala-lang.org/tutorials/tour/case-classes.html>,
here <http://www.scala-lang.org/docu/files/ScalaReference.pdf> and in
Programming
in Scala <http://www.artima.com/shop/programming_in_scala_2ed> ch 15.
Hope that helps!
Best,
Patrick
On Thu, Feb 26, 2015 at 6:37 AM, anamika gupta <anamika.guo...@gmail.com>
wrote:
> I am now getting the following error. I cross-checked my types and
> corrected three of them i.e. r26-->String, r27-->Timestamp,
> r28-->Timestamp. This error still persists.
>
> scala>
> sc.textFile("/home/cdhuser/Desktop/Sdp_d.csv").map(_.split(",")).map { r =>
> | val upto_time = sdf.parse(r(23).trim);
> | calendar.setTime(upto_time);
> | val r23 = new java.sql.Timestamp(upto_time.getTime)
> | val insert_time = sdf.parse(r(27).trim)
> | calendar.setTime(insert_time)
> | val r27 = new java.sql.Timestamp(insert_time.getTime)
> | val last_upd_time = sdf.parse(r(28).trim)
> | calendar.setTime(last_upd_time)
> | val r28 = new java.sql.Timestamp(last_upd_time.getTime)
> | new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
> r(25).trim, r(26).trim, r27, r28)
> | }.registerAsTable("sdp_d")
>
> <console>:26: error: type mismatch;
> found : Int
> required: Option[Int]
> new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
> r(25).trim, r(26).trim, r27, r28)
>
> On Wed, Feb 25, 2015 at 2:32 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> It says sdp_d not found, since it is a class you need to instantiate it
>> once. like:
>>
>> sc.textFile("derby.log").map(_.split(",")).map( r => {
>> val upto_time = sdf.parse(r(23).trim);
>> calendar.setTime(upto_time);
>> val r23 = new java.sql.Timestamp(upto_time.getTime);
>>
>> val insert_time = sdf.parse(r(26).trim);
>> calendar.setTime(insert_time);
>> val r26 = new java.sql.Timestamp(insert_time.getTime);
>>
>> val last_upd_time = sdf.parse(r(27).trim);
>> calendar.setTime(last_upd_time);
>> val r27 = new java.sql.Timestamp(last_upd_time.getTime);
>>
>> *new* *sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
>> r(25).trim, r26, r27, r(28).trim)*
>> }).registerAsTable("sdp")
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Feb 25, 2015 at 2:14 PM, anamika gupta <anamika.guo...@gmail.com>
>> wrote:
>>
>>> The link has proved helpful. I have been able to load data, register it
>>> as a table and perform simple queries. Thanks Akhil !!
>>>
>>> Though, I still look forward to knowing where I was going wrong with my
>>> previous technique of extending the Product Interface to overcome case
>>> class's limit of 22 fields.
>>>
>>> On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta <anamika.guo...@gmail.com
>>> > wrote:
>>>
>>>> Hi Akhil
>>>>
>>>> I guess it skipped my attention. I would definitely give it a try.
>>>>
>>>> While I would still like to know what is the issue with the way I have
>>>> created schema?
>>>>
>>>> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>>> wrote:
>>>>
>>>>> Did you happen to have a look at
>>>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>>>>>
>>>>> Thanks
>>>>> Best Regards
>>>>>
>>>>> On Tue, Feb 24, 2015 at 3:39 PM, anu <anamika.guo...@gmail.com> wrote:
>>>>>
>>>>>> My issue is posted here on stack-overflow. What am I doing wrong here?
>>>>>>
>>>>>>
>>>>>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>>>>>>
>>>>>> ------------------------------
>>>>>> View this message in context: Facing error while extending scala
>>>>>> class with Product interface to overcome limit of 22 fields in
>>>>>> spark-shell
>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
>>>>>> Sent from the Apache Spark User List mailing list archive
>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
