Hi Patrick Thanks a ton for your in-depth answer. The compilation error is now resolved.
Thanks a lot again !! On Thu, Feb 26, 2015 at 2:40 PM, Patrick Varilly < patrick.vari...@dataminded.be> wrote: > Hi, Akhil, > > In your definition of sdp_d > <http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi>, > all your fields are of type Option[X]. In Scala, a value of type Option[X] > can hold one of two things: > > 1. None > 2. Some(x), where x is of type X > > So to fix your immediate problem, wrap all your parameters to the sdp_d > constructor in Some(...), as follows: > > new sdp_d(Some(r(0).trim.toInt), Some(r(1).trim.toInt), > Some(r(2).trim), ... > > Your earlier question of why writing sdp_d(...) for a case class works but > you need to write new sdp_d(...) for an explicit class, there's a simple > answer. When you create a case class X in scala, Scala also makes a > companion object X behind the scenes with an apply method that calls new > (see below). Scala's rules will call this apply method automatically. So, > when you write "X(...)", you're really calling "X.apply(...)" which in turn > calls "new X(...)". (This is the same trick behind writing things like > List(1,2,3)) If you don't use a case class, you'd have to make the > companion object yourself explicitly. > > For reference, this statement: > > case class X(a: A, b: B) > > is conceptually equivalent to > > class X(val a: A, val b: B) extends ... { > > override def toString: String = // Auto-generated > override def hashCode: Int = // Auto-generated > override def equals(that: Any): Boolean = // Auto-generated > > ... more convenience methods ... > } > > object X { > def apply(a: A, b: B) = new X(a, b) > ... more convenience methods ... > } > > If you want to peek under the hood, try compiling a simple X.scala file > with the line "case class X(a: Int, b: Double)", then taking apart the > generated X.class and X$.class (e.g., "javap X.class"). > > More info here > <http://docs.scala-lang.org/tutorials/tour/case-classes.html>, here > <http://www.scala-lang.org/docu/files/ScalaReference.pdf> and in Programming > in Scala <http://www.artima.com/shop/programming_in_scala_2ed> ch 15. > > Hope that helps! > > Best, > > Patrick > > On Thu, Feb 26, 2015 at 6:37 AM, anamika gupta <anamika.guo...@gmail.com> > wrote: > >> I am now getting the following error. I cross-checked my types and >> corrected three of them i.e. r26-->String, r27-->Timestamp, >> r28-->Timestamp. This error still persists. >> >> scala> >> sc.textFile("/home/cdhuser/Desktop/Sdp_d.csv").map(_.split(",")).map { r => >> | val upto_time = sdf.parse(r(23).trim); >> | calendar.setTime(upto_time); >> | val r23 = new java.sql.Timestamp(upto_time.getTime) >> | val insert_time = sdf.parse(r(27).trim) >> | calendar.setTime(insert_time) >> | val r27 = new java.sql.Timestamp(insert_time.getTime) >> | val last_upd_time = sdf.parse(r(28).trim) >> | calendar.setTime(last_upd_time) >> | val r28 = new java.sql.Timestamp(last_upd_time.getTime) >> | new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim, >> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim, >> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim, >> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim, >> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim, >> r(25).trim, r(26).trim, r27, r28) >> | }.registerAsTable("sdp_d") >> >> <console>:26: error: type mismatch; >> found : Int >> required: Option[Int] >> new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim, >> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim, >> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim, >> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim, >> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim, >> r(25).trim, r(26).trim, r27, r28) >> >> On Wed, Feb 25, 2015 at 2:32 PM, Akhil Das <ak...@sigmoidanalytics.com> >> wrote: >> >>> It says sdp_d not found, since it is a class you need to instantiate it >>> once. like: >>> >>> sc.textFile("derby.log").map(_.split(",")).map( r => { >>> val upto_time = sdf.parse(r(23).trim); >>> calendar.setTime(upto_time); >>> val r23 = new java.sql.Timestamp(upto_time.getTime); >>> >>> val insert_time = sdf.parse(r(26).trim); >>> calendar.setTime(insert_time); >>> val r26 = new java.sql.Timestamp(insert_time.getTime); >>> >>> val last_upd_time = sdf.parse(r(27).trim); >>> calendar.setTime(last_upd_time); >>> val r27 = new java.sql.Timestamp(last_upd_time.getTime); >>> >>> *new* *sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim, >>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim, >>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim, >>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim, >>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim, >>> r(25).trim, r26, r27, r(28).trim)* >>> }).registerAsTable("sdp") >>> >>> Thanks >>> Best Regards >>> >>> On Wed, Feb 25, 2015 at 2:14 PM, anamika gupta <anamika.guo...@gmail.com >>> > wrote: >>> >>>> The link has proved helpful. I have been able to load data, register it >>>> as a table and perform simple queries. Thanks Akhil !! >>>> >>>> Though, I still look forward to knowing where I was going wrong with my >>>> previous technique of extending the Product Interface to overcome case >>>> class's limit of 22 fields. >>>> >>>> On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta < >>>> anamika.guo...@gmail.com> wrote: >>>> >>>>> Hi Akhil >>>>> >>>>> I guess it skipped my attention. I would definitely give it a try. >>>>> >>>>> While I would still like to know what is the issue with the way I have >>>>> created schema? >>>>> >>>>> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <ak...@sigmoidanalytics.com >>>>> > wrote: >>>>> >>>>>> Did you happen to have a look at >>>>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema >>>>>> >>>>>> Thanks >>>>>> Best Regards >>>>>> >>>>>> On Tue, Feb 24, 2015 at 3:39 PM, anu <anamika.guo...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> My issue is posted here on stack-overflow. What am I doing wrong >>>>>>> here? >>>>>>> >>>>>>> >>>>>>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi >>>>>>> >>>>>>> ------------------------------ >>>>>>> View this message in context: Facing error while extending scala >>>>>>> class with Product interface to overcome limit of 22 fields in >>>>>>> spark-shell >>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html> >>>>>>> Sent from the Apache Spark User List mailing list archive >>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at >>>>>>> Nabble.com. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
