Re: Scala Limitation - Case Class definition with more than 22 arguments
Hi Petr,
Could you please let me know if I am missing anything on the code as my
code is as same as snippet shared by you but still i am getting the below
error:
*error type mismatch: found String required: Serializable*
Please let me know if any fix to be applied on this
Regards,
Satish Chandra
On Sat, Oct 3, 2015 at 9:31 AM, satish chandra j
wrote:
> Hi,
> I am getting the below error while implementing the above custom class
> code given by you
>
> error type mismatch: found String required: Serializable
>
> Please let me know if i am missing anything here
>
> Regards,
> Satish Chandra
>
> On Wed, Sep 23, 2015 at 12:34 PM, Petr Novak wrote:
>
>> You can implement your own case class supporting more then 22 fields. It
>> is something like:
>>
>> class MyRecord(val val1: String, val val2: String, ... more then 22, in this
>> case f.e. 26)
>> extends Product with Serializable {
>>
>> def canEqual(that: Any): Boolean = that.isInstanceOf[MyRecord]
>>
>> def productArity: Int = 26 // example value, it is amount of arguments
>>
>> def productElement(n: Int): Serializable = n match {
>> case 1 => val1
>> case 2 => val2
>> //... cases up to 26
>> }
>> }
>>
>> You can google it for more details.
>>
>> Petr
>>
>
>
Re: Scala Limitation - Case Class definition with more than 22 arguments
Hi,
I am getting the below error while implementing the above custom class code
given by you
error type mismatch: found String required: Serializable
Please let me know if i am missing anything here
Regards,
Satish Chandra
On Wed, Sep 23, 2015 at 12:34 PM, Petr Novak wrote:
> You can implement your own case class supporting more then 22 fields. It
> is something like:
>
> class MyRecord(val val1: String, val val2: String, ... more then 22, in this
> case f.e. 26)
> extends Product with Serializable {
>
> def canEqual(that: Any): Boolean = that.isInstanceOf[MyRecord]
>
> def productArity: Int = 26 // example value, it is amount of arguments
>
> def productElement(n: Int): Serializable = n match {
> case 1 => val1
> case 2 => val2
> //... cases up to 26
> }
> }
>
> You can google it for more details.
>
> Petr
>
Re: Scala Limitation - Case Class definition with more than 22 arguments
While case classes no longer have the 22-element limitation as of Scala
2.11, tuples are still limited to 22 elements. For various technical
reasons, this limitation probably won't be removed any time soon.
However, you can nest tuples, like case classes, in most contexts. So, the
last bit of your example,
(r: ResultSet) => (r.getInt("col1"),r.getInt("col2")...r.getInt("col37")
)
could add nested () to group elements and keep the outer number of elements
<= 22.
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Thu, Sep 24, 2015 at 6:01 AM, satish chandra j
wrote:
> HI All,
>
> In addition to Case Class limitation in Scala, I finding Tuple limitation
> too please find the explanation below
>
> //Query to pull data from Source Table
>
> var SQL_RDD= new JdbcRDD( sc, ()=>
> DriverManager.getConnection(url,user,pass),"select col1, col2,
> col3..col 37 from schema.Table LIMIT ? OFFSET ?",100,0,*1*,(r:
> ResultSet) => (r.getInt("col1"),r.getInt("col2")...r.getInt("col37")))
>
>
> //Define Case Class
>
> case class sqlrow(col1:Int,col2:Int..col37)
>
>
> var SchSQL= SQL_RDD.map(p => new sqlrow(p._1,p._2.p._37))
>
>
> followed by apply CreateSchema to RDD and than apply registerTempTable for
> defining a table to make use in SQL Context in Spark
>
> As per the above SQL query I need to fetch 37 columns from the source
> table, but it seems Scala has tuple restriction which I am defining by r
> ResultSet variable in the above SQL, please let me know if any work around
> for the same
>
> Regards,
> Satish Chandra
>
> On Thu, Sep 24, 2015 at 3:18 PM, satish chandra j <
> jsatishchan...@gmail.com> wrote:
>
>> HI All,
>> As it is for SQL purpose I understand, need to go ahead with Custom Case
>> Class approach
>> Could anybody have a sample code for creating Custom Case Class to refer
>> which would be really helpful
>>
>> Regards,
>> Satish Chandra
>>
>> On Thu, Sep 24, 2015 at 2:51 PM, Adrian Tanase wrote:
>>
>>> +1 on grouping the case classes and creating a hierarchy – as long as
>>> you use the data programatically. For DataFrames / SQL the other ideas
>>> probably scale better…
>>>
>>> From: Ted Yu
>>> Date: Wednesday, September 23, 2015 at 7:07 AM
>>> To: satish chandra j
>>> Cc: user
>>> Subject: Re: Scala Limitation - Case Class definition with more than 22
>>> arguments
>>>
>>> Can you switch to 2.11 ?
>>>
>>> The following has been fixed in 2.11:
>>> https://issues.scala-lang.org/browse/SI-7296
>>>
>>> Otherwise consider packaging related values into a case class of their
>>> own.
>>>
>>> On Tue, Sep 22, 2015 at 8:48 PM, satish chandra j <
>>> jsatishchan...@gmail.com> wrote:
>>>
>>>> HI All,
>>>> Do we have any alternative solutions in Scala to avoid limitation in
>>>> defining a Case Class having more than 22 arguments
>>>>
>>>> We are using Scala version 2.10.2, currently I need to define a case
>>>> class with 37 arguments but getting an error as "*error:Implementation
>>>> restriction:caseclasses cannot have more than 22parameters.*"
>>>>
>>>> It would be a great help if any inputs on the same
>>>>
>>>> Regards,
>>>> Satish Chandra
>>>>
>>>>
>>>>
>>>
>>
>
Re: Scala Limitation - Case Class definition with more than 22 arguments
HI All,
In addition to Case Class limitation in Scala, I finding Tuple limitation
too please find the explanation below
//Query to pull data from Source Table
var SQL_RDD= new JdbcRDD( sc, ()=>
DriverManager.getConnection(url,user,pass),"select col1, col2,
col3..col 37 from schema.Table LIMIT ? OFFSET ?",100,0,*1*,(r:
ResultSet) => (r.getInt("col1"),r.getInt("col2")...r.getInt("col37")))
//Define Case Class
case class sqlrow(col1:Int,col2:Int..col37)
var SchSQL= SQL_RDD.map(p => new sqlrow(p._1,p._2.p._37))
followed by apply CreateSchema to RDD and than apply registerTempTable for
defining a table to make use in SQL Context in Spark
As per the above SQL query I need to fetch 37 columns from the source
table, but it seems Scala has tuple restriction which I am defining by r
ResultSet variable in the above SQL, please let me know if any work around
for the same
Regards,
Satish Chandra
On Thu, Sep 24, 2015 at 3:18 PM, satish chandra j
wrote:
> HI All,
> As it is for SQL purpose I understand, need to go ahead with Custom Case
> Class approach
> Could anybody have a sample code for creating Custom Case Class to refer
> which would be really helpful
>
> Regards,
> Satish Chandra
>
> On Thu, Sep 24, 2015 at 2:51 PM, Adrian Tanase wrote:
>
>> +1 on grouping the case classes and creating a hierarchy – as long as you
>> use the data programatically. For DataFrames / SQL the other ideas probably
>> scale better…
>>
>> From: Ted Yu
>> Date: Wednesday, September 23, 2015 at 7:07 AM
>> To: satish chandra j
>> Cc: user
>> Subject: Re: Scala Limitation - Case Class definition with more than 22
>> arguments
>>
>> Can you switch to 2.11 ?
>>
>> The following has been fixed in 2.11:
>> https://issues.scala-lang.org/browse/SI-7296
>>
>> Otherwise consider packaging related values into a case class of their
>> own.
>>
>> On Tue, Sep 22, 2015 at 8:48 PM, satish chandra j <
>> jsatishchan...@gmail.com> wrote:
>>
>>> HI All,
>>> Do we have any alternative solutions in Scala to avoid limitation in
>>> defining a Case Class having more than 22 arguments
>>>
>>> We are using Scala version 2.10.2, currently I need to define a case
>>> class with 37 arguments but getting an error as "*error:Implementation
>>> restriction:caseclasses cannot have more than 22parameters.*"
>>>
>>> It would be a great help if any inputs on the same
>>>
>>> Regards,
>>> Satish Chandra
>>>
>>>
>>>
>>
>
Re: Scala Limitation - Case Class definition with more than 22 arguments
HI All, As it is for SQL purpose I understand, need to go ahead with Custom Case Class approach Could anybody have a sample code for creating Custom Case Class to refer which would be really helpful Regards, Satish Chandra On Thu, Sep 24, 2015 at 2:51 PM, Adrian Tanase wrote: > +1 on grouping the case classes and creating a hierarchy – as long as you > use the data programatically. For DataFrames / SQL the other ideas probably > scale better… > > From: Ted Yu > Date: Wednesday, September 23, 2015 at 7:07 AM > To: satish chandra j > Cc: user > Subject: Re: Scala Limitation - Case Class definition with more than 22 > arguments > > Can you switch to 2.11 ? > > The following has been fixed in 2.11: > https://issues.scala-lang.org/browse/SI-7296 > > Otherwise consider packaging related values into a case class of their own. > > On Tue, Sep 22, 2015 at 8:48 PM, satish chandra j < > jsatishchan...@gmail.com> wrote: > >> HI All, >> Do we have any alternative solutions in Scala to avoid limitation in >> defining a Case Class having more than 22 arguments >> >> We are using Scala version 2.10.2, currently I need to define a case >> class with 37 arguments but getting an error as "*error:Implementation >> restriction:caseclasses cannot have more than 22parameters.*" >> >> It would be a great help if any inputs on the same >> >> Regards, >> Satish Chandra >> >> >> >
Re: Scala Limitation - Case Class definition with more than 22 arguments
+1 on grouping the case classes and creating a hierarchy – as long as you use the data programatically. For DataFrames / SQL the other ideas probably scale better… From: Ted Yu Date: Wednesday, September 23, 2015 at 7:07 AM To: satish chandra j Cc: user Subject: Re: Scala Limitation - Case Class definition with more than 22 arguments Can you switch to 2.11 ? The following has been fixed in 2.11: https://issues.scala-lang.org/browse/SI-7296 Otherwise consider packaging related values into a case class of their own. On Tue, Sep 22, 2015 at 8:48 PM, satish chandra j mailto:jsatishchan...@gmail.com>> wrote: HI All, Do we have any alternative solutions in Scala to avoid limitation in defining a Case Class having more than 22 arguments We are using Scala version 2.10.2, currently I need to define a case class with 37 arguments but getting an error as "error:Implementation restriction:caseclasses cannot have more than 22parameters." It would be a great help if any inputs on the same Regards, Satish Chandra
Re: Scala Limitation - Case Class definition with more than 22 arguments
Yes and I would recommend it because it can be made generic and reusable too On Wed, Sep 23, 2015 at 5:37 PM, satish chandra j wrote: > HI Andy, > So I believe if I opt pro grammatically building the schema approach, than > it would not have have any restriction as such in "case Class not allowing > more than 22 Arguments" > > As I need to define a schema of around 37 arguments > > Regards, > Satish Chandra > > On Wed, Sep 23, 2015 at 9:50 AM, Andy Huang > wrote: > >> Alternatively, I would suggest you looking at programmatically building >> the schema >> >> refer to >> http://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema >> >> Cheers >> Andy >> >> On Wed, Sep 23, 2015 at 2:07 PM, Ted Yu wrote: >> >>> Can you switch to 2.11 ? >>> >>> The following has been fixed in 2.11: >>> https://issues.scala-lang.org/browse/SI-7296 >>> >>> Otherwise consider packaging related values into a case class of their >>> own. >>> >>> On Tue, Sep 22, 2015 at 8:48 PM, satish chandra j < >>> jsatishchan...@gmail.com> wrote: >>> HI All, Do we have any alternative solutions in Scala to avoid limitation in defining a Case Class having more than 22 arguments We are using Scala version 2.10.2, currently I need to define a case class with 37 arguments but getting an error as "*error: Implementation restriction: case classes cannot have more than 22 parameters.*" It would be a great help if any inputs on the same Regards, Satish Chandra >>> >> >> >> -- >> Andy Huang | Managing Consultant | Servian Pty Ltd | t: 02 9376 0700 | >> f: 02 9376 0730| m: 0433221979 >> > > -- Andy Huang | Managing Consultant | Servian Pty Ltd | t: 02 9376 0700 | f: 02 9376 0730| m: 0433221979
Re: Scala Limitation - Case Class definition with more than 22 arguments
HI Andy, So I believe if I opt pro grammatically building the schema approach, than it would not have have any restriction as such in "case Class not allowing more than 22 Arguments" As I need to define a schema of around 37 arguments Regards, Satish Chandra On Wed, Sep 23, 2015 at 9:50 AM, Andy Huang wrote: > Alternatively, I would suggest you looking at programmatically building > the schema > > refer to > http://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema > > Cheers > Andy > > On Wed, Sep 23, 2015 at 2:07 PM, Ted Yu wrote: > >> Can you switch to 2.11 ? >> >> The following has been fixed in 2.11: >> https://issues.scala-lang.org/browse/SI-7296 >> >> Otherwise consider packaging related values into a case class of their >> own. >> >> On Tue, Sep 22, 2015 at 8:48 PM, satish chandra j < >> jsatishchan...@gmail.com> wrote: >> >>> HI All, >>> Do we have any alternative solutions in Scala to avoid limitation in >>> defining a Case Class having more than 22 arguments >>> >>> We are using Scala version 2.10.2, currently I need to define a case >>> class with 37 arguments but getting an error as "*error: Implementation >>> restriction: case classes cannot have more than 22 parameters.*" >>> >>> It would be a great help if any inputs on the same >>> >>> Regards, >>> Satish Chandra >>> >>> >>> >> > > > -- > Andy Huang | Managing Consultant | Servian Pty Ltd | t: 02 9376 0700 | > f: 02 9376 0730| m: 0433221979 >
Re: Scala Limitation - Case Class definition with more than 22 arguments
If you need to understand what is the magic Product then google up
Algebraic Data Types and learn it together with what is Sum type. One
option is http://www.stephanboyer.com/post/18/algebraic-data-types
Enjoy,
Petr
On Wed, Sep 23, 2015 at 9:07 AM, Petr Novak wrote:
> I'm unsure if it completely equivalent to a case class and if it has some
> limitations compared to case class or if it needs some more methods
> implemented.
>
> Petr
>
> On Wed, Sep 23, 2015 at 9:04 AM, Petr Novak wrote:
>
>> You can implement your own case class supporting more then 22 fields. It
>> is something like:
>>
>> class MyRecord(val val1: String, val val2: String, ... more then 22, in this
>> case f.e. 26)
>> extends Product with Serializable {
>>
>> def canEqual(that: Any): Boolean = that.isInstanceOf[MyRecord]
>>
>> def productArity: Int = 26 // example value, it is amount of arguments
>>
>> def productElement(n: Int): Serializable = n match {
>> case 1 => val1
>> case 2 => val2
>> //... cases up to 26
>> }
>> }
>>
>> You can google it for more details.
>>
>> Petr
>>
>
>
Re: Scala Limitation - Case Class definition with more than 22 arguments
I'm unsure if it completely equivalent to a case class and if it has some
limitations compared to case class or if it needs some more methods
implemented.
Petr
On Wed, Sep 23, 2015 at 9:04 AM, Petr Novak wrote:
> You can implement your own case class supporting more then 22 fields. It
> is something like:
>
> class MyRecord(val val1: String, val val2: String, ... more then 22, in this
> case f.e. 26)
> extends Product with Serializable {
>
> def canEqual(that: Any): Boolean = that.isInstanceOf[MyRecord]
>
> def productArity: Int = 26 // example value, it is amount of arguments
>
> def productElement(n: Int): Serializable = n match {
> case 1 => val1
> case 2 => val2
> //... cases up to 26
> }
> }
>
> You can google it for more details.
>
> Petr
>
Re: Scala Limitation - Case Class definition with more than 22 arguments
You can implement your own case class supporting more then 22 fields. It is
something like:
class MyRecord(val val1: String, val val2: String, ... more then 22,
in this case f.e. 26)
extends Product with Serializable {
def canEqual(that: Any): Boolean = that.isInstanceOf[MyRecord]
def productArity: Int = 26 // example value, it is amount of arguments
def productElement(n: Int): Serializable = n match {
case 1 => val1
case 2 => val2
//... cases up to 26
}
}
You can google it for more details.
Petr
Re: Scala Limitation - Case Class definition with more than 22 arguments
Alternatively, I would suggest you looking at programmatically building the schema refer to http://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema Cheers Andy On Wed, Sep 23, 2015 at 2:07 PM, Ted Yu wrote: > Can you switch to 2.11 ? > > The following has been fixed in 2.11: > https://issues.scala-lang.org/browse/SI-7296 > > Otherwise consider packaging related values into a case class of their own. > > On Tue, Sep 22, 2015 at 8:48 PM, satish chandra j < > jsatishchan...@gmail.com> wrote: > >> HI All, >> Do we have any alternative solutions in Scala to avoid limitation in >> defining a Case Class having more than 22 arguments >> >> We are using Scala version 2.10.2, currently I need to define a case >> class with 37 arguments but getting an error as "*error: Implementation >> restriction: case classes cannot have more than 22 parameters.*" >> >> It would be a great help if any inputs on the same >> >> Regards, >> Satish Chandra >> >> >> > -- Andy Huang | Managing Consultant | Servian Pty Ltd | t: 02 9376 0700 | f: 02 9376 0730| m: 0433221979
Re: Scala Limitation - Case Class definition with more than 22 arguments
Can you switch to 2.11 ? The following has been fixed in 2.11: https://issues.scala-lang.org/browse/SI-7296 Otherwise consider packaging related values into a case class of their own. On Tue, Sep 22, 2015 at 8:48 PM, satish chandra j wrote: > HI All, > Do we have any alternative solutions in Scala to avoid limitation in > defining a Case Class having more than 22 arguments > > We are using Scala version 2.10.2, currently I need to define a case class > with 37 arguments but getting an error as "*error: Implementation > restriction: case classes cannot have more than 22 parameters.*" > > It would be a great help if any inputs on the same > > Regards, > Satish Chandra > > >
