Fetching Hive table data from external cluster

HI All, Currently using Spark 14.2 version Please provide inputs if anyone have encountered below mentioned scenario Fetching Hive table data from external Hadoop cluster into a Dataframe via Spark Job, I am interested in having data directly into a Dataframe and apply transformation on top of

unsubscribe

unsubscribe

Calender Obj to java.util.date conversion issue

HI All, Please find the below snippet referring to UDF which subtracts a day from the given date value *Snippet **:* *val* sub_a_day = udf((d:Date) => {cal.setTime(d) cal.add(Calendar.DATE, -1) cal.getTime() }) *Error *: Exception in thread "main"

Re: Spark DataFrameNaFunctions unrecognized

er details required on the same Regards, Satish Chandra On Tue, Feb 16, 2016 at 1:03 PM, Ted Yu <yuzhih...@gmail.com> wrote: > bq. I am getting compile time error > > Do you mind pastebin'ning the error you got ? > > Cheers > > On Mon, Feb 15, 2016 at 11:08 PM, satish c

Re: Spark DataFrameNaFunctions unrecognized

I tried the following using spark-shell on master branch: > > scala> df.na.fill(0) > res0: org.apache.spark.sql.DataFrame = [col: int] > > Cheers > > On Mon, Feb 15, 2016 at 3:36 AM, satish chandra j < > jsatishchan...@gmail.com> wrote: > >> Hi All, >>

Spark DataFrameNaFunctions unrecognized

ot; of DataFrame "df" to be replaced with value "" as given in the above snippet. I understand, code does not require any additional packages to support DataFrameNaFunctions Please let me know if I am missing anything so that I can make these DataFrameNaFunctions working Regards, Satish Chandra J

Re: createDataFrame question

HI, Hope you are aware of "toDF()" which is used to convert your RDD to DataFrame Regards, Satish Chandra On Tue, Feb 9, 2016 at 5:52 PM, jdkorigan wrote: > Hi, > > I would like to transform my rdd to a sql.dataframe.Dataframe, is there a > possible conversion to do the

Re: DataFrame First method is resulting different results in each iteration

8 PM, Hemant Bhanawat <hemant9...@gmail.com> wrote: > Missing order by? > > Hemant Bhanawat > SnappyData (http://snappydata.io/) > > > On Wed, Feb 3, 2016 at 3:45 PM, satish chandra j <jsatishchan...@gmail.com > > wrote: > >> HI All, >> I have d

DataFrame First method is resulting different results in each iteration

10 003 20 002 Not sure why output varies on each iteration as no change in code and values in DataFrame Please let me know if any inputs on this Regards, Satish Chandra J

Passing binding variable in query used in Data Source API

Hi All, We have requirement to fetch data from source PostgreSQL database as per a condition, hence need to pass a binding variable in query used in Data Source API as below: var DeptNbr = 10 val dataSource_dF=cc.load("jdbc",Map("url"->"jdbc:postgresql://

Re: Window Functions importing issue in Spark 1.4.0

nd "import org.apache.spark.sql.functions.rowNumber" Thanks for providing your valuable inputs Regards, Satish Chandra J On Thu, Jan 7, 2016 at 4:41 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Please take a look at the following for sample on how rowNumber is used: > https://github.com/apache/spark/pu

Window Functions importing issue in Spark 1.4.0

HI All, Currently using Spark 1.4.0 version, I have a requirement to add a column having Sequential Numbering to an existing DataFrame I understand Window Function "rowNumber" serves my purpose hence I have below import statements to include the same import org.apache.spark.sql.expressions.Window

Re: spark-submit for dependent jars

Hi Rajesh, Could you please try giving your cmd as mentioned below: ./spark-submit --master local --class --jars Regards, Satish Chandra On Mon, Dec 21, 2015 at 6:45 PM, Madabhattula Rajesh Kumar < mrajaf...@gmail.com> wrote: > Hi, > > How to add dependent jars in spark-submit command. For

Concatenate a string to a Column of type string in DataFrame

HI, I am trying to update a column value in DataFrame, incrementing a column of integer data type than the below code works val new_df=old_df.select(df("Int_Column")+10) If I implement the similar approach for appending a string to a column of string datatype as below than it does not error out

Error Handling approach for SparkSQL queries in Spark version 1.4

HI All, Any inputs on error handling approach for Spark SQL or DataFrames Thanks for all your valuable inputs in advance Regards, Satish Chandra

Re: Re: RE: Error not found value sqlContext

Thanks for all the support. It was a code issue which I overlooked it Regards, Satish Chandra On Mon, Nov 23, 2015 at 3:49 PM, satish chandra j <jsatishchan...@gmail.com> wrote: > Sorry, just to understand my issue.if Eclipse could not understand > Scala syntax properly than it

Issue while Spark Job fetching data from Cassandra DB

HI All, I am getting "*.UnauthorizedException: User has no SELECT permission on or any of its parents*" error while Spark job is fetching data from Cassandra but could able to save data into Cassandra with out any issues Note: With the same user , I could able to access and query the table in

Re: Issue while Spark Job fetching data from Cassandra DB

the table in CQL UI and code used in Spark Job has been tested in Spark Shell and it is working fine Regards, Satish Chandra On Tue, Nov 17, 2015 at 11:45 PM, satish chandra j <jsatishchan...@gmail.com > wrote: > HI All, > I am getting "*.UnauthorizedException: User has no SEL

Re: Best practises

HI All, Yes, any such doc will be a great help!!! On Fri, Oct 30, 2015 at 4:35 PM, huangzheng <1106944...@qq.com> wrote: > I have the same question.anyone help us. > > > -- 原始邮件 -- > *发件人:* "Deepak Sharma"; > *发送时间:* 2015年10月30日(星期五)

Re: JdbcRDD Constructor

let me know if any default approach Spark is going to implement if do not give any such inputs as "lowerbound" and "upperbound" to JDBCRDD Constructor or DataSourceAPI Thanks in advance for your inputs Regards, Satish Chandra J On Thu, Sep 24, 2015 at 10:18 PM, Deenar Tor

Convert SchemaRDD to RDD

Hi All, To convert SchemaRDD to RDD below snipped is working if SQL statement has columns in a row are less than 22 as per tuple restriction rdd.map(row => row.toString) But if SQL statement has columns more than 22 than the above snippet will error "*object Tuple27 is not a member of package

Re: Convert SchemaRDD to RDD

HI Ted, I have implemented the below snipped but getting an error"type mismatch found String required Serializable" as mentioned in mail chain class MyRecord(val val1: String, val val2: String, ... more then 22, in this case f.e. 26) extends Product with Serializable { def canEqual(that:

Fwd: Partition Column in JDBCRDD or Datasource API

, Satish Chandra Jummula -- Forwarded message -- From: satish chandra j <jsatishchan...@gmail.com> Date: Wed, Sep 30, 2015 at 2:10 PM Subject: Partition Column in JDBCRDD or Datasource API To: user <user@spark.apache.org> HI All, Please provide your inputs on Partition Colum

Re: Scala Limitation - Case Class definition with more than 22 arguments

Chandra On Sat, Oct 3, 2015 at 9:31 AM, satish chandra j <jsatishchan...@gmail.com> wrote: > Hi, > I am getting the below error while implementing the above custom class > code given by you > > error type mismatch: found String required: Serializable > > Please let me know if

Re: Scala Limitation - Case Class definition with more than 22 arguments

Hi, I am getting the below error while implementing the above custom class code given by you error type mismatch: found String required: Serializable Please let me know if i am missing anything here Regards, Satish Chandra On Wed, Sep 23, 2015 at 12:34 PM, Petr Novak

Fetching Date value from RDD of type spark.sql.row

HI All, Currently using Spark 1.2.2, as getDate method is not defined in *Public Class Row* for this Spark version hence trying to fetch Date value of a specific coulmn using *get* method as specified in API docs as mentioned below:

Partition Column in JDBCRDD or Datasource API

HI All, Please provide your inputs on Partition Column to be used in DataSourceAPI or JDBCRDD in a scenerio where the source table does not have a Numeric Columns which is sequential and unique such that proper partitioning can take place in Spark Regards, Satish

Re: Fetching Date value from spark.sql.row in Spark 1.2.2

HI All, If any alternate solutions to get the Date value from org.apache.spark.sql.row please suggest me Regards, Satish Chandra On Tue, Sep 29, 2015 at 4:41 PM, satish chandra j <jsatishchan...@gmail.com> wrote: > HI All, > Currently using Spark 1.2.2, as getDate method is

Fetching Date value from spark.sql.row in Spark 1.2.2

HI All, Currently using Spark 1.2.2, as getDate method is not defined in this version hence trying to fetch Date value of a specific coulmn using *get* method as specified in docs (ref URL given below:) https://spark.apache.org/docs/1.2.2/api/java/index.html?org/apache/spark/sql/api/java/Row.html

Re: JdbcRDD Constructor

ive range (<= and >=).. > > e.g. select title, author from books where ? <= id and id <= ? > > Are you doing this? > > Deenar > > > > > *Think Reactive Ltd* > deenar.toras...@thinkreactive.co.uk > 07714140812 > > > > On 23 September

Re: Scala Limitation - Case Class definition with more than 22 arguments

wrote: > +1 on grouping the case classes and creating a hierarchy – as long as you > use the data programatically. For DataFrames / SQL the other ideas probably > scale better… > > From: Ted Yu > Date: Wednesday, September 23, 2015 at 7:07 AM > To: satish chandra j >

Re: Scala Limitation - Case Class definition with more than 22 arguments

lease let me know if any work around for the same Regards, Satish Chandra On Thu, Sep 24, 2015 at 3:18 PM, satish chandra j <jsatishchan...@gmail.com> wrote: > HI All, > As it is for SQL purpose I understand, need to go ahead with Custom Case > Class approach > Could anybody

Not fetching all records from Cassandra DB

HI All, Not sure why all records are not retrieved from Cassasndra even though there are no condition applied in SQL query executed on Cassandra SQL Context in Spark 1.2.2 version Note: Its a simple lookup purpose table which has only 10 to 15 records Please let me know if any inputs on the

Re: JdbcRDD Constructor

get count = 100, irrespective of num > partitions. > > On Wed, Sep 23, 2015 at 5:00 PM, satish chandra j < > jsatishchan...@gmail.com> wrote: > >> HI, >> Currently using Spark 1.2.2, could you please let me know correct results >> output count which you got i

Re: JdbcRDD Constructor

I can get correct results using > JdbcRDD. Infact there is a test suite precisely for this (JdbcRDDSuite) . > I changed according to your input and got correct results from this test > suite. > > On Wed, Sep 23, 2015 at 11:00 AM, satish chandra j < > jsatishchan...@gmail.com>

Re: Scala Limitation - Case Class definition with more than 22 arguments

t; Andy > > On Wed, Sep 23, 2015 at 2:07 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Can you switch to 2.11 ? >> >> The following has been fixed in 2.11: >> https://issues.scala-lang.org/browse/SI-7296 >> >> Otherwise consider packaging related valu

Scala Limitation - Case Class definition with more than 22 arguments

HI All, Do we have any alternative solutions in Scala to avoid limitation in defining a Case Class having more than 22 arguments We are using Scala version 2.10.2, currently I need to define a case class with 37 arguments but getting an error as "*error: Implementation restriction: case classes

JdbcRDD Constructor

HI All, JdbcRDD constructor has following parameters, *JdbcRDD *(SparkContext

Spark SQL vs Spark Programming

HI All, As a developer I understand certain scenario's can be achieved by Spark SQL and Spark Programming(RDD transformation). More over I need to consider the below points: Performance Implementation approach Specific use cases suitable for each of the approach Could you

Re: Joining using mulitimap or array

Hi, If you join logic is correct, it seems to be a similar issue which i faced recently Can you try by *SparkContext(conf).set(spark.driver.allowMultipleContexts,true)* Regards, Satish Chandra On Mon, Aug 24, 2015 at 2:51 PM, Ilya Karpov i.kar...@cleverdata.ru wrote: Hi, guys I'm confused

Re: Transformation not happening for reduceByKey or GroupByKey

() val sc = new SparkContext(conf).set(spark.driver.allowMultipleContexts,true) val DataRDD = SC.makeRDD(Seq((0,1),(0,2),(1,2),(1,3),(2,4))) DataRDD.reduceByKey(_+_).collect Result: Array((0,3),(1,5),(2,4)) Regards, Satish Chandra On Sat, Aug 22, 2015 at 11:27 AM, satish chandra j jsatishchan

Re: Transformation not happening for reduceByKey or GroupByKey

= RDD.reduceByKey((x,y) = x+y) rdd2.take(3) -Abhishek- On Aug 20, 2015, at 3:05 AM, satish chandra j jsatishchan...@gmail.com wrote: HI All, I have data in RDD as mentioned below: RDD : Array[(Int),(Int)] = Array((0,1), (0,2),(1,20),(1,30),(2,40)) I am expecting output as Array((0,3

Re: Transformation not happening for reduceByKey or GroupByKey

@spark.apache.org Hi Satish, I don't see where spark support -i, so suspect it is provided by DSE. In that case, it might be bug of DSE. On Fri, Aug 21, 2015 at 6:02 PM, satish chandra j jsatishchan...@gmail.com wrote: HI Robin, Yes, it is DSE but issue is related to Spark only Regards, Satish

Re: Transformation not happening for reduceByKey or GroupByKey

) -Abhishek- On Aug 20, 2015, at 3:05 AM, satish chandra j jsatishchan...@gmail.com wrote: HI All, I have data in RDD as mentioned below: RDD : Array[(Int),(Int)] = Array((0,1), (0,2),(1,20),(1,30),(2,40)) I am expecting output as Array((0,3),(1,50),(2,40)) just a sum function on Values

Re: Transformation not happening for reduceByKey or GroupByKey

Yes, DSE 4.7 Regards, Satish Chandra On Fri, Aug 21, 2015 at 3:06 PM, Robin East robin.e...@xense.co.uk wrote: Not sure, never used dse - it’s part of DataStax Enterprise right? On 21 Aug 2015, at 10:07, satish chandra j jsatishchan...@gmail.com wrote: HI Robin, Yes, below mentioned

Re: Transformation not happening for reduceByKey or GroupByKey

HI Robin, Yes, it is DSE but issue is related to Spark only Regards, Satish Chandra On Fri, Aug 21, 2015 at 3:06 PM, Robin East robin.e...@xense.co.uk wrote: Not sure, never used dse - it’s part of DataStax Enterprise right? On 21 Aug 2015, at 10:07, satish chandra j jsatishchan...@gmail.com

Re: Transformation not happening for reduceByKey or GroupByKey

] at makeRDD at console:28 scala pairs.reduceByKey((x,y) = x + y).collect res43: Array[(Int, Int)] = Array((0,3), (1,50), (2,40)) On 20 Aug 2015, at 11:05, satish chandra j jsatishchan...@gmail.com wrote: HI All, I have data in RDD as mentioned below: RDD : Array[(Int),(Int)] = Array((0,1

Transformation not happening for reduceByKey or GroupByKey

HI All, I have data in RDD as mentioned below: RDD : Array[(Int),(Int)] = Array((0,1), (0,2),(1,20),(1,30),(2,40)) I am expecting output as Array((0,3),(1,50),(2,40)) just a sum function on Values for each key Code: RDD.reduceByKey((x,y) = x+y) RDD.take(3) Result in console: RDD:

Re: Transformation not happening for reduceByKey or GroupByKey

HI All, Could anybody let me know what is that i missing here, it should work as its a basic transformation Please let me know if any additional information required Regards, Satish On Thu, Aug 20, 2015 at 3:35 PM, satish chandra j jsatishchan...@gmail.com wrote: HI All, I have data in RDD

to retrive full stack trace

HI All, Please let me know if any arguments to be passed in CLI to retrieve FULL STACK TRACE in Apache Spark I am stuck in a issue for which it would be helpful to analyze full stack trace Regards, Satish Chandra

Re: saveToCassandra not working in Spark Job but works in Spark Shell

chandra j jsatishchan...@gmail.com wrote: HI, Please let me know if I am missing anything in the below mail, to get the issue fixed Regards, Satish Chandra On Wed, Aug 12, 2015 at 6:59 PM, satish chandra j jsatishchan...@gmail.com wrote: HI, The below mentioned code is working very well

Re: saveToCassandra not working in Spark Job but works in Spark Shell

HI, Please let me know if I am missing anything in the below mail, to get the issue fixed Regards, Satish Chandra On Wed, Aug 12, 2015 at 6:59 PM, satish chandra j jsatishchan...@gmail.com wrote: HI, The below mentioned code is working very well fine in Spark Shell but when the same

dse spark-submit multiple jars issue

*HI,* Please let me know if i am missing anything in the command below *Command:* dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar ///home/missingmerch/dse.jar

Re: dse spark-submit multiple jars issue

) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Regards, Satish Chandra J On Tue, Aug 11, 2015 at 6:15 PM, Javier Domingo Cansino javier.domi...@fon.com wrote: use --verbose, it might give you some insights on what0s happening, [image: Fon] http://www.fon.com/Javier Domingo

Re: Spark Cassandra Connector issue

://polyglotprogramming.com On Mon, Aug 10, 2015 at 8:23 AM, satish chandra j jsatishchan...@gmail.com wrote: Hi, Thanks for quick input, now I am getting class not found error *Command:* dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld --jars ///home/missingmerch/postgresql-9.4-1201

Re: Differents in loading data using spark datasource api and using jdbc

Hi, As I understand JDBC is meant for moderate voulme of data but Datasource api is a better option if volume of data volume is more Datasource API is not available is lower version of Spark such as 1.2.0 Regards, Satish On Tue, Aug 11, 2015 at 8:53 AM, 李铖 lidali...@gmail.com wrote:

Spark Cassandra Connector issue

HI All, Please help me to fix Spark Cassandra Connector issue, find the details below *Command:* dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar *Error:* WARN

Re: Spark Cassandra Connector issue

/product/0636920033073.do (O'Reilly) Typesafe http://typesafe.com @deanwampler http://twitter.com/deanwampler http://polyglotprogramming.com On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j jsatishchan...@gmail.com wrote: HI All, Please help me to fix Spark Cassandra Connector issue, find

Re: Spark-Submit error

...@gmail.com wrote: Hi Satish, Can you add more error or log info to the email? Guru Medasani gdm...@gmail.com On Jul 31, 2015, at 1:06 AM, satish chandra j jsatishchan...@gmail.com wrote: HI, I have submitted a Spark Job with options jars,class,master as *local* but i am getting

Spark-Submit error

HI, I have submitted a Spark Job with options jars,class,master as *local* but i am getting an error as below *dse spark-submit spark error exception in thread main java.io.ioexception: Invalid Request Exception(Why you have not logged in)* *Note: submitting datastax spark node* please let me

Spark Shell No suitable driver found error

HI All, I have issues to make external jar available to Spark Shell I have used -jars options while starting Spark Shell to make these available when I give command Class.forName(org.postgresql.Driver it is not giving any error But when action operation is performed on RDD than I am getting

[no subject]

HI All, I have issues to make external jar available to Spark Shell I have used -jars options while starting Spark Shell to make these available when I give command Class.forName(org.postgresql.Driver it is not giving any error But when action operation is performed on RDD than I am getting