Cache in Spark

Hi Guys, May I know whether cache is enabled in spark by default? Thanks, Vinod

Re: Cache in Spark

s in memory unless you tell it too. > > You have to explicitly call the cache method in your RDD: > linesWithSpark.cache() > > Thanks, > Natu > > > > > On Fri, Oct 9, 2015 at 10:47 AM, vinod kumar <vinodsachin...@gmail.com> > wrote: > >> H

Buffer Overflow exception

Hi, I am getting buffer over flow exception while using spark via thrifserver base.May I know how to overcome this? Code: HqlConnection con = new HqlConnection(localhost, 10001, HiveServer.HiveServer2); con.Open(); HqlCommand createCommand = new HqlCommand(tablequery,

Functions in Spark SQL

Hi, May I know how to use the functions mentioned in http://spark.apache.org/docs/1.4.0/api/scala/index.html#org.apache.spark.sql.functions$ in spark sql? when I use like Select last(column) from tablename I am getting error like 15/07/27 03:00:00 INFO exec.FunctionRegistry: Unable to lookup

Re: Functions in Spark SQL

code here ? And which version of Spark are u using ? Best, Sun. -- fightf...@163.com *From:* vinod kumar vinodsachin...@gmail.com *Date:* 2015-07-27 15:04 *To:* User user@spark.apache.org *Subject:* Functions in Spark SQL Hi, May I know how to use

Create table from local machine

Hi, I am in need to create a table in spark.for that I have uploaded a csv file in HDFS and created a table using following query CREATE EXTERNAL table IF NOT EXISTS + tableName + (teams string,runs int) + ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION ' + hdfspath + '; May I know is

SQL Server to Spark

Hi Everyone, I am in need to use the table from MsSQLSERVER in SPARK.Any one please share me the optimized way for that? Thanks in advance, Vinod

Re: Spark Intro

at 3:50 AM, Akhil Das ak...@sigmoidanalytics.com wrote: This is where you can get started https://spark.apache.org/docs/latest/sql-programming-guide.html Thanks Best Regards On Mon, Jul 13, 2015 at 3:54 PM, vinod kumar vinodsachin...@gmail.com wrote: Hi Everyone, I am developing

Re: Spark Intro

are nothing. Thanks Best Regards On Tue, Jul 14, 2015 at 3:09 PM, vinod kumar vinodsachin...@gmail.com wrote: Hi Akhil Is my choice to switch to spark is good? because I don't have enough information regards limitation and working environment of spark. I tried spark SQL but it seems

Spark Intro

Hi Everyone, I am developing application which handles bulk of data around millions(This may vary as per user's requirement) records.As of now I am using MsSqlServer as back-end and it works fine but when I perform some operation on large data I am getting overflow exceptions.I heard about spark

Caching in spark

Hi Guys, Can any one please share me how to use caching feature of spark via spark sql queries? -Vinod

SPARK vs SQL

Hi Everyone, Is there is any document/material which compares spark with SQL server? If so please share me the details. Thanks, Vinod

Re: Data Processing speed SQL Vs SPARK

For records below 50,000 SQL is better right? On Fri, Jul 10, 2015 at 12:18 AM, ayan guha guha.a...@gmail.com wrote: With your load, either should be fine. I would suggest you to run couple of quick prototype. Best Ayan On Fri, Jul 10, 2015 at 2:06 PM, vinod kumar vinodsachin

Re: Data Processing speed SQL Vs SPARK

you would want to process? On 9 Jul 2015 22:28, vinod kumar vinodsachin...@gmail.com wrote: Hi Everyone, I am new to spark. Am using SQL in my application to handle data in my application.I have a thought to move to spark now. Is data processing speed of spark better than SQL server

Data Processing speed SQL Vs SPARK

Hi Everyone, I am new to spark. Am using SQL in my application to handle data in my application.I have a thought to move to spark now. Is data processing speed of spark better than SQL server? Thank, Vinod

Re: UDF in spark

wrote: Hi, sqlContext.udf.register(udfname, functionname _) example: def square(x:Int):Int = { x * x} register udf as below sqlContext.udf.register(square,square _) Thanks, Vishnu On Wed, Jul 8, 2015 at 2:23 PM, vinod kumar vinodsachin...@gmail.com wrote: Hi Everyone, I am new

Re: UDF in spark

, VISHNU SUBRAMANIAN johnfedrickena...@gmail.com wrote: HI Vinod, Yes If you want to use a scala or python function you need the block of code. Only Hive UDF's are available permanently. Thanks, Vishnu On Wed, Jul 8, 2015 at 5:17 PM, vinod kumar vinodsachin...@gmail.com wrote: Thanks

UDF in spark

Hi Everyone, I am new to spark.may I know how to define and use User Define Function in SPARK SQL. I want to use defined UDF by using sql queries. My Environment Windows 8 spark 1.3.1 Warm Regards, Vinod