any support to use Spark UDF in HIVE

HI , I have seen many hive udf are getting used in spark SQL,so is there any way to do it reverse.I want to write some code on spark for UDF and the same can be used in HIVE. please suggest me all possible approaches in spark with JAVA. Thaks in advance. Regards, Manoh -- View this message

javaRDD to collectasMap throuwa ava.lang.NegativeArraySizeException

HI All, getting the below Exception while converting my rdd to Map below is the code.and my data size is hardly 200MD snappy file and the code looks like this @SuppressWarnings("unchecked") public Tuple2, String> getMatchData(String location, String key) {

Class Cast Exception while read from GS and write to S3.I feel gettng while writeing to s3.

Hi All, able to run my simple spark job Read and write to S3 in local ,when i move to cluster gettng below cast exception.Spark Environment a using 2.0.1. please help out if any has faced this kind of issue already. 02/18 10:35:23 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,

JavaRDD text matadata(file name) findings

Hi All, myspark job is reading data from a folder having different files with same structured data. the red JavaRdd processed line by line but is there any way to know from which file the line of data came. Team thank you in advance for your reply coming. Thanks, -- View this message in

Will be in around 12:30pm due to some personal stuff

Get Outlook for Android Happiest Minds Disclaimer This message is for the sole use of the intended recipient(s) and may contain confidential, proprietary or legally privileged information. Any unauthorized review, use, disclosure or

Spark Read from Google store and save in AWS s3

Hi All, Using spark is interoperability communication between two clouds(Google,AWS) possible. in my use case i need to take Google store as input to spark and do some processing and finally needs to store in S3 and my spark engine runs on AWS Cluster. Please let me back is there any way for

Spark java with Google Store

Hi Team, Can some please share any examples on spark java read and write files from Google Store. Thanks You in advance. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-java-with-Google-Store-tp28276.html Sent from the Apache Spark User List

Spark Version upgrade isue:Exception in thread main java.lang.NoSuchMethodError

Hi Team, I upgraded spark older versions to 1.4.1 after maven build i tried to ran my simple application but it failed and giving the below stacktrace. Exception in thread main java.lang.NoSuchMethodError:

DataFrame InsertIntoJdbc() Runtime Exception on cluster

Hi All, Am trying to add few new rows for existing table in mysql using DataFrame.But it is adding new rows to the table in local environment but on spark cluster below is the runtime exception. Exception in thread main java.lang.RuntimeException: Table msusers_1 already exists. at

RE: lower and upper offset not working in spark with mysql database

I think you should mention partitionColumn like below and the Colum type should be numeric. It works for my case. options.put(partitionColumn, revision); Thanks, Manohar From: Hafiz Mujadid [via Apache Spark User List] [mailto:ml-node+s1001560n23635...@n3.nabble.com] Sent: Monday,

JDBCRDD sync with mssql

Hi Team, in my usecase i need to sync the data with mssql for any operation in mssql.but as per my spark knowledge we have JDBCRDD it will read data from rdbms tables with upper and lower limits. someone please help is there any API to sync data automatically from single rdbms table for any DML

JavaDStreamString read and write rdbms

Hi Team, How to split and put the red JavaDStreamString in to mysql in java. any existing api in sark 1.3/1.4. team can you please share the code snippet if any body have it. Thanks, Manohar -- View this message in context:

N kafka topics vs N spark Streaming

Hi Everybody, I have four kafks topics each for separateoperation(Add,Delete,Update,Merge). so spark also will have four consumed streams,so how we can run my spark job here? should i run four spark jobs separately? is there any way to bundle all streams into singlejar and run as single Job?

how to maintain the offset for spark streaming if HDFS is the source

Hi All, In my usecase HDFS file as source for Spark Stream, the job will process the data line by line but how will make sure to maintain the offset line number(data already processed) while restarting/new code push . Team can you please reply on this is there any configuration in Spark.