Unsubscribe

Unsubscribe

Re: Spark SQL query for List

ue, Apr 26, 2016 at 4:41 PM, Hyukjin Kwon <gurwls...@gmail.com> wrote: > Doesn't get(0) give you the Array[String] for CITY (am I missing > something?) > On 26 Apr 2016 11:02 p.m., "Ramkumar V" <ramkumar.c...@gmail.com> wrote: > > JavaSparkContext ctx = new JavaS

Re: Spark SQL query for List

https://in.linkedin.com/in/ramkumarcs31> On Tue, Apr 26, 2016 at 3:48 PM, Hyukjin Kwon <gurwls...@gmail.com> wrote: > Could you maybe share your codes? > On 26 Apr 2016 9:51 p.m., "Ramkumar V" <ramkumar.c...@gmail.com> wrote: > >> Hi, >> >>

Spark SQL query for List

Hi, I had loaded JSON file in parquet format into SparkSQL. I can't able to read List which is inside JSON. Sample JSON { "TOUR" : { "CITIES" : ["Paris","Berlin","Prague"] }, "BUDJET" : 100 } I want to read value of CITIES. *Thanks*,

Re: Read Parquet in Java Spark

HI, Any idea on this ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Mon, Apr 4, 2016 at 2:47 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > I wasn't knowing you have a parquet file containing json data. > > Thanks > Best Regards > > On Mon, Apr

Re: Read Parquet in Java Spark

; > to String (the Row.mkstring() Operation) > > Thanks > Best Regards > > On Mon, Apr 4, 2016 at 12:02 PM, Ramkumar V <ramkumar.c...@gmail.com> > wrote: > >> Any idea on this ? How to convert parquet file into JavaRDD ? >> >> *Thanks*, >> <

Re: Read Parquet in Java Spark

Any idea on this ? How to convert parquet file into JavaRDD ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Thu, Mar 31, 2016 at 4:30 PM, Ramkumar V <ramkumar.c...@gmail.com> wrote: > Hi, > > Thanks for the reply. I tried this. It's returning JavaRDD instead >

Re: Read Parquet in Java Spark

u, Mar 31, 2016 at 2:57 PM, UMESH CHAUDHARY <umesh9...@gmail.com> wrote: > From Spark Documentation: > > DataFrame parquetFile = sqlContext.read().parquet("people.parquet"); > > JavaRDD jRDD= parquetFile.javaRDD() > > javaRDD() method will convert the DF to RDD &g

Read Parquet in Java Spark

Hi, I'm trying to read parquet log files in Java Spark. Parquet log files are stored in hdfs. I want to read and convert that parquet file into JavaRDD. I could able to find Sqlcontext dataframe api. How can I read if it is sparkcontext and rdd ? what is the best way to read it ? *Thanks*,

Re: Spark with MapDB

lt;fengdo...@everstring.com> wrote: > Can you detail your question? what looks like your previous batch and the > current batch? > > > > > > On Dec 8, 2015, at 3:52 PM, Ramkumar V <ramkumar.c...@gmail.com> wrote: > > Hi, > > I'm running java over spark in c

Re: Spark with MapDB

hers? > > val keys = sqlContext.read.orc(“your previous batch data > path”).select($”uniq_key”).collect > val broadCast = sc.broadCast(keys) > > val rdd = your_current_batch_data > rdd.filter( line => line.key not in broadCase.value) > > > > > > >

Re: Spark with MapDB

com/in/ramkumarcs31> On Tue, Dec 8, 2015 at 5:30 PM, Jörn Franke <jornfra...@gmail.com> wrote: > You may want to use a bloom filter for this, but make sure that you > understand how it works > > On 08 Dec 2015, at 09:44, Ramkumar V <ramkumar.c...@gmail.com> wrote: > > Im run

Spark with MapDB

Hi, I'm running java over spark in cluster mode. I want to apply filter on javaRDD based on some previous batch values. if i store those values in mapDB, is it possible to apply filter during the current batch ? *Thanks*,

Re: Distinct on key-value pair of JavaRDD

17 Nov 2015 22:00, "Ramkumar V" <ramkumar.c...@gmail.com> wrote: > >> Hi, >> >> I have JavaRDD<String,String>. I would like to do distinct only on key >> but the normal distinct applies on both key and value. i want to apply only >> on key.

Distinct on key-value pair of JavaRDD

Hi, I have JavaRDD. I would like to do distinct only on key but the normal distinct applies on both key and value. i want to apply only on key. How to do that ? Any help is appreciated. *Thanks*,

Re: Exception while reading from kafka stream

t_2 in a single createDirectStream call. > Then you can use hasOffsetRanges to see what the topic for a given > partition is. > > On Mon, Nov 2, 2015 at 7:26 AM, Ramkumar V <ramkumar.c...@gmail.com> > wrote: > >> if i try like below code snippet , it shows exception , how to avoid

Re: Exception while reading from kafka stream

t; On Fri, Oct 30, 2015 at 7:34 PM, Cody Koeninger <c...@koeninger.org> wrote: > Just put them all in one stream and switch processing based on the topic > > On Fri, Oct 30, 2015 at 6:29 AM, Ramkumar V <ramkumar.c...@gmail.com> > wrote: > >> i want to join all those l

Re: Exception while reading from kafka stream

} }); *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Fri, Oct 30, 2015 at 1:57 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > What Spark version are you using, also a small code snippet of how you use > Spark Streaming would be greatly helpful. > &

Re: Exception while reading from kafka stream

I can able to read and print few lines. Afterthat i'm getting this exception. Any idea for this ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Thu, Oct 29, 2015 at 6:14 PM, Ramkumar V <ramkumar.c...@gmail.com> wrote: > Hi, > > I'm trying to read from kafk

Re: Exception while reading from kafka stream

s this system > unstable. > > On Fri, Oct 30, 2015 at 5:13 PM, Ramkumar V <ramkumar.c...@gmail.com> > wrote: > >> No, i dont have any special settings. if i keep only reading line in my >> code, it's throwing NPE. >> >> *Thanks*, >> <https://in.linke

Re: Exception while reading from kafka stream

ill be verified at the start, unless some place changes it's value in > the runtime that makes it null, but I cannot image how this happened. Maybe > you could add some logs around the place where exception happens if you > could reproduce it. > > On Fri, Oct 30, 2015 at 5:31 PM, R

Re: Exception while reading from kafka stream

, I don't think it will > incur NPE at that place. > > On Fri, Oct 30, 2015 at 4:32 PM, Ramkumar V <ramkumar.c...@gmail.com> > wrote: > >> spark version - spark 1.4.1 >> >> my code snippet: >> >> String brokers = "ip:port,ip:port"; >

Re: Exception while reading from kafka stream

In general , i need to consume five different type of logs from kafka in spark. I have different set of topics for each log. How to start five different stream in spark ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Fri, Oct 30, 2015 at 4:40 PM, Ramkumar V <ramkumar.c...@

Re: Exception while reading from kafka stream

one > jvm, you cannot use in such way. Instead you could run multiple streaming > applications, since you're using Yarn. > > 2015年10月30日星期五，Ramkumar V <ramkumar.c...@gmail.com> 写道： > >> I found NPE is mainly because of im using the same JavaStreamingContext >> for som

Exception while reading from kafka stream

Hi, I'm trying to read from kafka stream and printing it textfile. I'm using java over spark. I dont know why i'm getting the following exception. Also exception message is very abstract. can anyone please help me ? Log Trace : 15/10/29 12:15:09 ERROR scheduler.JobScheduler: Error in job

Saving offset while reading from kafka

Hi, I had written spark streaming application using kafka stream and its writing to hdfs for every hour(batch time). I would like to know how to get offset or commit offset of kafka stream while writing to hdfs so that if there is any issue or redeployment, i'll start from the point where i did a

Conf setting for Java Spark

Hi, I'm using java over spark for processing 30 GB of data every hour. I'm doing spark-submit in cluster mode. I have a cluster of 11 machines (9 - 64 GB memory and 2 - 32 GB memory ) but it takes 30 mins to process 30 GB of data every hour. How can i optimize this ? How to compute the driver and

Re: OutOfMemoryError

How to increase the Xmx of the workers ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Mon, Oct 5, 2015 at 3:48 PM, Ramkumar V <ramkumar.c...@gmail.com> wrote: > No. I didn't try to increase xmx. > > *Thanks*, > <https://in.linkedin.com/in/ramkumarcs31>

OutOfMemoryError

Hi, When i submit java spark job in cluster mode, i'm getting following exception. *LOG TRACE :* INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms1024m, -Xmx1024m, -Djava.io.tmpdir={{PWD}}/tmp,

Re: OutOfMemoryError

No. I didn't try to increase xmx. *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Mon, Oct 5, 2015 at 1:36 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Ramkumar, > > did you try to increase Xmx of the workers ? > > Regards > JB > > On 1

Combine key-value pair in spark java

Hi, I have key value pair of JavaRDD (JavaPairRDD rdd) but i want to concatenate into one RDD String (JavaRDD result ). How can i do that ? What i have to use (map,flatmap)? can anyone please give me the syntax for this in java ? *Thanks*,

Re: Combine key-value pair in spark java

rds, > Andy (Nam) Dang > > On Wed, Sep 30, 2015 at 10:34 AM, Ramkumar V <ramkumar.c...@gmail.com> > wrote: > >> Hi, >> >> I have key value pair of JavaRDD (JavaPairRDD<String, String> rdd) but i >> want to concatenate into one RDD String (Java

GroupBy Java objects in Java Spark

Hi, I want to know whether grouping by java class objects is possible or not in java Spark. I have Tuple2< JavaObject, JavaObject>. i want to groupbyKey and then i'll do some operations in values after grouping. *Thanks*,

Re: Spark Streaming failing on YARN Cluster

? You can use command yarn logs -your_app_id to get the yarn app logs. On Thu, Aug 20, 2015 at 1:15 AM, Ramkumar V ramkumar.c...@gmail.com wrote: I'm getting some spark exception. Please look this log trace ( *http://pastebin.com/xL9jaRUa http://pastebin.com/xL9jaRUa* ). *Thanks

Re: Spark Streaming failing on YARN Cluster

issues with the files getting distributed to the cluster. What is the exception you are getting now? On Wednesday, August 19, 2015, Ramkumar V ramkumar.c...@gmail.com wrote: Thanks a lot for your suggestion. I had modified HADOOP_CONF_DIR in spark-env.sh so that core-site.xml is under

Re: Spark Streaming failing on YARN Cluster

is the environment variable point to the hadoop conf directory. Not sure how CDH organize that, make sure core-site.xml is under HADOOP_CONF_DIR. On Wed, Aug 19, 2015 at 4:06 PM, Ramkumar V ramkumar.c...@gmail.com wrote: We are using Cloudera-5.3.1. since it is one of the earlier version

Re: Spark Streaming failing on YARN Cluster

On Thu, Aug 13, 2015 at 2:50 PM, Ramkumar V ramkumar.c...@gmail.com wrote: Hi, I have a cluster of 1 master and 2 slaves. I'm running a spark streaming in master and I want to utilize all nodes in my cluster. i had specified some parameters like driver memory and executor memory in my code. when

Spark Streaming failing on YARN Cluster

, it gives the following error. Log link : *http://pastebin.com/kfyVWDGR http://pastebin.com/kfyVWDGR* How to fix this issue ? Please help me if i'm doing wrong. *Thanks*, Ramkumar V

Re: Spark Streaming failing on YARN Cluster

...@sigmoidanalytics.com wrote: Just make sure this file is available: appattempt_1437639737006_3808_02 exited with exitCode: -1000 due to: File *file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip* does not exist Thanks Best Regards On Thu, Aug 13, 2015 at 12:20 PM, Ramkumar V ramkumar.c