Re: [Erorr:]vieiwng Web UI on EMR cluster

Hi, I think the spark UI will be accessible whenever you launch a spark app in the cluster it should be the Application Tracker link. Regards, Natu On Tue, Sep 13, 2016 at 9:37 AM, Divya Gehlot wrote: > Hi , > Thank you all.. > Hurray ...I am able to view the hadoop

Can Spark Streaming checkpoint only metadata ?

Hi, I wonder if it is possible to checkpoint only metadata and not the data in RDD's and dataframes. Thanks, Natu

Spark not using all the cluster instances in AWS EMR

Hi, I am running some spark loads . I notice that in it only uses one of the machines(instead of the 3 available) of the cluster. Is there any parameter that can be set to force it to use all the cluster. I am using AWS EMR with Yarn. Thanks, Natu

RE: difference between dataframe and dataframwrite

Hi Does anyone know wich one aws emr uses by default? Thanks, Natu On Jun 16, 2016 5:12 PM, "David Newberger" wrote: > DataFrame is a collection of data which is organized into named columns. > > DataFrame.write is an interface for saving the contents of a

Re: concat spark dataframes

Hi, You can select the common collumns and use DataFrame.union all . Regards, Natu On Wed, Jun 15, 2016 at 8:57 PM, spR wrote: > hi, > > how to concatenate spark dataframes? I have 2 frames with certain columns. > I want to get a dataframe with columns from both the

Re: Spark Streamming checkpoint and restoring files from S3

Hi, It seems to me that the checkpoint command is not persisting the SparkContext hadoop configuration correctly . Can this be a possibility ? Thanks, Natu On Mon, Jun 13, 2016 at 11:57 AM, Natu Lauchande <nlaucha...@gmail.com> wrote: > Hi, > > I am testing disaster reco

Spark Streamming checkpoint and restoring files from S3

Hi, I am testing disaster recovery from Spark and having some issues when trying to restore an input file from s3 : 2016-06-13 11:42:52,420 [main] INFO org.apache.spark.streaming.dstream.FileInputDStream$FileInputDStreamCheckpointData - Restoring files for time 146581086 ms -

Issues when using the streaming checkpoint

Hi, I am having the following error when using checkpoint in a spark streamming app : java.io.NotSerializableException: DStream checkpointing has been enabled but the DStreams with their functions are not serializable I am following the example available in

Re: My notes on Spark Performance & Tuning Guide

Hi Mich, I am also interested in the write up. Regards, Natu On Thu, May 12, 2016 at 12:08 PM, Mich Talebzadeh wrote: > Hi Al,, > > > Following the threads in spark forum, I decided to write up on > configuration of Spark including allocation of resources and

Re: Init/Setup worker

Hi, Not sure if this might be helpful to you : https://github.com/ondra-m/ruby-spark . Regards, Natu On Tue, May 10, 2016 at 4:37 PM, Lionel PERRIN wrote: > Hello, > > > > I’m looking for a solution to use jruby on top of spark. The only tricky > point is that I

Re: DStream how many RDD's are created by batch

ry 500ms. If it’s 2 seconds then an RDD is created > every 2 seconds. > > > > Cheers, > > > > *David* > > > > *From:* Natu Lauchande [mailto:nlaucha...@gmail.com] > *Sent:* Tuesday, April 12, 2016 7:09 AM > *To:* user@spark.apache.org > *Subject:* DStr

DStream how many RDD's are created by batch

Hi, What's the criteria for the number of RDD's created for each micro bath iteration ? Thanks, Natu

Can i have a hive context and sql context in the same app ?

Hi, Is it possible to have both a sqlContext and a hiveContext in the same application ? If yes would there be any performance pernalties of doing so. Regards, Natu

Re: Unable run Spark in YARN mode

How are you trying to run spark ? locally ? spark submit ? On Sat, Apr 9, 2016 at 7:57 AM, maheshmath wrote: > I have set SPARK_LOCAL_IP=127.0.0.1 still getting below error > > 16/04/09 10:36:50 INFO spark.SecurityManager: Changing view acls to: mahesh > 16/04/09 10:36:50

Re: Monitoring S3 Bucket with Spark Streaming

t? > > Thanks, > Ben > > On Apr 8, 2016, at 9:15 PM, Natu Lauchande <nlaucha...@gmail.com> wrote: > > Hi Benjamin, > > I have done it . The critical configuration items are the ones below : > > ssc.sparkContext.hadoopConfiguratio

Re: Use only latest values

I don't see this happening without a store. You can try parquet on top of hdfs. This will at least avoid third party systems burden. On 09 Apr 2016 9:04 AM, "Daniela S" wrote: > Hi, > > I would like to cache values and to use only the latest "valid" values to > build a sum.

Re: Monitoring S3 Bucket with Spark Streaming

tifications: > http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html > > > On Fri, Apr 8, 2016 at 9:15 PM Natu Lauchande <nlaucha...@gmail.com> > wrote: > >> Hi Benjamin, >> >> I have done it . The critical configuration items are the ones

Re: Monitoring S3 Bucket with Spark Streaming

Hi Benjamin, I have done it . The critical configuration items are the ones below : ssc.sparkContext.hadoopConfiguration.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem") ssc.sparkContext.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", AccessKeyId)

Develop locally with Yarn

Hi, I working on a spark streamming app , when in local i use the "local[*]" as the master of my Spark Streamming Context . I wonder what would be need to develop locally and run it in Yarn through the IDE i am using IntelliJ idea. Thanks, Natu

Question around spark on EMR

Hi, I am setting up a Scala spark streaming app in EMR . I wonder if anyone in the list can help me with the following question : 1. What's the approach that you guys have been using to submit in an EMR job step environment variables that will be needed by the Spark application ? 2. Can i have

Question Spark streaming - S3 textFileStream- How to get the current file name ?

Hi, I am using spark streamming and using the input strategy of watching for files in S3 directories. Using the textFileStream method in the streamming context. The filename contains relevant for my pipeline manipulation i wonder if there is a more robust way to get this name other than

Re: Programatically create RDDs based on input

Hi Amit, I don't see any default constructor in the JavaRDD docs https://spark.apache.org/docs/latest/api/java/org/apache/spark/api/java/JavaRDD.html . Have you tried the following ? JavaRDD jRDD[] ; jRDD.add( jsc.textFile("/file1.txt") ) jRDD.add( jsc.textFile("/file2.txt") ) .. ; Natu On

Re: How to lookup by a key in an RDD

Hi, Looking here for the lookup function might help you: https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions Natu On Sat, Oct 31, 2015 at 6:04 PM, swetha wrote: > Hi, > > I have a requirement wherein I have to load data

Re: Cache in Spark

I don't think so. Spark is not keeping the results in memory unless you tell it too. You have to explicitly call the cache method in your RDD: linesWithSpark.cache() Thanks, Natu On Fri, Oct 9, 2015 at 10:47 AM, vinod kumar wrote: > Hi Guys, > > May I know

Re: Networking issues with Spark on EC2

Hi, Are you using EMR ? Natu On Sat, Sep 26, 2015 at 6:55 AM, SURAJ SHETH wrote: > Hi Ankur, > Thanks for the reply. > This is already done. > If I wait for a long amount of time(10 minutes), a few tasks get > successful even on slave nodes. Sometime, a fraction of the

Re: why is spark + scala code so slow, compared to python?

Are you using Scala in a distributed enviroment or in a standalone mode ? Natu On Thu, Dec 11, 2014 at 8:23 PM, ll duy.huynh@gmail.com wrote: hi.. i'm converting some of my machine learning python code into scala + spark. i haven't been able to run it on large dataset yet, but on small

Re: Ideas on how to use Spark for anomaly detection on a stream of data

This is elaborated in a chapter of an upcoming book that's available in early release; you can look at the accompanying source code to get some ideas too: https://github.com/sryza/aas/tree/master/kmeans On Mon, Nov 24, 2014 at 10:17 PM, Natu Lauchande nlaucha...@gmail.com wrote: Hi all, I am getting

Ideas on how to use Spark for anomaly detection on a stream of data

Hi all, I am getting started with Spark. I would like to use for a spike on anomaly detection in a massive stream of metrics. Can Spark easily handle this use case ? Thanks, Natu