Re: test mail

[Sudhanshu Janghel]

test email recieved ;p

On 4 Jul 2017 7:40 am, "Sudha KS"  wrote:

-- 

*Disclaimer: The information in this email is confidential and may be 
legally privileged. Access to this email by anyone other than the intended 
addressee is unauthorized. If you are not the intended recipient of this 
message, any review, disclosure, copying, distribution, retention, or any 
action taken or omitted to be taken in reliance on it is prohibited and may 
be unlawful.*

Unsubscribe

[Sudhanshu Janghel]

Hello,

I wish to unsubscribe from the channel.

KIND REGARDS,
SUDHANSHU

UNSUBSCRIBE

[Sudhanshu Janghel]

building kafka project on intellij Help is much appreciated

[Sudhanshu Janghel]

Hello,

I am new to building kafka and wish to understand how to make fat jars in
intellij.
The sbt assembly seems confusing and I am unable to resolve the
dependencies.

here is my build.sbt

name := "twitter"

version := "1.0"
scalaVersion := "2.10.4"



//libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.7" % "provided"
//libraryDependencies += "org.slf4j" % "slf4j-log4j12" % "1.7.7" % "provided"
//libraryDependencies += "com.google.guava" % "guava" % "11.0.2"
exclude("log4j", "log4j") exclude("org.slf4j","slf4j-log4j12")
exclude("org.slf4j","slf4j-api")
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.0"
libraryDependencies +=   "org.apache.kafka" %% "kafka"  % "1.6.0"
libraryDependencies += "org.apache.spark" %
"spark-streaming-kafka_2.10" % "1.6.0"



adn here is my assembly.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")


the error faced is

Error:Error while importing SBT project:...[info]
Resolving org.scala-sbt#tracking;0.13.8 ...
[info] Resolving org.scala-sbt#cache;0.13.8 ...
[info] Resolving org.scala-sbt#testing;0.13.8 ...
[info] Resolving org.scala-sbt#test-agent;0.13.8 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving org.scala-sbt#main-settings;0.13.8 ...
[info] Resolving org.scala-sbt#apply-macro;0.13.8 ...
[info] Resolving org.scala-sbt#command;0.13.8 ...
[info] Resolving org.scala-sbt#logic;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_8_2;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_9_2;0.13.8 ...
[info] Resolving org.scala-sbt#precompiled-2_9_3;0.13.8 ...
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for
the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency:
com.eed3si9n#sbt-assembly;0.14.3: not found
[error] unresolved dependency: org.apache.kafka#kafka_2.10;1.6.0: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved
dependency: com.eed3si9n#sbt-assembly;0.14.3: not found
[error] unresolved dependency: org.apache.kafka#kafka_2.10;1.6.0: not found
[error] Total time: 18 s, completed Apr 7, 2016 5:05:06 PM

Re: Help: Get Timeout error and FileNotFoundException when shuffling large files

[Sudhanshu Janghel]

Can you please paste the stack trace.

Sudhanshu

Re: Can't filter

[Sudhanshu Janghel]

be sure to mention the class name using the *--class* parameter to
spark-submit ..

I see no other reason for a "class not found" exception.


Sudhanshu

On Thu, Dec 10, 2015 at 11:50 AM, Harsh J  wrote:

> Are you sure you do not have any messages preceding the trace, such as one
> quoting which class is found to be missing? That'd be helpful to see and
> suggest what may (exactly) be going wrong. It appear similar to
> https://issues.apache.org/jira/browse/SPARK-8368, but I cannot tell for
> certain cause I don't know if your code uses the SparkSQL features.
>
> Also, what version is your Spark running?
>
> I am able to run your program without a problem in Spark 1.5.x (with a
> sample Seq).
>
> On Thu, Dec 10, 2015 at 5:01 PM Бобров Виктор  wrote:
>
>> 0 = {StackTraceElement@7132}
>> "com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.a(Unknown
>> Source)"
>>
>> 1 = {StackTraceElement@7133}
>> "com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.(Unknown
>> Source)"
>>
>> 2 = {StackTraceElement@7134}
>> "org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:40)"
>>
>> 3 = {StackTraceElement@7135}
>> "org.apache.spark.util.ClosureCleaner$.getInnerClosureClasses(ClosureCleaner.scala:81)"
>>
>> 4 = {StackTraceElement@7136}
>> "org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:187)"
>>
>> 5 = {StackTraceElement@7137}
>> "org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)"
>>
>> 6 = {StackTraceElement@7138}
>> "org.apache.spark.SparkContext.clean(SparkContext.scala:2030)"
>>
>> 7 = {StackTraceElement@7139}
>> "org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:331)"
>>
>> 8 = {StackTraceElement@7140}
>> "org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:330)"
>>
>> 9 = {StackTraceElement@7141}
>> "org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)"
>>
>> 10 = {StackTraceElement@7142}
>> "org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)"
>>
>> 11 = {StackTraceElement@7143}
>> "org.apache.spark.rdd.RDD.withScope(RDD.scala:306)"
>>
>> 12 = {StackTraceElement@7144}
>> "org.apache.spark.rdd.RDD.filter(RDD.scala:330)"
>>
>> 13 = {StackTraceElement@7145}
>> "SimpleApp$GeneratedEvaluatorClass$44$1.invoke(FileToCompile0.scala:30)"
>>
>> 14 = {StackTraceElement@7146} "SimpleApp$.main(test1.scala:26)"
>>
>> 15 = {StackTraceElement@7147} "SimpleApp.main(test1.scala)"
>>
>>
>>
>> *From:* Ndjido Ardo Bar [mailto:ndj...@gmail.com]
>> *Sent:* Thursday, December 10, 2015 2:20 PM
>> *To:* Бобров Виктор 
>> *Cc:* user@spark.apache.org
>> *Subject:* Re: Can't filter
>>
>>
>>
>> Please send your call stack with the full description of the exception .
>>
>>
>> On 10 Dec 2015, at 12:10, Бобров Виктор  wrote:
>>
>> Hi, I can’t filter my rdd.
>>
>>
>>
>> *def *filter1(tp: ((Array[String], Int), (Array[String], Int))):
>> Boolean= {
>>   tp._1._2 > tp._2._2
>> }
>> *val *mail_rdd = sc.parallelize(A.toSeq).cache()
>> *val *step1 = mail_rdd.cartesian(mail_rdd)
>> *val *step2 = step1.filter(filter1)
>>
>>
>>
>> Get error “Class not found”. What I’m doing wrong ? Thanks for help.
>>
>>
>>
>>
>>
>>
>>
>>

Re: Debug Spark

[Sudhanshu Janghel]

Hi,

Here is a doc that I had created for my team. This has steps along with
snapshots of how to setup debugging in spark using IntelliJ locally.

https://docs.google.com/a/cloudwick.com/document/d/13kYPbmK61di0f_XxxJ-wLP5TSZRGMHE6bcTBjzXD0nA/edit?usp=sharing

Kind Regards,
Sudhanshu

On Thu, Dec 3, 2015 at 6:46 AM, Akhil Das 
wrote:

> This doc will get you started
> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
>
> Thanks
> Best Regards
>
> On Sun, Nov 29, 2015 at 9:48 PM, Masf  wrote:
>
>> Hi
>>
>> Is it possible to debug spark locally with IntelliJ or another IDE?
>>
>> Thanks
>>
>> --
>> Regards.
>> Miguel Ángel
>>
>
>

Re: How to kill spark applications submitted using spark-submit reliably?

[Sudhanshu Janghel]

I have noticed that the UI takes some time to reflect the requested changes. Is 
that the issue ? Have you tried waiting for a few minutes after killing the 
spark job from terminal ?

Regards,
Sudhanshu

Kind Regards,
Sudhanshu

On 23 Nov 2015, at 1:43 a.m., Ted Yu  wrote:

>> If you ask about trapping the SIGKILL signal in your script, see the 
>> following:
>> 
>> http://linuxcommand.org/wss0160.php
>> 
>> Cheers
>  
>>> On Fri, Nov 20, 2015 at 10:02 PM, Vikram Kone  wrote:
>>> I tried adding shutdown hook to my code but it didn't help. Still same issue
>>> 
>>> 
 On Fri, Nov 20, 2015 at 7:08 PM, Ted Yu  wrote:
 Which Spark release are you using ?
 
 Can you pastebin the stack trace of the process running on your machine ?
 
 Thanks
 
> On Nov 20, 2015, at 6:46 PM, Vikram Kone  wrote:
> 
> Hi,
> I'm seeing a strange problem. I have a spark cluster in standalone mode. 
> I submit spark jobs from a remote node as follows from the terminal
> 
> spark-submit --master spark://10.1.40.18:7077  --class com.test.Ping 
> spark-jobs.jar
> 
> when the app is running , when I press ctrl-C on the console terminal, 
> then the process is killed and so is the app in the spark master UI. When 
> I go to spark master ui, i see that this app is in state Killed under 
> Completed applications, which is what I expected to see.
> 
> Now, I created a shell script as follows to do the same
> 
> #!/bin/bash
> spark-submit --master spark://10.1.40.18:7077  --class com.test.Ping 
> spark-jobs.jar
> echo $! > my.pid
> 
> When I execute the shell script from terminal, as follows
> 
> $> bash myscript.sh
> 
> The application is submitted correctly to spark master and I can see it 
> as one of the running apps in teh spark master ui. But when I kill the 
> process in my terminal as follows
> 
> $> ps kill $(cat my.pid)
> 
> I see that the process is killed on my machine but the spark appliation 
> is still running in spark master! It doesn't get killed.
> 
> I noticed one more thing that, when I launch the spark job via shell 
> script and kill the application from spark master UI by clicking on 
> "kill" next to the running application, it gets killed in spark ui but I 
> still see the process running in my machine. 
> 
> In both cases, I would expect the remote spark app to be killed and my 
> local process to be killed.
> 
> Why is this happening? and how can I kill a spark app from the terminal 
> launced via shell script w.o going to the spark master UI?
> 
> I want to launch the spark app via script and log the pid so i can 
> monitor it remotely
> 
> thanks for the help
>