Re: Spark Debugging

[Akhil Das]

Thanks
Best Regards

On Thu, Oct 30, 2014 at 1:43 PM, Akhil Das 
wrote:

> Awesome.
>
> Thanks
> Best Regards
>
> On Thu, Oct 30, 2014 at 1:30 PM, Naveen Kumar Pokala <
> npok...@spcapitaliq.com> wrote:
>
>> Thanks Akhil, It is working.
>>
>>
>>
>> Regards,
>>
>> Naveen
>>
>>
>>
>> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
>> *Sent:* Thursday, October 30, 2014 1:24 PM
>>
>> *To:* Naveen Kumar Pokala
>> *Subject:* Re: Spark Debugging
>>
>>
>>
>> Yes you can, but the values might not give you a complete clue. Give it a
>> try
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Thu, Oct 30, 2014 at 1:20 PM, Naveen Kumar Pokala <
>> npok...@spcapitaliq.com> wrote:
>>
>> Do you mean can’t I debug line by line?
>>
>>
>>
>> Thanks and Regards
>>
>> Naveen
>>
>>
>>
>> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
>> *Sent:* Thursday, October 30, 2014 1:19 PM
>>
>>
>> *To:* Naveen Kumar Pokala
>> *Subject:* Re: Spark Debugging
>>
>>
>>
>> Master machine. The execution will get pause in eclipse regardless of
>> where its executing the program, you might not get a complete flow, but you
>> can see whats happening.
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Thu, Oct 30, 2014 at 1:10 PM, Naveen Kumar Pokala <
>> npok...@spcapitaliq.com> wrote:
>>
>> Hi Akhil,
>>
>>
>>
>> With below configuration I am successfully able to connect to  Node
>> A(Master node) where I am running the spark submit program, Spark may give
>> driver program to another Node B also.
>>
>>
>>
>> Which machine IP should I use to debug the programs from eclipse.
>>
>>
>>
>> Thanks,
>>
>> Naveen
>>
>>
>>
>> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
>> *Sent:* Thursday, October 30, 2014 12:35 PM
>> *To:* Naveen Kumar Pokala
>> *Subject:* Re: Spark Debugging
>>
>>
>>
>> ​Hi Naveen,
>>
>>
>>
>> You should be able to connect to  port also from your windows machine.
>> ​
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Thu, Oct 30, 2014 at 12:32 PM, Naveen Kumar Pokala <
>> npok...@spcapitaliq.com> wrote:
>>
>> Hi Akhil,
>>
>>
>>
>> I have gone through the article. But I am  not starting my java program
>> with Java command, I am submitting through spark-submit and I have added
>> like below is that ok?
>>
>>
>>
>> bin/spark-submit --class sample.spark.test.SparkJob --master yarn-cluster
>> --driver-memory 4g --executor-memory  --conf
>> "spark.executor.extraJavaOptions=-Xdebug
>> -agentlib:jdwp=transport=dt_socket,address=,server=y,suspend=n"
>> /home/npokala/data/test.jar
>>
>>
>>
>> I am able to connect to cluster from my windows machine only through
>> putty. I am able to access the cluster information and spark information
>> from my browser. Is that sufficient.?
>>
>>
>>
>>
>>
>> Regards,
>>
>> Naveen.
>>
>>
>>
>> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
>> *Sent:* Thursday, October 30, 2014 12:10 PM
>> *To:* Naveen Kumar Pokala
>> *Cc:* user@spark.apache.org
>> *Subject:* Re: Spark Debugging
>>
>>
>>
>> Its called remote debugging. You can read this article
>> <http://www.eclipse.org/jetty/documentation/current/enable-remote-debugging.html>
>> for more information. You will have to make sure that the network between
>> your cluster and windows machine can communicate with each other.
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Thu, Oct 30, 2014 at 12:06 PM, Naveen Kumar Pokala <
>> npok...@spcapitaliq.com> wrote:
>>
>> Hi,
>>
>>
>>
>>   I have installed 2 node hadoop cluster (For example, on Unix machines A
>> and B. A master node and data node, B is data node)
>>
>>
>>
>>   I am submitting my driver programs through SPARK 1.1.0 with
>> bin/spark-submit from Putty Client from my Windows machine.
>>
>>
>>
>> I want to debug my program from Eclipse on my local machine. I am not
>> able to find a way to debug.
>>
>>
>>
>> Please let me know the ways to debug my driver program as well as
>> executor programs
>>
>>
>>
>>
>>
>> Regards,
>>
>>
>>
>> Naveen.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

Spark Debugging

[Naveen Kumar Pokala]

Hi,

  I have installed 2 node hadoop cluster (For example, on Unix machines A and 
B. A master node and data node, B is data node)

  I am submitting my driver programs through SPARK 1.1.0 with bin/spark-submit 
from Putty Client from my Windows machine.

I want to debug my program from Eclipse on my local machine. I am not able to 
find a way to debug.

Please let me know the ways to debug my driver program as well as executor 
programs



[cid:image001.jpg@01CFF439.BBA1F3A0]
Naveen kumar pokala
+91 8801169530

Re: Spark Debugging

[Akhil Das]

Its called remote debugging. You can read this article

for more information. You will have to make sure that the network between
your cluster and windows machine can communicate with each other.

Thanks
Best Regards

On Thu, Oct 30, 2014 at 12:06 PM, Naveen Kumar Pokala <
npok...@spcapitaliq.com> wrote:

> Hi,
>
>
>
>   I have installed 2 node hadoop cluster (For example, on Unix machines A
> and B. A master node and data node, B is data node)
>
>
>
>   I am submitting my driver programs through SPARK 1.1.0 with
> bin/spark-submit from Putty Client from my Windows machine.
>
>
>
> I want to debug my program from Eclipse on my local machine. I am not able
> to find a way to debug.
>
>
>
> Please let me know the ways to debug my driver program as well as executor
> programs
>
>
>
>
>
> Regards,
>
>
>
> Naveen.
>

Spark Debugging

[Naveen Kumar Pokala]

Hi,

  I have installed 2 node hadoop cluster (For example, on Unix machines A and 
B. A master node and data node, B is data node)

  I am submitting my driver programs through SPARK 1.1.0 with bin/spark-submit 
from Putty Client from my Windows machine.

I want to debug my program from Eclipse on my local machine. I am not able to 
find a way to debug.

Please let me know the ways to debug my driver program as well as executor 
programs


Regards,

Naveen.