How to debug Spark source using IntelliJ/ Eclipse

[jatinganhotra]

Hi,

I am trying to understand Spark internal code and wanted to debug Spark
source, to add a new feature. I have tried the steps lined out here on the 
Spark Wiki page IDE setup

 
, but they don't work.

I also found other posts in the Dev mailing list such as - 

1.  Spark-1-5-0-setting-up-debug-env

 
, and

2.  using-IntelliJ-to-debug-SPARK-1-1-Apps-with-mvn-sbt-for-beginners

  

But, I found many issues with both the links. I have tried both these
articles many times, often re-starting the whole process from scratch after
deleting everything and re-installing again, but I always face some
dependency issues.

It would be great if someone from the Spark developers group could point me
to the steps for setting up Spark debug environment.



--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-debug-Spark-source-using-IntelliJ-Eclipse-tp15477.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [ML] Missing documentation for the IndexToString feature transformer

[Holden Karau]

I'd be more than happy to help review the docs if that would be useful :)

On Sat, Dec 5, 2015 at 2:21 PM, Joseph Bradley 
wrote:

> Thanks for reporting this!  I just added a JIRA:
> https://issues.apache.org/jira/browse/SPARK-12159
> That would be great if you could send a PR for it; thanks!
> Joseph
>
> On Sat, Dec 5, 2015 at 5:02 AM, Benjamin Fradet  > wrote:
>
>> Hi,
>>
>> I was wondering why the IndexToString
>> 
>>  label
>> transformer was not documented in ml-features.md
>> .
>>
>> If it's not intentional, having used it a few times, I'd be happy to
>> submit a jira and the pr associated.
>>
>> Best,
>> Ben.
>>
>> --
>> Ben Fradet.
>>
>
>


-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau

Re: [ML] Missing documentation for the IndexToString feature transformer

[Joseph Bradley]

Thanks for reporting this!  I just added a JIRA:
https://issues.apache.org/jira/browse/SPARK-12159
That would be great if you could send a PR for it; thanks!
Joseph

On Sat, Dec 5, 2015 at 5:02 AM, Benjamin Fradet 
wrote:

> Hi,
>
> I was wondering why the IndexToString
> 
>  label
> transformer was not documented in ml-features.md
> .
>
> If it's not intentional, having used it a few times, I'd be happy to
> submit a jira and the pr associated.
>
> Best,
> Ben.
>
> --
> Ben Fradet.
>

Re: Returning numpy types from udfs

[Justin Uang]

Filed here:

https://issues.apache.org/jira/browse/SPARK-12157

On Sat, Dec 5, 2015 at 3:08 PM Reynold Xin  wrote:

> Not aware of any jira ticket, but it does sound like a great idea.
>
>
> On Sat, Dec 5, 2015 at 11:03 PM, Justin Uang 
> wrote:
>
>> Hi,
>>
>> I have fallen into the trap of returning numpy types from udfs, such as
>> np.float64 and np.int. It's hard to find the issue because they behave
>> pretty much as regular pure Python floats and doubles, so can we make
>> PYSPARK automatically translate them?
>>
>> If so, I'll create a Jira ticket.
>>
>> Justin
>>
>
>

Re: Returning numpy types from udfs

[Reynold Xin]

Not aware of any jira ticket, but it does sound like a great idea.


On Sat, Dec 5, 2015 at 11:03 PM, Justin Uang  wrote:

> Hi,
>
> I have fallen into the trap of returning numpy types from udfs, such as
> np.float64 and np.int. It's hard to find the issue because they behave
> pretty much as regular pure Python floats and doubles, so can we make
> PYSPARK automatically translate them?
>
> If so, I'll create a Jira ticket.
>
> Justin
>

Returning numpy types from udfs

[Justin Uang]

Hi,

I have fallen into the trap of returning numpy types from udfs, such as
np.float64 and np.int. It's hard to find the issue because they behave
pretty much as regular pure Python floats and doubles, so can we make
PYSPARK automatically translate them?

If so, I'll create a Jira ticket.

Justin

[ML] Missing documentation for the IndexToString feature transformer

[Benjamin Fradet]

Hi,

I was wondering why the IndexToString

label
transformer was not documented in ml-features.md
.

If it's not intentional, having used it a few times, I'd be happy to submit
a jira and the pr associated.

Best,
Ben.

-- 
Ben Fradet.