[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-15 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172571#comment-14172571
 ] 

Sean Owen commented on SPARK-1209:
--

... and why wouldn't you, that's the title of the JIRA, oops. It's not that 
class that moves or even changes actually, and yes it should not move. Let me 
fix the title and fix my PR too. Maybe that's a more palatable change.

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Ryza
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-15 Thread Patrick Wendell (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172496#comment-14172496
 ] 

Patrick Wendell commented on SPARK-1209:


Hey Sean the class I thought this pertained to was SparkHadoopUtil:

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Ryza
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-15 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172479#comment-14172479
 ] 

Apache Spark commented on SPARK-1209:
-

User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/2814

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Ryza
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-14 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170617#comment-14170617
 ] 

Sean Owen commented on SPARK-1209:
--

Hm, it's {{private[apache]}} though. Couldn't this only be used by people 
writing in the {{org.apache}} namespace? naturally a project might do just this 
to access this code, but I hadn't though this was promised as an stable API. 
People that pull this trick can I suppose declare their hack in 
{{org.apache.spark}}, although that's a source change.

I can set up a forwarder and deprecate to see how that looks but wanted to 
check if it's really these classes in question that are being used outside 
Spark.

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Ryza
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-13 Thread Patrick Wendell (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170302#comment-14170302
 ] 

Patrick Wendell commented on SPARK-1209:


Actually a full set of forwarders might be overkill anyways since there's not 
much code.

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Pérez González
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-13 Thread Patrick Wendell (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170298#comment-14170298
 ] 

Patrick Wendell commented on SPARK-1209:


It's possible this previously used package-private code that it doesn't need 
any more. I agree it's better not to pollute the Hadoop namespace, but I also 
think many people use this and I don't think it would be justified to break 
this API for the purpose of cleanliness of our own code. IIRC we tried to make 
this private[spark] earlier but people asked to open it up.

I think it would be okay to deprecate the old one and add forwarder methods to 
a new thing inside of the spark package. But wholesale moving it IMO isn't 
justified for the purpose of cleanliness alone.

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Pérez González
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-13 Thread Sandy Ryza (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169846#comment-14169846
 ] 

Sandy Ryza commented on SPARK-1209:
---

Definitely worth changing, in my opinion.  This has been bothering me for a 
while

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Pérez González
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-10-13 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169835#comment-14169835
 ] 

Sean Owen commented on SPARK-1209:
--

Yes, I wonder too, does SparkHadoopMapRedUtil and SparkHadoopMapReduceUtil need 
to live in {{org.apache.hadoop}} anymore? I assume they may have in the past to 
access some package-private Hadoop code. But I've tried moving them under 
{{org.apache.spark}} and compiling versus a few Hadoop versions and it all 
seems fine.

Am I missing something or is this worth changing? it's private to Spark (well, 
org.apache right now by necessity) so think it's fair game to move.

See https://github.com/srowen/spark/tree/SPARK-1209

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Pérez González
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-06-19 Thread Mark Grover (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038328#comment-14038328
 ] 

Mark Grover commented on SPARK-1209:


ok, I will take over. Thanks Sandy.

> SparkHadoopUtil should not use package org.apache.hadoop
> 
>
> Key: SPARK-1209
> URL: https://issues.apache.org/jira/browse/SPARK-1209
> Project: Spark
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Sandy Pérez González
>Assignee: Mark Grover
>
> It's private, so the change won't break compatibility



--
This message was sent by Atlassian JIRA
(v6.2#6252)