Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8976
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-149151638
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-149024871
I think this much is OK -- removing the unused var / args in internal code
and uniformly deprecating its use in the public API. I'm still eyeing
SPARK-4352 but figure
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-148561702
[Test build #1907 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1907/console)
for PR 8976 at commit
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-148512484
@srowen Mind looking at the PR once again? I'd appreciate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-148538673
[Test build #1907 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1907/consoleFull)
for PR 8976 at commit
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-147645261
Yeah you'll still have to filter this spurious binary compatibility check
warning:
```
[error] * method
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-147602196
Hey @srowen @jerryshao How does the change look like now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8976#discussion_r41827562
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -147,10 +139,9 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-147606659
[Test build #1886 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1886/console)
for PR 8976 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-147602759
[Test build #1886 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1886/consoleFull)
for PR 8976 at commit
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146465163
From my understanding for some batch workloads like ETL, traditional
process pattern is to read data from HDFS, process and output to HDFS, in such
process pattern
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146474747
Fully agree. I'm however not skilled to work on the feature to enable
locality preference and haven't heard about anyone working on it, and the code
as it is now
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146462053
At some point in the future, very much likely, but it's not happening now
and I'd remove it now (to make the code clearer) and once it's needed it gets
added in
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146459679
A-ha so you mean there is a pretty real intent to begin to use this data
structure? enough that we shouldn't remove it?
---
If your project is set up for it, you can
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146510001
I don't think it causes trouble. Most people don't ever use this
constructor, and using it doesn't cause any problems (well, other than doing
nothing as advertised). If
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146477140
Yeah, but this is also changing the public-facing API. We'd be deprecating
and then undeprecating interfaces, and then potentially deprecating other
interfaces. If this
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146506978
Sure, but how do you know **now** how the public API is going to look like?
It is **currently** causing troubles in understanding the code and we only
**wish** /
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146545328
Yes, as @srowen said, this API is seldom used, and currently we already
marked it as deprecated, also even calling this API will not introduce any
problem, so keeping
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146574566
SPARK-2089 is marked as resolved so any discussion there is done (or am I
mistaken?)
Also, I'm saying that the comments say it's used in YARN while it's
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146595930
@jaceklaskowski I think you're going to have to back out your API changes
anyway since they aren't compatible. The rest, yeah, no real problem with
updating docs, or
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-146384050
With [SPARK-4352](https://issues.apache.org/jira/browse/SPARK-4352) getting
merged, this `SparkContext.preferredNodeLocationData` can easily be achieved by
computing
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/8976#discussion_r41303229
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -147,23 +139,41 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145977387
`./dev/mima` will run MiMa checks for you. Jenkins will automatically
retest if you push anything else to the branch.
---
If your project is set up for it, you can
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145976915
Taking these out seems like the right thing to do to me
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145958130
Thanks @vanzin @srowen! That helped a lot. I'm going to fix it. How can I
run the compatibility check locally? Is it `./build/sbt
core/mima-report-binary-issues`
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145931588
The second one is because the default value for the
`preferredNodeLocationData` parameter is being removed. You can't do that, it
breaks the ABI.
---
If your project
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8976#discussion_r41289941
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -147,23 +139,41 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145910412
@srowen @SparkQA says "This patch fails MiMa tests.", but I can't seem to
decipher what exactly the pull request has broken. I remember you mentioned the
change
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145925233
```
[error] * method preferredNodeLocationData_=(scala.collection.Map)Unit in
class org.apache.spark.SparkContext does not have a correspondent in new version
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145492004
[Test build #1841 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1841/consoleFull)
for PR 8976 at commit
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8976#discussion_r41130010
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala ---
@@ -49,10 +49,11 @@ private[spark] class YarnRMClient(args:
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145491891
@jerryshao @sryza do you have a thought on this one? looks OK to remove as
it's unused, but was there any hint of a plan to use it again at some point?
---
If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145494283
[Test build #1841 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1841/console)
for PR 8976 at commit
GitHub user jaceklaskowski opened a pull request:
https://github.com/apache/spark/pull/8976
[SPARK-10921] [YARN] Completely remove the use of SparkContext.preferâ¦
â¦redNodeLocationData
You can merge this pull request into a Git repository by running:
$ git pull
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8976#issuecomment-145432564
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
36 matches
Mail list logo
