[GitHub] spark pull request: [SPARK-6469] The YARN driver in yarn-client mo...

2015-03-24 Thread preaudc 
GitHub user preaudc opened a pull request:

https://github.com/apache/spark/pull/5165

[SPARK-6469] The YARN driver in yarn-client mode will not use the local 
directories configured for YARN

Clarify the local directories usage in YARN

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/preaudc/spark yarn-doc-local-dirs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/5165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5165


commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2014-10-20T09:58:56Z

Ensure that files are fetched atomically

tempFile is created in the same directory than targetFile, so that the
move from tempFile to targetFile is always atomic

commit 7456a33932bdef4bdfe64ba2c13ce1d91b8d96fe
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2014-10-23T08:27:49Z

Merge remote-tracking branch 'upstream/master'

commit c6a5590c9866366a70dd696ff7b2bffb185201d1
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2014-11-03T13:25:05Z

Revert commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd

commit 54419ae204c79fb65bc4d4ab00eb1f82bef8a097
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2014-11-03T13:26:05Z

Merge remote-tracking branch 'upstream/master'

commit 9ba89caafc57e8d8b94f7b98e39c4613f0f5158d
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2014-11-03T13:37:46Z

Ensure that files are fetched atomically

tempFile is created in the same directory than targetFile, so that the
move from tempFile to targetFile is always atomic

commit a49a2cefd3b2117910d8f1d1eb933d7312061c1e
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2015-03-23T10:39:50Z

Merge remote-tracking branch 'upstream/master'

Conflicts:
core/src/main/scala/org/apache/spark/util/Utils.scala

commit 608dbfa18aa9fe52ef8950219e2d01068480f323
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2015-03-24T09:22:14Z

Merge remote-tracking branch 'upstream/master'

commit 876ae5e18492d20a38c1032547c3357394c7e7f7
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2015-03-24T09:52:13Z

Clarify the local directories usage in YARN

commit 436fb7d3f154aea7a184d9a5294d1b3fc91fe057
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2015-03-24T10:01:34Z

Revert Clarify the local directories usage in YARN

This reverts commit 876ae5e18492d20a38c1032547c3357394c7e7f7.

commit eaaf5191ca9e93b8504a28c4f18ab71a529db811
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2015-03-24T10:06:40Z

Clarify the local directories usage in YARN




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6469] The YARN driver in yarn-client mo...

2015-03-24 Thread preaudc 
Github user preaudc commented on the pull request:

https://github.com/apache/spark/pull/5165#issuecomment-85435545
  
I don't know if this is normal that all commits show up in the PR, I'm not 
yet familiar with this tool... Anyway, only the last one is relevant for this 
PR, and that's what can be seen in the Files changed tab.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6469] Improving documentation on YARN l...

2015-03-24 Thread preaudc 
Github user preaudc commented on the pull request:

https://github.com/apache/spark/pull/5165#issuecomment-85545150
  
Sure, is the new title ok?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4764] Ensure that files are fetched ato...

2015-01-05 Thread preaudc 
Github user preaudc commented on the pull request:

https://github.com/apache/spark/pull/2855#issuecomment-68702554
  
Yes, my bad, {{targetDir}} is indeed already a {{File}}. @JoshRosen , how 
could I fix this, should I create a new pull request, or can this one be 
reopened?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4764] Ensure that files are fetched ato...

2014-12-08 Thread preaudc 
Github user preaudc commented on the pull request:

https://github.com/apache/spark/pull/2855#issuecomment-66131633
  
Thanks for the review, @JoshRosen, I've created a new JIRA as requested.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] Ensure that files are fetched ato...

2014-11-21 Thread preaudc 
Github user preaudc commented on the pull request:

https://github.com/apache/spark/pull/2855#issuecomment-63969515
  
I have no clue what could bring this MiMa test failure, and how to fix it. 
Can anybody give me a hand?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] Ensure that files are fetched ato...

2014-11-07 Thread preaudc 
Github user preaudc commented on the pull request:

https://github.com/apache/spark/pull/2855#issuecomment-62119248
  
As @ryan-williams pointed out, this is initially only a workaround to 
SPARK-3967.
I have still no idea why the move fails (with a {{Permission denied}}) when 
the source and target files are not on the same partition (it is no more 
atomic, but it should succeed anyway).
I have made this patch because it does not seem necessary to download the 
file into another local directory then move it (it may cause a copy instead of 
a rename, and does in fact here).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] Ensure that files are fetched ato...

2014-11-03 Thread preaudc 
Github user preaudc commented on the pull request:

https://github.com/apache/spark/pull/2855#issuecomment-61479433
  
I've just resync my local fork so that the patch merges cleanly (i.e. there 
are no other modifications compared to the initial patch)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] Ensure that files are fetched ato...

2014-10-20 Thread preaudc 
GitHub user preaudc opened a pull request:

https://github.com/apache/spark/pull/2855

[SPARK-3967] Ensure that files are fetched atomically

tempFile is created in the same directory than targetFile, so that the
move from tempFile to targetFile is always atomic

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/preaudc/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2855.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2855


commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd
Author: Christophe Préaud christophe.pre...@kelkoo.com
Date:   2014-10-20T09:58:56Z

Ensure that files are fetched atomically

tempFile is created in the same directory than targetFile, so that the
move from tempFile to targetFile is always atomic




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org