[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user vitillo commented on the issue: https://github.com/apache/spark/pull/17012 @zsxwing I have rewritten the test code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user hejix commented on the issue: https://github.com/apache/spark/pull/17012 @zsxwing Is it possible for you to give some approximate timeframe for when this might be accepted? A week or month? Since we are blocked in our spark streaming adoption (this HDFS error happens not only in restart but also in first run of our more complex aggregating drivers), I don't know if I should wait for this fix to make it into the nightly build soon or build a custom dist ourselves and update our automation. Any input as to the fix timeline would be greatly appreciated! Also this fix handles the same issue as SPARK-19645 which has a different solution. Is this simple solution sufficient or the more complex fix in https://github.com/apache/spark/pull/16980 required? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user hejix commented on the issue: https://github.com/apache/spark/pull/17012 Just some feedback that I did some initial regression testing with this pull request on a full YARN (v2.7.3) 4 node cluster on GCP and it appears to have fixed the two issues we had- our structured streaming drivers now restart normally and our complex aggregation driver runs for the first time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17012 What is the proposed semantics from this PR now ? - If file exists, ignore. - If file does not exist, try to rename - if fails, throw exception. Is this right ? If yes, the PR title/description should be changed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17012 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17012 > Is this right ? If yes, the PR title/description should be changed. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17012 **[Test build #73548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73548/testReport)** for PR 17012 at commit [`530c027`](https://github.com/apache/spark/commit/530c027e8ac22caa6fac3770ae24c6727ab7c018). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17012 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17012 **[Test build #73548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73548/testReport)** for PR 17012 at commit [`530c027`](https://github.com/apache/spark/commit/530c027e8ac22caa6fac3770ae24c6727ab7c018). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17012 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73548/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17012 Just two minor comments. Otherwise, LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17012 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user vitillo commented on the issue: https://github.com/apache/spark/pull/17012 @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17012 The fix looks good to me. But the test doesn't test anything. Right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17012: [SPARK-19677][SS] Renaming a file atop an existing one s...
Github user vitillo commented on the issue: https://github.com/apache/spark/pull/17012 @zsxwing The test is there just to make sure the new code path is executed during tests; should I remove it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
