[GitHub] spark pull request #20392: Update ApplicationMaster.scala
Github user Sangrho closed the pull request at: https://github.com/apache/spark/pull/20392 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20392: Update ApplicationMaster.scala
GitHub user Sangrho opened a pull request: https://github.com/apache/spark/pull/20392 Update ApplicationMaster.scala I have one question. I think when maxNumExecutorFailures is calculated, MAX_EXECUTOR_FAILURES is already defined by specific by spark document (as numExecutors * 2, with minimum of 3) So the annotation added by me in the code is not valid. Give me the answer please. Thank you ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Sangrho/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20392.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20392 commit 2eb87e032582f3b398997f3877d6f27ec2b1653e Author: Josh LEE <leoniz127@...> Date: 2018-01-25T04:53:41Z Update ApplicationMaster.scala I have one question. I think when maxNumExecutorFailures is calculated, MAX_EXECUTOR_FAILURES is already defined by specific by spark document (as numExecutors * 2, with minimum of 3) So the annotation added by me in the code is not valid. Give me the answer please. Thank you --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[jira] [Commented] (HADOOP-8545) Filesystem Implementation for OpenStack Swift
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13650485#comment-13650485 ] SangRho Lee commented on HADOOP-8545: - hello.im korean openstack user. nice to meet you. can i ask something about swift? In swift, streaming service works? for example, On downloading, gzip file is unzipping. Is it work?? Sent from my iPhone Filesystem Implementation for OpenStack Swift - Key: HADOOP-8545 URL: https://issues.apache.org/jira/browse/HADOOP-8545 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.3-alpha, 1.1.2 Reporter: Tim Miller Assignee: Dmitry Mezhensky Labels: hadoop, patch Attachments: HADOOP-8545-10.patch, HADOOP-8545-11.patch, HADOOP-8545-12.patch, HADOOP-8545-13.patch, HADOOP-8545-14.patch, HADOOP-8545-15.patch, HADOOP-8545-16.patch, HADOOP-8545-17.patch, HADOOP-8545-18.patch, HADOOP-8545-19.patch, HADOOP-8545-1.patch, HADOOP-8545-20.patch, HADOOP-8545-21.patch, HADOOP-8545-22.patch, HADOOP-8545-23.patch, HADOOP-8545-24.patch, HADOOP-8545-2.patch, HADOOP-8545-3.patch, HADOOP-8545-4.patch, HADOOP-8545-5.patch, HADOOP-8545-6.patch, HADOOP-8545-7.patch, HADOOP-8545-8.patch, HADOOP-8545-9.patch, HADOOP-8545-javaclouds-2.patch, HADOOP-8545.patch, HADOOP-8545.patch ,Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8545) Filesystem Implementation for OpenStack Swift
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SangRho Lee updated HADOOP-8545: Release Note: Added file system implementation for OpenStack Swift. There are two implementation: block and native (similar to Amazon S3 integration). Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress. To use implementation add to core-site.xml following: ... property namefs.swift.impl/name valuecom.mirantis.fs.SwiftFileSystem/value /property property namefs.swift.block.impl/name valuecom.mirantis.fs.block.SwiftBlockFileSystem/value /property ... In MapReduce job specify following configs for OpenStack Keystone authentication: conf.set(swift.auth.url, http://172.18.66.117:5000/v2.0/tokens;); conf.set(swift.tenant, superuser); conf.set(swift.username, admin1); conf.set(swift.password, password); conf.setInt(swift.http.port, 8080); conf.setInt(swift.https.port, 443); Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integrationㅏㅏ was: Added file system implementation for OpenStack Swift. There are two implementation: block and native (similar to Amazon S3 integration). Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress. To use implementation add to core-site.xml following: ... property namefs.swift.impl/name valuecom.mirantis.fs.SwiftFileSystem/value /property property namefs.swift.block.impl/name valuecom.mirantis.fs.block.SwiftBlockFileSystem/value /property ... In MapReduce job specify following configs for OpenStack Keystone authentication: conf.set(swift.auth.url, http://172.18.66.117:5000/v2.0/tokens;); conf.set(swift.tenant, superuser); conf.set(swift.username, admin1); conf.set(swift.password, password); conf.setInt(swift.http.port, 8080); conf.setInt(swift.https.port, 443); Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration Filesystem Implementation for OpenStack Swift - Key: HADOOP-8545 URL: https://issues.apache.org/jira/browse/HADOOP-8545 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.3-alpha, 1.1.2 Reporter: Tim Miller Assignee: Dmitry Mezhensky Labels: hadoop, patch Attachments: HADOOP-8545-10.patch, HADOOP-8545-11.patch, HADOOP-8545-12.patch, HADOOP-8545-13.patch, HADOOP-8545-14.patch, HADOOP-8545-15.patch, HADOOP-8545-16.patch, HADOOP-8545-17.patch, HADOOP-8545-18.patch, HADOOP-8545-19.patch, HADOOP-8545-1.patch, HADOOP-8545-20.patch, HADOOP-8545-21.patch, HADOOP-8545-22.patch, HADOOP-8545-2.patch, HADOOP-8545-3.patch, HADOOP-8545-4.patch, HADOOP-8545-5.patch, HADOOP-8545-6.patch, HADOOP-8545-7.patch, HADOOP-8545-8.patch, HADOOP-8545-9.patch, HADOOP-8545-javaclouds-2.patch, HADOOP-8545.patch, HADOOP-8545.patch ,Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8545) Filesystem Implementation for OpenStack Swift
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SangRho Lee updated HADOOP-8545: Release Note: Added file system implementation for OpenStack Swift. There are two implementation: block and native (similar to Amazon S3 integration). Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress. To use implementation add to core-site.xml following: ... property namefs.swift.impl/name valuecom.mirantis.fs.SwiftFileSystem/value /property property namefs.swift.block.impl/name valuecom.mirantis.fs.block.SwiftBlockFileSystem/value /property ... In MapReduce job specify following configs for OpenStack Keystone authentication: conf.set(swift.auth.url, http://172.18.66.117:5000/v2.0/tokens;); conf.set(swift.tenant, superuser); conf.set(swift.username, admin1); conf.set(swift.password, password); conf.setInt(swift.http.port, 8080); conf.setInt(swift.https.port, 443); Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration was: Added file system implementation for OpenStack Swift. There are two implementation: block and native (similar to Amazon S3 integration). Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress. To use implementation add to core-site.xml following: ... property namefs.swift.impl/name valuecom.mirantis.fs.SwiftFileSystem/value /property property namefs.swift.block.impl/name valuecom.mirantis.fs.block.SwiftBlockFileSystem/value /property ... In MapReduce job specify following configs for OpenStack Keystone authentication: conf.set(swift.auth.url, http://172.18.66.117:5000/v2.0/tokens;); conf.set(swift.tenant, superuser); conf.set(swift.username, admin1); conf.set(swift.password, password); conf.setInt(swift.http.port, 8080); conf.setInt(swift.https.port, 443); Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integrationㅏㅏ Filesystem Implementation for OpenStack Swift - Key: HADOOP-8545 URL: https://issues.apache.org/jira/browse/HADOOP-8545 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.3-alpha, 1.1.2 Reporter: Tim Miller Assignee: Dmitry Mezhensky Labels: hadoop, patch Attachments: HADOOP-8545-10.patch, HADOOP-8545-11.patch, HADOOP-8545-12.patch, HADOOP-8545-13.patch, HADOOP-8545-14.patch, HADOOP-8545-15.patch, HADOOP-8545-16.patch, HADOOP-8545-17.patch, HADOOP-8545-18.patch, HADOOP-8545-19.patch, HADOOP-8545-1.patch, HADOOP-8545-20.patch, HADOOP-8545-21.patch, HADOOP-8545-22.patch, HADOOP-8545-2.patch, HADOOP-8545-3.patch, HADOOP-8545-4.patch, HADOOP-8545-5.patch, HADOOP-8545-6.patch, HADOOP-8545-7.patch, HADOOP-8545-8.patch, HADOOP-8545-9.patch, HADOOP-8545-javaclouds-2.patch, HADOOP-8545.patch, HADOOP-8545.patch ,Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
