[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231349#comment-14231349 ] Hudson commented on YARN-2765: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #23 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/23/]) YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe (jianhe: rev a7fba0bc28764e0fb36c335ea60cc58079fe007f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231372#comment-14231372 ] Hudson commented on YARN-2765: -- ABORTED: Integrated in Hadoop-Hdfs-trunk #1951 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1951/]) YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe (jianhe: rev a7fba0bc28764e0fb36c335ea60cc58079fe007f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231389#comment-14231389 ] Hudson commented on YARN-2765: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #761 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/761/]) YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe (jianhe: rev a7fba0bc28764e0fb36c335ea60cc58079fe007f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231518#comment-14231518 ] Hudson commented on YARN-2765: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #23 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/23/]) YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe (jianhe: rev a7fba0bc28764e0fb36c335ea60cc58079fe007f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231613#comment-14231613 ] Hudson commented on YARN-2765: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1975 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1975/]) YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe (jianhe: rev a7fba0bc28764e0fb36c335ea60cc58079fe007f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231632#comment-14231632 ] Hudson commented on YARN-2765: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #23 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/23/]) YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe (jianhe: rev a7fba0bc28764e0fb36c335ea60cc58079fe007f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230598#comment-14230598
]
Hadoop QA commented on YARN-2765:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12684476/YARN-2765v3.patch
against trunk revision 86e3993.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5963//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5963//console
This message is automatically generated.
Add leveldb-based implementation for RMStateStore
-
Key: YARN-2765
URL: https://issues.apache.org/jira/browse/YARN-2765
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch
It would be nice to have a leveldb option to the resourcemanager recovery
store. Leveldb would provide some benefits over the existing filesystem store
such as better support for atomic operations, fewer I/O ops per state update,
and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230754#comment-14230754 ] Jian He commented on YARN-2765: --- looks good, +1 Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230782#comment-14230782 ] Hudson commented on YARN-2765: -- FAILURE: Integrated in Hadoop-trunk-Commit #6622 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6622/]) YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe (jianhe: rev a7fba0bc28764e0fb36c335ea60cc58079fe007f) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.7.0 Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225262#comment-14225262
]
Jian He commented on YARN-2765:
---
Thanks, Jason! Patch looks good. just a few minor comments.
- Patch needs updated on top of YARN-2404.
- we may use the following constants from the RMStateStore base class
— DB_SCHEMA_VERSION_KEY - VERSION_NODE
— EPOCH_KEY - EPOCH_NODE
— RM_DT_SEQUENCE_NUMBER_KEY: use DELEGATION_TOKEN_SEQUENCE_NUMBER_PREFIX for
“RMDTSequentialNumber”
- this check is always true and so unnecessary, as attemptState is earlier
initialized with attemptId
{code}
if (!attemptId.equals(attemptState.getAttemptId())) {
throw new YarnRuntimeException(The database entry for + attemptId
+ contains data for + attemptState.getAttemptId());
}
{code}
Some naming suggestions:
- loadRMAppAttemptState- createAttemptState
- loadRMAppState(String appIdStr, byte[] data) - createApplicationState
- {{String itemName = key.substring(keyPrefix.length());}} rename itemName to
attemptId, and keyPrefix to attemptNodePrefix
- Question: does the levelDB client API handle retry ?
Add leveldb-based implementation for RMStateStore
-
Key: YARN-2765
URL: https://issues.apache.org/jira/browse/YARN-2765
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-2765.patch, YARN-2765v2.patch
It would be nice to have a leveldb option to the resourcemanager recovery
store. Leveldb would provide some benefits over the existing filesystem store
such as better support for atomic operations, fewer I/O ops per state update,
and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221326#comment-14221326
]
Zhijie Shen commented on YARN-2765:
---
bq. I knew rocksdb could be used as a cache of data that came from HDFS or
could be backed-up to HDFS, but I didn't think it could read/write directly to
it as part of normal operations.
Hm... I should have wrongly understand the feature. Thanks for correction.
One question about the patch: Why is it necessary to try create the DB with
{{options.createIfMissing(false);}} and then {{options.createIfMissing(true);}}
if it fails at first attempt? Can't we do one create if missing?
Add leveldb-based implementation for RMStateStore
-
Key: YARN-2765
URL: https://issues.apache.org/jira/browse/YARN-2765
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-2765.patch, YARN-2765v2.patch
It would be nice to have a leveldb option to the resourcemanager recovery
store. Leveldb would provide some benefits over the existing filesystem store
such as better support for atomic operations, fewer I/O ops per state update,
and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221448#comment-14221448 ] Jason Lowe commented on YARN-2765: -- bq. Can't we do one create if missing? This is to distinguish between a state store that wasn't there (and thus needs to be created) vs. opening an empty, existing state store. We log different messages during startup so it's easy to distinguish between these cases. IMHO it's important to know when the state store wasn't there on startup and needed to be created. Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-2765.patch, YARN-2765v2.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218766#comment-14218766 ] Jian He commented on YARN-2765: --- thanks Jason for working on this. It's useful to have leveldb as an option for RMStateStore, as its more lightweight compared to others. reviewing the patch Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-2765.patch, YARN-2765v2.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14189934#comment-14189934 ] Tsuyoshi OZAWA commented on YARN-2765: -- Currently we can assume that LeveldbRMStateStore is access from single process, thus failure detection itself is done by EmbeddedElector which depends on ZooKeeper in addition to no support of fencing on LevelDBRMStateStore. It means we need to launch ZooKeeper and it's normal decision to use ZKRMStateStore in this case. Please correct me if I'm wrong. On another front, if we use RockDB as a backend db of timeline server, we don't need to use leveldb and it's good decision to switch the dependency. Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-2765.patch, YARN-2765v2.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188589#comment-14188589
]
Hadoop QA commented on YARN-2765:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677911/YARN-2765v2.patch
against trunk revision ec63a3f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5626//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5626//console
This message is automatically generated.
Add leveldb-based implementation for RMStateStore
-
Key: YARN-2765
URL: https://issues.apache.org/jira/browse/YARN-2765
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-2765.patch, YARN-2765v2.patch
It would be nice to have a leveldb option to the resourcemanager recovery
store. Leveldb would provide some benefits over the existing filesystem store
such as better support for atomic operations, fewer I/O ops per state update,
and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188595#comment-14188595 ] Zhijie Shen commented on YARN-2765: --- bq. This should work if the leveldb database is on a network store like a filer. Thanks for sharing. This is an interesting use case that I'm not aware of before. bq. I briefly considered using rocksdb for this but decided against it for a couple of reasons: It's not particularly related to this Jira, but I just want to think it out loudly. It seems that rocksdb claims to have better performance in terms of I/O than leveldb, while their APIs are very similar to each other. After we have the leveldb impl, it shouldn't be that difficult to make a rocksdb impl. Probably leveldb is enough to serve as the state store for RM/NM/JHS, but the timeline server may want a stronger one. Rocksdb may be a compromise before migrating to fully distributed storage solution based on HBase. And one other merit I've heard about rocksdb is that it can ride on HDFS. Correct me if I'm wrong, but it seems that rocksdb can also help to scale out the storage problem as well as support RM HA deployment in a shared nothing environment (e.g. without a network storage). I'm not saying we should go with rocksdb now instead of leveldb, as we know it has been used for other components already. I'm trying to propose if we can think of rocksdb, which looks stronger but still reasonably simple alternate. Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-2765.patch, YARN-2765v2.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188755#comment-14188755 ] Jason Lowe commented on YARN-2765: -- I agree that the timeline server seems like a worthy candidate for rocksdb. IIUC rocksdb's main use-case over leveldb is better performance when the database is larger than the node's RAM, which is likely in the case of the timeline server. bq. And one other merit I've heard about rocksdb is that it can ride on HDFS. This is news to me. I knew rocksdb could be used as a cache of data that came from HDFS or could be backed-up to HDFS, but I didn't think it could read/write directly to it as part of normal operations. bq. There's a rocksdb jni which seems to have windows support: https://github.com/fusesource/rocksdbjni Awesome, thanks for finding that. I was looking at the standard org.rocksdb package. Only concern with the fusesource option would be if it starts to diverge significantly from the standard one. The API is already slightly different between the two, and the fusesource one hasn't been touched in a year while the org.rocksdb package was updated just last week. Probably best to continue this conversation in a separate JIRA proposing we consider rocksdb for the timeline server. If it works well there it should be very straightforward to provide store backends for the RM, NM, and JHS if it makes sense for them as well. Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-2765.patch, YARN-2765v2.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187931#comment-14187931
]
Hadoop QA commented on YARN-2765:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677800/YARN-2765.patch
against trunk revision 3c5f5af.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5618//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5618//console
This message is automatically generated.
Add leveldb-based implementation for RMStateStore
-
Key: YARN-2765
URL: https://issues.apache.org/jira/browse/YARN-2765
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-2765.patch
It would be nice to have a leveldb option to the resourcemanager recovery
store. Leveldb would provide some benefits over the existing filesystem store
such as better support for atomic operations, fewer I/O ops per state update,
and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187955#comment-14187955
]
Tsuyoshi OZAWA commented on YARN-2765:
--
[~jlowe], great work! It looks good to me overall including error handling and
resource management.
Minor nits:
How about adding helper methods like getKeyPrefix/getNodePath for getting key
prefix and node path? ZKRMStateStore also does so.
{code}
String keyPrefix = RM_APP_ROOT + / + appId + /;
...
String appKey = RM_APP_ROOT + / + appId
{code}
I found that the patch includes lots hard-coded /. I think it's better to
have private field SEPARATOR = /.
Add leveldb-based implementation for RMStateStore
-
Key: YARN-2765
URL: https://issues.apache.org/jira/browse/YARN-2765
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-2765.patch
It would be nice to have a leveldb option to the resourcemanager recovery
store. Leveldb would provide some benefits over the existing filesystem store
such as better support for atomic operations, fewer I/O ops per state update,
and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187987#comment-14187987 ] Zhijie Shen commented on YARN-2765: --- [~jlowe], the leveldb should be a good choice over local file system in terms of the merits mentioned in the description. One drawback I can think of is that while LeveldbRMStateStore is lightweight for single RM restarting, multiple RMs of HA are not able to share this single-host database. Correct me if I'm wrong. BTW, I saw you also have other Jira for JHS using leveldb. Did you have a chance to think of an enhanced k/v db: rocksdb? I guess it will benefit the RM case as well as the timeline case. Add leveldb-based implementation for RMStateStore - Key: YARN-2765 URL: https://issues.apache.org/jira/browse/YARN-2765 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-2765.patch It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
