[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-04-30 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522480#comment-14522480 ] Colin Patrick McCabe commented on HDFS-7836: Hi [~xinwei], The discussion on

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-04-29 Thread Xinwei Qin (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518976#comment-14518976 ] Xinwei Qin commented on HDFS-7836: --- Hi [~cmccabe], [~clamb], This is a very meaningful

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-09 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353386#comment-14353386 ] Charles Lamb commented on HDFS-7836: JOIN WEBEX MEETING

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-09 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353455#comment-14353455 ] Charles Lamb commented on HDFS-7836: If you are planning on attending the meeting

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-06 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350878#comment-14350878 ] Colin Patrick McCabe commented on HDFS-7836: I don't think the block map size

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-05 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349438#comment-14349438 ] Charles Lamb commented on HDFS-7836: We'll hold a design review meeting and discussion

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-05 Thread Chris Nauroth (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349599#comment-14349599 ] Chris Nauroth commented on HDFS-7836: - The thing we're losing is the ability to measure

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-04 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347999#comment-14347999 ] Colin Patrick McCabe commented on HDFS-7836: I think it makes sense to have

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-04 Thread stack (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348237#comment-14348237 ] stack commented on HDFS-7836: - The JVM by default publishes via JMX offheap usage in the

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-03 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345012#comment-14345012 ] Charles Lamb commented on HDFS-7836: Yes, there would definitely be a webex available.

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-03 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344735#comment-14344735 ] Yi Liu commented on HDFS-7836: -- Good proposal, Colin and Charles. {quote} Reduce lock

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-03 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345482#comment-14345482 ] Colin Patrick McCabe commented on HDFS-7836: bq. Generally we have some ways to

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-03 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345475#comment-14345475 ] Colin Patrick McCabe commented on HDFS-7836: Thanks, Yi. We'll definitely

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-03 Thread Chris Nauroth (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345610#comment-14345610 ] Chris Nauroth commented on HDFS-7836: - I have one additional operability concern. With

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-03 Thread Yi Liu (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344742#comment-14344742 ] Yi Liu commented on HDFS-7836: -- {quote} Charles Lamb is going to be around on 3/10, 3/11, and

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-03-02 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343695#comment-14343695 ] Colin Patrick McCabe commented on HDFS-7836: bq. Arpit said: We do use one RPC

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-27 Thread Daryn Sharp (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340474#comment-14340474 ] Daryn Sharp commented on HDFS-7836: --- Colin, please take a look and provide feedback on

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-27 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341074#comment-14341074 ] Arpit Agarwal commented on HDFS-7836: - bq. 1 M blocks per disk, on 10 disks, and 24

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338394#comment-14338394 ] Charles Lamb commented on HDFS-7836: Hi [~arpit99], Thanks for reading over the design

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338395#comment-14338395 ] Charles Lamb commented on HDFS-7836: Hi [~arpit99], Thanks for reading over the design

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Chris Nauroth (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339007#comment-14339007 ] Chris Nauroth commented on HDFS-7836: - bq. With regard to compatibility... even if the

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338901#comment-14338901 ] Charles Lamb commented on HDFS-7836: bq. Are you proposing that off-heaping is an

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338926#comment-14338926 ] Colin Patrick McCabe commented on HDFS-7836: Thanks for taking a look at this,

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Chris Nauroth (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338865#comment-14338865 ] Chris Nauroth commented on HDFS-7836: - +1 for the proposal overall. Thanks for the

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339258#comment-14339258 ] Colin Patrick McCabe commented on HDFS-7836: Thanks, Chris. I agree that \-Xmx

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339353#comment-14339353 ] Arpit Agarwal commented on HDFS-7836: - Thanks for the responses Charles and Colin. bq.

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339385#comment-14339385 ] Charles Lamb commented on HDFS-7836: bq. it would be useful to see some perf comparison

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339695#comment-14339695 ] Colin Patrick McCabe commented on HDFS-7836: bq. The hashing scheme should

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-26 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339693#comment-14339693 ] Colin Patrick McCabe commented on HDFS-7836: bq. These two sound contradictory.

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-25 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337977#comment-14337977 ] Arpit Agarwal commented on HDFS-7836: - Hi Charles, thank you for posting the design

[jira] [Commented] (HDFS-7836) BlockManager Scalability Improvements

2015-02-24 Thread Charles Lamb (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335747#comment-14335747 ] Charles Lamb commented on HDFS-7836: Problem Statement The number of blocks stored by