[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983736#comment-13983736
]
stack commented on HBASE-9953:
--
This seems to work nicely for me. Funny how when the block cache is 4G, if you
set the size to get to 4G, you fall out of BC. If you set it to 3G, you still
do. 2.8G gets the BC 100% used w/ no misses. We've lots of overhead it seems
(smile). Let me remove my modulo malarky.
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: Performance
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-9953.00.patch, HBASE-9953.01.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983855#comment-13983855
]
Nick Dimiduk commented on HBASE-9953:
-
Yes, I've observed our population doesn't quite match the space available.
Something is lying, or at least off by some small amount that's repeated many
times over, but I don't yet know what.
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: Performance
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-9953.00.patch, HBASE-9953.01.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981952#comment-13981952
]
Hudson commented on HBASE-9953:
---
SUCCESS: Integrated in HBase-TRUNK #5120 (See
[https://builds.apache.org/job/HBase-TRUNK/5120/])
HBASE-9953 PerformanceEvaluation: Decouple data size from client concurrency
(ndimiduk: rev 1590216)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: Performance
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-9953.00.patch, HBASE-9953.01.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981171#comment-13981171
]
Nick Dimiduk commented on HBASE-9953:
-
[~jmspaggi]
bq. 2) If both (opts.size == DEFAULT_OPTS.size) and (opts.perClientRunRows ==
DEFAULT_OPTS.perClientRunRows) then opts.totalRows is never set?
Yes, that's fine; in this case the default totalRows value of 1024 * 1024 is
used.
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: test
Reporter: Nick Dimiduk
Priority: Minor
Attachments: HBASE-9953.00.patch, HBASE-9953.01.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981174#comment-13981174
]
Nick Dimiduk commented on HBASE-9953:
-
ping [~nkeywal], [~stack].
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: Performance
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
Attachments: HBASE-9953.00.patch, HBASE-9953.01.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981273#comment-13981273
]
stack commented on HBASE-9953:
--
+1 Commit and I'll try it. Let me explain over in the modulo issue how it does
similar but I like your take better (After trying this, may back out the modulo
arg)
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: Performance
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
Attachments: HBASE-9953.00.patch, HBASE-9953.01.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981287#comment-13981287
]
Hadoop QA commented on HBASE-9953:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641941/HBASE-9953.01.patch
against trunk revision .
ATTACHMENT ID: 12641941
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 8
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9401//console
This message is automatically generated.
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: Performance
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Priority: Minor
Attachments: HBASE-9953.00.patch, HBASE-9953.01.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913151#comment-13913151
]
Jean-Marc Spaggiari commented on HBASE-9953:
2 comments.
1) You might want to update printUsage too to provide usage.
2) If both (opts.size == DEFAULT_OPTS.size) and (opts.perClientRunRows ==
DEFAULT_OPTS.perClientRunRows) then opts.totalRows is never set?
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: test
Reporter: Nick Dimiduk
Priority: Minor
Attachments: HBASE-9953.00.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13909584#comment-13909584
]
Nick Dimiduk commented on HBASE-9953:
-
Attached patch is post-refactor.
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: test
Reporter: Nick Dimiduk
Priority: Minor
Attachments: HBASE-9953.00.patch
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900303#comment-13900303
]
Jean-Marc Spaggiari commented on HBASE-9953:
Hi [~ndimiduk] any update on this one?
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: test
Reporter: Nick Dimiduk
Priority: Minor
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900549#comment-13900549
]
Nick Dimiduk commented on HBASE-9953:
-
Yes, but not on the tool itself. I ended up writing a driver script that does
the calculation for me. Nothing complicated. Let me throw together a patch.
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: test
Reporter: Nick Dimiduk
Priority: Minor
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819772#comment-13819772
]
Jean-Marc Spaggiari commented on HBASE-9953:
-1 for --size instead of --rows
+1 for --size in addition of --rows
;)
I think the 2 can (and need to) leave together so we can still easily compare
previous release to new release.
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: test
Reporter: Nick Dimiduk
Priority: Minor
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819777#comment-13819777
]
Nick Dimiduk commented on HBASE-9953:
-
Yes, my intention isn't to remove existing functionality, at least not right
away. Let's prove a new idea useful first.
PerformanceEvaluation: Decouple data size from client concurrency
-
Key: HBASE-9953
URL: https://issues.apache.org/jira/browse/HBASE-9953
Project: HBase
Issue Type: Test
Components: test
Reporter: Nick Dimiduk
Priority: Minor
PerfEval tool provides a {{--rows=R}} for specifying the number of records to
work with and requires the user provide a value of N, used as the concurrency
level. From what I can tell, every concurrent process will interact with R
rows. In order to perform an apples-to-apples test, the user must
re-calculate the value R for every new value of N.
Instead, I propose accepting a {{--size=S}} for the amount of data to
interact with and let PerfEval divide that amongst the N clients on the
user's behalf.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
