[jira] [Updated] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15122:
--
Status: Patch Available (was: Open)
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106149#comment-15106149
]
Heng Chen commented on HBASE-15122:
---
relates testcase has passed.
https://builds.apache.org/job/PreCommit-HBASE-Build/167/testReport/org.apache.hadoop.hbase.http.jmx/TestJMXJsonServlet/
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106173#comment-15106173 ] Heng Chen commented on HBASE-15128: --- Sounds good! If you have not prepared for it, i can take it. [~enis] :) > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen reassigned HBASE-15128: - Assignee: Heng Chen > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15128: -- Attachment: HBASE-15128.patch upload the first version patch. If it has no big problems, i will add relates ruby scripts. > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15128: -- Status: Patch Available (was: Open) > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110018#comment-15110018 ] Heng Chen commented on HBASE-15128: --- mvn -DHBasePatchProcess clean test-compile -DskipTests=true could pass locally. It seems QA compile hbase-server with old hbase-client.jar, not sure why, [~busbey]. > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15128: -- Attachment: HBASE-15128_v1.patch add relates ruby scripts. > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch, HBASE-15128_v1.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111876#comment-15111876 ] Heng Chen commented on HBASE-15128: --- [~busbey] I read the email you post above, could you explain in detail how can i trigger QA with USE_YETUS_PRERELEASE? > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch, HBASE-15128_v1.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116503#comment-15116503
]
Heng Chen commented on HBASE-15122:
---
[~stack] the link return 404.
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15169: -- Attachment: HBASE-15169-branch-1.1.patch Jobs done. :) > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15169: -- Status: Patch Available (was: Open) > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15165) AsyncProcess can spin wait indefinitly
[ https://issues.apache.org/jira/browse/HBASE-15165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116948#comment-15116948 ] Heng Chen commented on HBASE-15165: --- I met similar problem in our production cluster. I am not sure it is due to the same reason. As current logic, when all RPC handlers on RS are exhausted, client still sends requests to this server. Then, the requests will timeout and client do retry. It makes the situation worse. IMO client should not do retry for old requests and not accept new requests when it finds that the RS is in heavy load. To to it, we can throw some kind exception to client when the situation of no handlers to use last long enough. wdyt? [~eclark] > AsyncProcess can spin wait indefinitly > -- > > Key: HBASE-15165 > URL: https://issues.apache.org/jira/browse/HBASE-15165 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Elliott Clark >Priority: Critical > > When the max outstanding requests per region or per server is reached, all > threads trying to send more requests to that server will spin and will spin > forever with no sleep, and no regard for timeouts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116973#comment-15116973 ] Heng Chen commented on HBASE-15169: --- trigger manually, failed testcase has no relates with this patch. > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15178) metrics on web UI sometimes flakey
Heng Chen created HBASE-15178: - Summary: metrics on web UI sometimes flakey Key: HBASE-15178 URL: https://issues.apache.org/jira/browse/HBASE-15178 Project: HBase Issue Type: Bug Reporter: Heng Chen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15178) metrics on web UI sometimes flakey
[ https://issues.apache.org/jira/browse/HBASE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15178: -- Attachment: 70DD704D-7E05-49BA-AC84-0646671F67AA.png > metrics on web UI sometimes flakey > -- > > Key: HBASE-15178 > URL: https://issues.apache.org/jira/browse/HBASE-15178 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen > Attachments: 70DD704D-7E05-49BA-AC84-0646671F67AA.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15178) metrics on web UI sometimes flakey
[ https://issues.apache.org/jira/browse/HBASE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15178: -- Affects Version/s: 0.98.6 > metrics on web UI sometimes flakey > -- > > Key: HBASE-15178 > URL: https://issues.apache.org/jira/browse/HBASE-15178 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6 >Reporter: Heng Chen > Attachments: 70DD704D-7E05-49BA-AC84-0646671F67AA.png > > > see attachement -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15178) metrics on web UI sometimes flakey
[ https://issues.apache.org/jira/browse/HBASE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15178: -- Description: see attachement > metrics on web UI sometimes flakey > -- > > Key: HBASE-15178 > URL: https://issues.apache.org/jira/browse/HBASE-15178 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6 >Reporter: Heng Chen > Attachments: 70DD704D-7E05-49BA-AC84-0646671F67AA.png > > > see attachement -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15178) metrics on web UI sometimes flakey
[ https://issues.apache.org/jira/browse/HBASE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120105#comment-15120105 ] Heng Chen commented on HBASE-15178: --- duplicate with HBASE-12961 > metrics on web UI sometimes flakey > -- > > Key: HBASE-15178 > URL: https://issues.apache.org/jira/browse/HBASE-15178 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6 >Reporter: Heng Chen > Attachments: 70DD704D-7E05-49BA-AC84-0646671F67AA.png > > > see attachement -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15178) metrics on web UI sometimes flakey
[ https://issues.apache.org/jira/browse/HBASE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen resolved HBASE-15178. --- Resolution: Duplicate > metrics on web UI sometimes flakey > -- > > Key: HBASE-15178 > URL: https://issues.apache.org/jira/browse/HBASE-15178 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6 >Reporter: Heng Chen > Attachments: 70DD704D-7E05-49BA-AC84-0646671F67AA.png > > > see attachement -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120119#comment-15120119 ] Heng Chen commented on HBASE-15169: --- TestWALProcedureStoreOnHDFS still not run. org.apache.hadoop.hbase.namespace.TestNamespaceAuditor should be fixed in HBASE-15167 > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Assignee: Heng Chen >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120121#comment-15120121
]
Heng Chen commented on HBASE-15167:
---
It has relates with HBASE-14650?
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
> Attachments: blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15169: -- Attachment: HBASE-15169-branch-1.1.patch retry > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Assignee: Heng Chen >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch, > HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15182) unify normalizer switch
Heng Chen created HBASE-15182: - Summary: unify normalizer switch Key: HBASE-15182 URL: https://issues.apache.org/jira/browse/HBASE-15182 Project: HBase Issue Type: Sub-task Reporter: Heng Chen After HBASE-15128, we will have an uniform way to do switch. Let's unify normalizer into it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15128: -- Attachment: HBASE-15128_v3.patch path v3 address [~enis] review board comments. I also submit here, Let's see what QA say. > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch, HBASE-15128_v1.patch, > HBASE-15128_v3.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120225#comment-15120225 ] Heng Chen commented on HBASE-15128: --- Thans [~busbey] for your help. > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch, HBASE-15128_v1.patch, > HBASE-15128_v3.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd*
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120264#comment-15120264 ] Heng Chen commented on HBASE-15159: --- what about append? Does it need same trick? If you have not start for this issue, i can take it. [~stack] > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* > --- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd*
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120267#comment-15120267 ] Heng Chen commented on HBASE-15159: --- If you have not start, i can take it. [~stack] :) PS: What about append? Does it need same trick? > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* > --- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd*
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15159: -- Comment: was deleted (was: what about append? Does it need same trick? If you have not start for this issue, i can take it. [~stack]) > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* > --- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123157#comment-15123157 ] Heng Chen commented on HBASE-15128: --- IMO we can do a tradeoff, firstly we go on this issue and patch, after it committed, we could disable region split and merge at least. And then, we create an issue as subtask of HBASE-13936 to refactor all switches based on dynamic configuration, if nobody do it, i can take it. wdyt? [~mbertozzi] > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch, HBASE-15128_v1.patch, > HBASE-15128_v3.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen reassigned HBASE-15167:
-
Assignee: Heng Chen
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123161#comment-15123161
]
Heng Chen commented on HBASE-15167:
---
OK. Let me take it.
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15167:
--
Attachment: HBASE-15167-branch-1.1.patch
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: HBASE-15167-branch-1.1.patch, blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15167:
--
Status: Patch Available (was: Open)
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: HBASE-15167-branch-1.1.patch, blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123501#comment-15123501
]
Heng Chen commented on HBASE-15167:
---
relates test has passed.
https://builds.apache.org/job/PreCommit-HBASE-Build/348/testReport/org.apache.hadoop.hbase.namespace/TestNamespaceAuditor/
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: HBASE-15167-branch-1.1.patch, blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15191) CopyTable and VerifyReplication - Option to specify batch size, versions
[
https://issues.apache.org/jira/browse/HBASE-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124389#comment-15124389
]
Heng Chen commented on HBASE-15191:
---
{{hbase.mapreduce.scan.batchsize}} and {{hbase.mapreduce.scan.cachedrows}} meet
your needs?
> CopyTable and VerifyReplication - Option to specify batch size, versions
>
>
> Key: HBASE-15191
> URL: https://issues.apache.org/jira/browse/HBASE-15191
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 0.98.16.1
>Reporter: Parth Shah
>Priority: Minor
> Attachments: HBASE_15191.patch
>
>
> Need option to specify batch size for CopyTable and VerifyReplication. We
> are working on patch for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125941#comment-15125941 ] Heng Chen commented on HBASE-15197: --- LGTM. +1 > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127521#comment-15127521
]
Heng Chen commented on HBASE-15122:
---
Thanks [~asamir] for your digging. You should introduce ESAPI.properties in
your patch, otherwise ESAPI will init failed.
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127532#comment-15127532
]
Heng Chen commented on HBASE-15122:
---
[~busbey] i did not see any Xerces reference on master with this patch. How
did you find the reference?
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15122:
--
Attachment: HBASE-15122_v1.patch
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch,
> HBASE-15122_v1.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127546#comment-15127546 ] Heng Chen commented on HBASE-15128: --- Ping [~apurtell] [~eclark] > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch, HBASE-15128_v1.patch, > HBASE-15128_v3.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127570#comment-15127570 ] Heng Chen commented on HBASE-15197: --- See HBASE-15128 Sean Busbey comment about build with USE_YETUS_PRERELEASE > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15197: -- Assignee: Eungsop Yoo Fix Version/s: 2.0.0 Status: Patch Available (was: Open) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15197: -- Attachment: HBASE-15197.patch > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197.patch, HBASE-15197.patch, > WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127641#comment-15127641 ] Heng Chen commented on HBASE-15197: --- OK https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-HBASE-Build/382/ > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197.patch, HBASE-15197.patch, > WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127654#comment-15127654
]
Heng Chen commented on HBASE-15198:
---
How about move AbstractRpcClient#getDefaultCodec into some common place, so you
can call it directly in MultiServerCallable to avoid duplicate code. :)
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Attachments: HBASE-15198.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129689#comment-15129689 ] Heng Chen commented on HBASE-15197: --- +1 on patch v2. If no other concerns, let me commit it later. [~tedyu] :) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129909#comment-15129909 ] Heng Chen commented on HBASE-15197: --- trigger QA. https://builds.apache.org/job/PreCommit-HBASE-Build/410/ > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129927#comment-15129927
]
Heng Chen commented on HBASE-15167:
---
Ping [~ndimiduk]
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: HBASE-15167-branch-1.1.patch, blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15210) Undo crazy load balancer logging at tens of lines per millisecond
[
https://issues.apache.org/jira/browse/HBASE-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130020#comment-15130020
]
Heng Chen commented on HBASE-15210:
---
what about these logs?
{code}
2016-01-28 05:56:22,572 DEBUG
[ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,572 DEBUG
[ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,572 DEBUG
[ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1]
balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions.
{code}
> Undo crazy load balancer logging at tens of lines per millisecond
> -
>
> Key: HBASE-15210
> URL: https://issues.apache.org/jira/browse/HBASE-15210
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.2.0
>Reporter: stack
>Assignee: stack
> Attachments: 15210.patch
>
>
> Sometimes the load balancer falls into a state of crazy logging in 1.2.0
> writing out ten lines a millisecond like this:
> {code}
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,382 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,384 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,386 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 2 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,387 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region index is 3 and its region
> server contains 4 regions
> 2016-02-02 17:15:01,388 DEBUG
> [ve0524.halxg.cloudera.com,16000,1454461776827_ChoreService_1]
> balancer.BaseLoadBalancer: Lowest locality region server with non zero
> regions is ve0540.halxg.cloudera.com with locality 0.22908874
> {code}
> This issue is about less logging.
> This log was added recently by this:
> commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
> Author: tedyu
> Date: Thu Aug 13 09:11:59 2015 -0700
> HBASE-13376 Improvements to Stochastic load balancer (Vandana
> Ayyalasomayajula)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133551#comment-15133551 ] Heng Chen commented on HBASE-15197: --- Push to master. > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15197: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[
https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133634#comment-15133634
]
Heng Chen commented on HBASE-15213:
---
Thanks for your explain! [~junegunn]
You are right, we could mark entry as complete after sync successfully
immediately, there is no need to wait other entries complete and current one
come to be first in writeQueue, so as your mentioned, other handlers could
remove it while current thread in waiting state.
{quote}
This stringent condition causes O(N^2) context switches where n is the number
of concurrent handlers processing requests.
{quote}
This is why you make writeQueue from list to set, right? Sorry, i have a
little confused about this sentence.
> Fix increment performance regression caused by HBASE-8763 on branch-1.0
> ---
>
> Key: HBASE-15213
> URL: https://issues.apache.org/jira/browse/HBASE-15213
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Reporter: Junegunn Choi
>Assignee: Junegunn Choi
> Attachments: HBASE-15213.branch-1.0.patch
>
>
> This is an attempt to fix the increment performance regression caused by
> HBASE-8763 on branch-1.0.
> I'm aware that hbase.increment.fast.but.narrow.consistency was added to
> branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing
> on master branch, but anyway, this is my take on the problem.
> I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what
> caused the slowdown but I could indeed reproduce the performance regression.
> Test setup:
> - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers,
> JDK 1.7)
> - Client: Another box of the same spec
> - Increments on random 10k records on a single-region table, recreated every
> time
> Increment throughput (TPS):
> || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0
> (narrow-consistency) ||
> || 1| 2661 | 2486| 2359 |
> || 2| 5048 | 5064| 4867 |
> || 4| 7503 | 8071| 8690 |
> || 8| 10471| 10886 | 13980 |
> || 16 | 15515| 9418| 18601 |
> || 32 | 17699| 5421| 20540 |
> || 64 | 20601| 4038| 25591 |
> || 96 | 19177| 3891| 26017 |
> We can clearly observe that the throughtput degrades as we increase the
> number of concurrent requests, which led me to believe that there's severe
> context switching overhead and I could indirectly confirm that suspicion with
> cs entry in vmstat output. branch-1.0 shows a much higher number of context
> switches even with much lower throughput.
> Here are the observations:
> - WriteEntry in the writeQueue can only be removed by the very handler that
> put it, only when it is at the front of the queue and marked complete.
> - Since a WriteEntry is marked complete after the wait-loop, only one entry
> can be removed at a time.
> - This stringent condition causes O(N^2) context switches where n is the
> number of concurrent handlers processing requests.
> So what I tried here is to mark WriteEntry complete before we go into
> wait-loop. With the change, multiple WriteEntries can be shifted at a time
> without context switches. I changed writeQueue to LinkedHashSet since fast
> containment check is needed as WriteEntry can be removed by any handler.
> The numbers look good, it's virtually identical to pre-HBASE-8763 era.
> || Num threads || branch-1.0 with fix ||
> || 1| 2459 |
> || 2| 4976 |
> || 4| 8033 |
> || 8| 12292|
> || 16 | 15234|
> || 32 | 16601|
> || 64 | 19994|
> || 96 | 20052|
> So what do you think about it? Please let me know if I'm missing anything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133697#comment-15133697 ] Heng Chen commented on HBASE-15213: --- Thanks! Got it now! > Fix increment performance regression caused by HBASE-8763 on branch-1.0 > --- > > Key: HBASE-15213 > URL: https://issues.apache.org/jira/browse/HBASE-15213 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Junegunn Choi >Assignee: Junegunn Choi > Attachments: HBASE-15213.branch-1.0.patch > > > This is an attempt to fix the increment performance regression caused by > HBASE-8763 on branch-1.0. > I'm aware that hbase.increment.fast.but.narrow.consistency was added to > branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing > on master branch, but anyway, this is my take on the problem. > I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what > caused the slowdown but I could indeed reproduce the performance regression. > Test setup: > - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, > JDK 1.7) > - Client: Another box of the same spec > - Increments on random 10k records on a single-region table, recreated every > time > Increment throughput (TPS): > || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 > (narrow-consistency) || > || 1| 2661 | 2486| 2359 | > || 2| 5048 | 5064| 4867 | > || 4| 7503 | 8071| 8690 | > || 8| 10471| 10886 | 13980 | > || 16 | 15515| 9418| 18601 | > || 32 | 17699| 5421| 20540 | > || 64 | 20601| 4038| 25591 | > || 96 | 19177| 3891| 26017 | > We can clearly observe that the throughtput degrades as we increase the > number of concurrent requests, which led me to believe that there's severe > context switching overhead and I could indirectly confirm that suspicion with > cs entry in vmstat output. branch-1.0 shows a much higher number of context > switches even with much lower throughput. > Here are the observations: > - WriteEntry in the writeQueue can only be removed by the very handler that > put it, only when it is at the front of the queue and marked complete. > - Since a WriteEntry is marked complete after the wait-loop, only one entry > can be removed at a time. > - This stringent condition causes O(N^2) context switches where n is the > number of concurrent handlers processing requests. > So what I tried here is to mark WriteEntry complete before we go into > wait-loop. With the change, multiple WriteEntries can be shifted at a time > without context switches. I changed writeQueue to LinkedHashSet since fast > containment check is needed as WriteEntry can be removed by any handler. > The numbers look good, it's virtually identical to pre-HBASE-8763 era. > || Num threads || branch-1.0 with fix || > || 1| 2459 | > || 2| 4976 | > || 4| 8033 | > || 8| 12292| > || 16 | 15234| > || 32 | 16601| > || 64 | 19994| > || 96 | 20052| > So what do you think about it? Please let me know if I'm missing anything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
[
https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133812#comment-15133812
]
Heng Chen commented on HBASE-15213:
---
As for patch, how about move w.markCompleted() out of sync scope, we could
mark entry without lock, it will be better.
{code}
do {
synchronized (writeQueue) {
- // writeQueue won't be empty at this point, the following is just a
safety check
+ w.markCompleted();
+
{code}
> Fix increment performance regression caused by HBASE-8763 on branch-1.0
> ---
>
> Key: HBASE-15213
> URL: https://issues.apache.org/jira/browse/HBASE-15213
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Reporter: Junegunn Choi
>Assignee: Junegunn Choi
> Attachments: HBASE-15213.branch-1.0.patch
>
>
> This is an attempt to fix the increment performance regression caused by
> HBASE-8763 on branch-1.0.
> I'm aware that hbase.increment.fast.but.narrow.consistency was added to
> branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing
> on master branch, but anyway, this is my take on the problem.
> I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what
> caused the slowdown but I could indeed reproduce the performance regression.
> Test setup:
> - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers,
> JDK 1.7)
> - Client: Another box of the same spec
> - Increments on random 10k records on a single-region table, recreated every
> time
> Increment throughput (TPS):
> || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0
> (narrow-consistency) ||
> || 1| 2661 | 2486| 2359 |
> || 2| 5048 | 5064| 4867 |
> || 4| 7503 | 8071| 8690 |
> || 8| 10471| 10886 | 13980 |
> || 16 | 15515| 9418| 18601 |
> || 32 | 17699| 5421| 20540 |
> || 64 | 20601| 4038| 25591 |
> || 96 | 19177| 3891| 26017 |
> We can clearly observe that the throughtput degrades as we increase the
> number of concurrent requests, which led me to believe that there's severe
> context switching overhead and I could indirectly confirm that suspicion with
> cs entry in vmstat output. branch-1.0 shows a much higher number of context
> switches even with much lower throughput.
> Here are the observations:
> - WriteEntry in the writeQueue can only be removed by the very handler that
> put it, only when it is at the front of the queue and marked complete.
> - Since a WriteEntry is marked complete after the wait-loop, only one entry
> can be removed at a time.
> - This stringent condition causes O(N^2) context switches where n is the
> number of concurrent handlers processing requests.
> So what I tried here is to mark WriteEntry complete before we go into
> wait-loop. With the change, multiple WriteEntries can be shifted at a time
> without context switches. I changed writeQueue to LinkedHashSet since fast
> containment check is needed as WriteEntry can be removed by any handler.
> The numbers look good, it's virtually identical to pre-HBASE-8763 era.
> || Num threads || branch-1.0 with fix ||
> || 1| 2459 |
> || 2| 4976 |
> || 4| 8033 |
> || 8| 12292|
> || 16 | 15234|
> || 32 | 16601|
> || 64 | 19994|
> || 96 | 20052|
> So what do you think about it? Please let me know if I'm missing anything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702182#comment-14702182
]
Heng Chen commented on HBASE-14227:
---
{quote}
Rather than a boolean I'd suggest a set of flags... I guess the Java idiom is a
List of Enum... for selecting what "types" of CFs to include in the compaction.
{quote}
I agree with this approach. We remove any "mob" method. And use one {{Enum}}
parameter to select compaction types.
By the way, i found the issue was assigned to me, but
[~jingcheng...@intel.com] has uploaded the patch.
Do i need to submit patch？ [~apurtell]
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Heng Chen
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14227.patch
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702353#comment-14702353
]
Heng Chen commented on HBASE-14227:
---
As your patch, IMO there is one problem. If user want to compact MOB CF, it
needs to call something like {{Admin.compactRegion(mobRegionName)}}.
It means the user has to know what mobRegionName it is. I think this is your
implementation detail, the user has no need to know.
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Heng Chen
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14227.patch
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: (was: HBASE-14230.patch)
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: HBASE-14230.patch
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14222) Improve DrainBarrier
[
https://issues.apache.org/jira/browse/HBASE-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702396#comment-14702396
]
Heng Chen commented on HBASE-14222:
---
I think the more simple way to fix is add another check in sync block of
{{stopAndDrainOps}}. just like
{code}
private void stopAndDrainOps(boolean ignoreRepeatedCalls) throws
InterruptedException {
long oldValAndFlags;
do {
} while (!valueAndFlags.compareAndSet(oldValAndFlags, dec(oldValAndFlags) |
DRAINING_FLAG));
if (getValue(oldValAndFlags) == 1) return; // There were no operations
outstanding.
synchronized (this) {
if (getValue(oldValAndFlags) == 1) return; // check once again
this.wait();
}
}
{code}
> Improve DrainBarrier
>
>
> Key: HBASE-14222
> URL: https://issues.apache.org/jira/browse/HBASE-14222
> Project: HBase
> Issue Type: Bug
> Components: util
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14222-V2.patch, HBASE-14222.patch
>
>
> 1. {{DrainBarrier.stopAndDrainOps}} may wait forever if
> {{DrainBarrier.endOp}} changes its state and calls {{Object.notifyAll}} just
> before {{stopAndDrainOps}} enters the synchronized block to call
> {{Object.wait}}. Moreover, {{Object.wait}} may wake up false-positively, and
> {{stopAndDrainOps}} may break the block before outstanding operations are
> complete.
> 2. Some tests for {{DrainBarrier}} catch and ignore {{AssertionError}}
> explicitly thrown JUnit's {{fail}} method.
> The implementation of {{DrainBarrier}} is a little complex, and I'll fix and
> refactor it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14222) Improve DrainBarrier
[
https://issues.apache.org/jira/browse/HBASE-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702546#comment-14702546
]
Heng Chen commented on HBASE-14222:
---
yeah, we should use {{while}} to replace {{if}}, just like:
{code}
while(getValue(oldValAndFlags) != 1) {
this.wait();
}
{code}
> Improve DrainBarrier
>
>
> Key: HBASE-14222
> URL: https://issues.apache.org/jira/browse/HBASE-14222
> Project: HBase
> Issue Type: Bug
> Components: util
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14222-V2.patch, HBASE-14222.patch
>
>
> 1. {{DrainBarrier.stopAndDrainOps}} may wait forever if
> {{DrainBarrier.endOp}} changes its state and calls {{Object.notifyAll}} just
> before {{stopAndDrainOps}} enters the synchronized block to call
> {{Object.wait}}. Moreover, {{Object.wait}} may wake up false-positively, and
> {{stopAndDrainOps}} may break the block before outstanding operations are
> complete.
> 2. Some tests for {{DrainBarrier}} catch and ignore {{AssertionError}}
> explicitly thrown JUnit's {{fail}} method.
> The implementation of {{DrainBarrier}} is a little complex, and I'll fix and
> refactor it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-14227: -- Attachment: HBASE-14227_v1.patch ok, this is first try to upload patch for this issue > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702710#comment-14702710
]
Heng Chen commented on HBASE-14189:
---
Thanks for your reply!
1.
{quote}
So this indicates whether the cache on read in ON at CF level.
{quote}
No no..
{{cfBlockCacheEnabled}} only means whether BC is enabled or not at CF Level, it
refer to the HCD schema option {{BLOCKCACHE}}.
If it is OFF, all BC operations will be disabled no matter write or read.
So there need some changes in patch, and {{shouldCacheBlockOnRead}} should be
something like below:
{code}
public boolean shouldCacheBlockOnRead(BlockCategory category) {
if (!isBlockCacheEnabled()) {
return false;
}
// skip CF Level BC check
if (category == BlockCategory.INDEX ||
category == BlockCategory.BLOOM ||
(prefetchOnOpen &&
(category != BlockCategory.META &&
category != BlockCategory.UNKNOWN))) {
return true;
}
return shouldCacheDataOnRead();
}
{code}
2.
{quote}
When cache on read is OFF at CF level but cache on write is ON, what we will do?
{quote}
{{cacheDataOnRead}} means we will cache block after read it. and
{{cacheDataOnWrite}} means we will cache block after writer it.
So if {{cacheDataOnRead}} is OFF and {{cacheDataOnWrite}} is ON, we will not
cache block after read, but cache it after write, and when we read the block,
we should read it from cache.
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch
>
>
> While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
> represents for whether we should cache block after read(write) block from(to)
> hdfs. We should honour BC setting and CF Level cache setting while using
> BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14189:
--
Attachment: HBASE-14189_v2.patch
update patch change {{shouldCacheBlockOnRead}} logic
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch
>
>
> While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
> represents for whether we should cache block after read(write) block from(to)
> hdfs. We should honour BC setting and CF Level cache setting while using
> BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14222) Improve DrainBarrier
[
https://issues.apache.org/jira/browse/HBASE-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702746#comment-14702746
]
Heng Chen commented on HBASE-14222:
---
The problem is below:
{code}
1.if (getValue(oldValAndFlags) == 1) {
2. synchronized (this) { this.notifyAll(); }
3.}
{code}
{code}
1.if (getValue(oldValAndFlags) == 1) return;
2.synchronized (this) { this.wait(); }
{code}
there will be problem when wait Thread run to line 2 but lock was acquired by
notify thread, the result is wait thread will wait forever.
So we add condition check in wait thread sync block.
{code}
while(getValue(oldValAndFlags) != 1) {
this.wait();
}
{code}
when wait thread acquired lock if will check condition first, because notify
has run, the condition {{getValue(oldValAndFlags) != 1}} will be false.
and wait thread will not wait.
> Improve DrainBarrier
>
>
> Key: HBASE-14222
> URL: https://issues.apache.org/jira/browse/HBASE-14222
> Project: HBase
> Issue Type: Bug
> Components: util
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14222-V2.patch, HBASE-14222.patch
>
>
> 1. {{DrainBarrier.stopAndDrainOps}} may wait forever if
> {{DrainBarrier.endOp}} changes its state and calls {{Object.notifyAll}} just
> before {{stopAndDrainOps}} enters the synchronized block to call
> {{Object.wait}}. Moreover, {{Object.wait}} may wake up false-positively, and
> {{stopAndDrainOps}} may break the block before outstanding operations are
> complete.
> 2. Some tests for {{DrainBarrier}} catch and ignore {{AssertionError}}
> explicitly thrown JUnit's {{fail}} method.
> The implementation of {{DrainBarrier}} is a little complex, and I'll fix and
> refactor it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: HBASE-14230.patch
QA confused me... retry
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: (was: HBASE-14230.patch)
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702781#comment-14702781
]
Heng Chen commented on HBASE-14227:
---
{quote}
HRegionInfo is public exposed. Can we avoid getMobRegionInfo added there?
MobUtil can have this method?
{quote}
{{MobUtils}} is in hbase-server module, but {{HbaseAdmin}} is in hbase-client.
If move this method into {{MobUtils}}, the client will depend on server jar,
is it good?
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Heng Chen
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14227.patch, HBASE-14227_v1.patch
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14222) Improve DrainBarrier
[
https://issues.apache.org/jira/browse/HBASE-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702784#comment-14702784
]
Heng Chen commented on HBASE-14222:
---
when called notify, it means the last outstanding operation is over, and
{{getValue(oldValAndFlags) == 1}} will be true
> Improve DrainBarrier
>
>
> Key: HBASE-14222
> URL: https://issues.apache.org/jira/browse/HBASE-14222
> Project: HBase
> Issue Type: Bug
> Components: util
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14222-V2.patch, HBASE-14222.patch
>
>
> 1. {{DrainBarrier.stopAndDrainOps}} may wait forever if
> {{DrainBarrier.endOp}} changes its state and calls {{Object.notifyAll}} just
> before {{stopAndDrainOps}} enters the synchronized block to call
> {{Object.wait}}. Moreover, {{Object.wait}} may wake up false-positively, and
> {{stopAndDrainOps}} may break the block before outstanding operations are
> complete.
> 2. Some tests for {{DrainBarrier}} catch and ignore {{AssertionError}}
> explicitly thrown JUnit's {{fail}} method.
> The implementation of {{DrainBarrier}} is a little complex, and I'll fix and
> refactor it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14254) Wrong error message when throwing NamespaceNotFoundException in shell
[
https://issues.apache.org/jira/browse/HBASE-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702847#comment-14702847
]
Heng Chen commented on HBASE-14254:
---
{code}
hbase(main):004:0> create 'ns:t1', {NAME => 'f1'}
ERROR: org.apache.hadoop.hbase.NamespaceNotFoundException: ns
at
org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2378)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1444)
at
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:422)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:48502)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2112)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
{code}
I use branch 1.1.1
> Wrong error message when throwing NamespaceNotFoundException in shell
> -
>
> Key: HBASE-14254
> URL: https://issues.apache.org/jira/browse/HBASE-14254
> Project: HBase
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
>
> Wrong error message when throwing NamespaceNotFoundException in shell
> {code}
> hbase(main):004:0> create 'ns:t1', {NAME => 'f1'}
> ERROR: Unknown namespace ns:t1!
> {code}
> The namespace shoud be {color:red}ns {color}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704166#comment-14704166
]
Heng Chen commented on HBASE-14230:
---
thanks for your reply!
{quote}
In https://builds.apache.org/job/PreCommit-HBASE-Build/15166/consoleFull, you
can see 66 occurrences of 'unable to create new native thread'
{quote}
I rerun the failed testcases locally, they failed no matter whether apply this
patch or not.
So I don't think the failed testcases are caused by this issue. The patch only
has little changes.
Any concerns about the patch itself? [~yuzhih...@gmail.com]
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704210#comment-14704210
]
Heng Chen commented on HBASE-14230:
---
{quote}
Shall we wait for trunk build to stabilize before integrating this patch ?
{quote}
Agreed. I will retry later. Thanks for your review. [~yuzhih...@gmail.com]
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-14227: -- Attachment: HBASE-14227_v2.patch Thanks for your review. update patch as your suggestions! [~jingcheng...@intel.com] > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14265) we should forbidden create table using 'hbase' namespace
Heng Chen created HBASE-14265:
-
Summary: we should forbidden create table using 'hbase' namespace
Key: HBASE-14265
URL: https://issues.apache.org/jira/browse/HBASE-14265
Project: HBase
Issue Type: Bug
Reporter: Heng Chen
Now, there is no limit for users who can create table under 'hbase' NameSpace.
I think it has some risk.
Because we use {{TableName.systemTable}} to decide whether this table is System
or not.
But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
{code}
if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
this.namespaceAsString = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
this.systemTable = true;
}
{code}
And we treat system table and normal table differently.
For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush fast
if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14262) Big Trunk unit tests failing with "OutOfMemoryError: unable to create new native thread"
[
https://issues.apache.org/jira/browse/HBASE-14262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704287#comment-14704287
]
Heng Chen commented on HBASE-14262:
---
Agree with [~Apache9], we can use BlockingQueue in {{TestContext}} instead of
{{Set testThreads}}, then we use a threads pool with less thread
number, which pick task from BlockingQueue to run.
> Big Trunk unit tests failing with "OutOfMemoryError: unable to create new
> native thread"
>
>
> Key: HBASE-14262
> URL: https://issues.apache.org/jira/browse/HBASE-14262
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>
> The bit unit tests are coming in with OOME, can't create native threads.
> I was also getting the OOME locally running on MBP. git bisect got me to
> HBASE-13065, where we upped the test heap for TestDistributedLogSplitting
> back in feb. Around the time that this went in, we had similar OOME issues
> but then it was because we were doing 32bit JVMs. It does not seem to be the
> case here.
> A recent run failed all the below and most are OOME:
> {code}
> {color:red}-1 core tests{color}. The patch failed these unit tests:
>
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint
>
> org.apache.hadoop.hbase.replication.TestPerTableCFReplication
>
> org.apache.hadoop.hbase.wal.TestBoundedRegionGroupingProvider
>
> org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSCompressed
>
> org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpointNoMaster
>
> org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS
>
> org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
> org.apache.hadoop.hbase.replication.TestMasterReplication
> org.apache.hadoop.hbase.mapred.TestTableMapReduce
>
> org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
> org.apache.hadoop.hbase.regionserver.TestRegionFavoredNodes
>
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
> org.apache.hadoop.hbase.zookeeper.TestZKLeaderManager
>
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable
> org.apache.hadoop.hbase.TestGlobalMemStoreSize
> org.apache.hadoop.hbase.wal.TestWALFiltering
>
> org.apache.hadoop.hbase.replication.TestReplicationSmallTests
>
> org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
> org.apache.hadoop.hbase.replication.TestReplicationWithTags
>
> org.apache.hadoop.hbase.master.procedure.TestTruncateTableProcedure
>
> org.apache.hadoop.hbase.replication.TestReplicationChangingPeerRegionservers
>
> org.apache.hadoop.hbase.wal.TestDefaultWALProviderWithHLogKey
>
> org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush
>
> org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient
>
> org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure
> org.apache.hadoop.hbase.wal.TestWALFactory
>
> org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure
>
> org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
>
> org.apache.hadoop.hbase.master.procedure.TestAddColumnFamilyProcedure
>
> org.apache.hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat
>
> org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure
>
> org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence
> org.apache.hadoop.hbase.TestStochasticBalancerJmxMetrics
> {color:red}-1 core zombie tests{color}. There are 16 zombie test(s):
> at
> org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testVisibilityLabelsWithComplexLabels(TestVisibilityLabels.java:216)
> at
> org.apache.hadoop.hbase.mapred.TestTableInputFormat.testTableRecordReaderScannerFail(TestTableInputFormat.java:281)
> at
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication.testMultiSlaveReplication(TestMultiSlaveReplication.java:129)
> at
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning(TestHRegion.java:3799)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-14227: -- Attachment: HBASE-14227_v3.patch Thanks for your review. update patch as your suggestions! [~anoop.hbase] changes: * add comments for CompactType enum and param 'compactType' > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbidden create table using 'hbase' namespace
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Attachment: HBASE-14265.patch
prepare a patch.
> we should forbidden create table using 'hbase' namespace
>
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbidden create table using 'hbase' namespace
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Status: Patch Available (was: Open)
> we should forbidden create table using 'hbase' namespace
>
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
[
https://issues.apache.org/jira/browse/HBASE-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704523#comment-14704523
]
Heng Chen commented on HBASE-14267:
---
IMO if you need to update the row content, you'd better clone it outside after
call {{Result.getRow()}}
> In Mapreduce on HBase scenario, restart in TableInputFormat will result in
> getting wrong data.
> --
>
> Key: HBASE-14267
> URL: https://issues.apache.org/jira/browse/HBASE-14267
> Project: HBase
> Issue Type: Bug
> Components: Client, mapreduce
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Attachments: HBASE_14267_trunk_v1.patch
>
>
> When I run a mapreduce job on HBase, I will modify the row got from
> Result.getRow(), for example, reverse the row. Since my program is very
> complicated to handle data, it takes long time, and the lease int Region
> server expired.
> Result#195
> {code}
> public byte [] getRow() {
> if (this.row == null) {
> this.row = (this.cells == null || this.cells.length == 0) ?
> null :
> CellUtil.cloneRow(this.cells[0]);
> }
> return this.row;
> }
> {code}
> TableInputFormat will restart the scan from last row, but the row has been
> modified, so it will read wrong data.
> TableRecordReaderImpl#218
> {code}
> } catch (IOException e) {
> // do not retry if the exception tells us not to do so
> if (e instanceof DoNotRetryIOException) {
> throw e;
> }
> // try to handle all other IOExceptions by restarting
> // the scanner, if the second call fails, it will be rethrown
> LOG.info("recovered from " + StringUtils.stringifyException(e));
> if (lastSuccessfulRow == null) {
> LOG.warn("We are restarting the first next() invocation," +
> " if your mapper has restarted a few other times like this" +
> " then you should consider killing this job and investigate" +
> " why it's taking so long.");
> }
> if (lastSuccessfulRow == null) {
> restart(scan.getStartRow());
> } else {
> restart(lastSuccessfulRow);
> scanner.next();// skip presumed already mapped row
> }
> value = scanner.next();
> if (value != null && value.isStale()) numStale++;
> numRestarts++;
> }
> if (value != null && value.size() > 0) {
> key.set(value.getRow());
> lastSuccessfulRow = key.get();
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706017#comment-14706017
]
Heng Chen commented on HBASE-14227:
---
How about this
{code}
+ /**
+ * {@code NORMAL} means do store files compaction;
+ * {@code MOB} means do mob files compaction.
+ * It is different, MOB compaction only take care of MOB files.
+ * Generally, MOB CF is larger than normal CF, a compaction on the MOB CF
will be more IO oriented than the normal CF.
+ * And the impact of the number of MOB files is not significant in read,
usually they are deleted when expired.
+ * So sometimes MOB compactions are not necessary. More MOB detail see
HBASE-11339
+ * */
+ public enum CompactType {
+
+NORMAL(0),
+MOB (1);
+
+CompactType(int value) {}
+ }
{code}
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Heng Chen
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14227.patch, HBASE-14227_v1.patch,
> HBASE-14227_v2.patch, HBASE-14227_v3.patch
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: HBASE-14230.patch
retry
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: (was: HBASE-14230.patch)
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbid creating table using 'hbase' namespace except by superuser
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Attachment: HBASE-14265_v2.patch
{quote}
Have you tried this patch with security features active? Does this prevent the
creation of the ACL and labels tables? If so this can be fixed by conditionally
allowing the create if the effective user is a superuser, see
Superusers#isSuperUser
{quote}
I see... You are right. I update patch as your suggestions! Thanks for your
review! [~apurtell]
> we should forbid creating table using 'hbase' namespace except by superuser
> ---
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch, HBASE-14265_v2.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14265) we should forbid creating table using 'hbase' namespace except by superuser
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706359#comment-14706359
]
Heng Chen commented on HBASE-14265:
---
[~apurtell] I reconsider this patch, and found this is not what i want.
As your suggestions, if security is active, everything is OK. And we has no
need to check table's NS in HMaster, because we can set normal users ACLs to
'hbase' Namespace.
But if we disable the security, user can still create table using 'hbase'
namespace.
IMO we should forbid this action.
> we should forbid creating table using 'hbase' namespace except by superuser
> ---
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch, HBASE-14265_v2.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706430#comment-14706430
]
Heng Chen commented on HBASE-14279:
---
yeah... maybe we can replace {{ConcurrentMap> container;}} with
{{ConcurrentMap>> container;}}
{{Pair.getFirst()}} is Lock for current key. {{Pair.getSecond()}} is real set.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Priority: Minor
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbid creating table using 'hbase' namespace except by superuser
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Attachment: HBASE-14265_v3.patch
When security enabled, we can depend on ACLs to protect 'hbase' namespace, So
i update the patch, changes below:
* we only forbid creating table using 'hbase' namespace when security disabled
> we should forbid creating table using 'hbase' namespace except by superuser
> ---
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch, HBASE-14265_v2.patch,
> HBASE-14265_v3.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14278) Fix NPE that is showing up since HBASE-14274 went in
[
https://issues.apache.org/jira/browse/HBASE-14278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706718#comment-14706718
]
Heng Chen commented on HBASE-14278:
---
I reproduce the problem locally. I think there is some risk in
{{JmxCacheBusterRunnable.run}},
{code}
if (DefaultMetricsSystem.instance() != null) {
DefaultMetricsSystem.instance().stop();
DefaultMetricsSystem.instance().start();
}
{code}
Maybe stop is not finished, but start is begin, so there are some conflicts.
So i add {{Thread.sleep(1000)}} after
{{DefaultMetricsSystem.instance().stop()}} and rerun testcase. NPE never
appear!
Hope it will help, and i will dig it too.
{code}
if (DefaultMetricsSystem.instance() != null) {
DefaultMetricsSystem.instance().stop();
Thread.sleep(1000);
DefaultMetricsSystem.instance().start();
}
{code}
> Fix NPE that is showing up since HBASE-14274 went in
>
>
> Key: HBASE-14278
> URL: https://issues.apache.org/jira/browse/HBASE-14278
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: stack
>
> Saw this in TestDistributedLogSplitting after HBASE-14274 was applied.
> {code}
> 119113 2015-08-20 15:31:10,704 WARN [HBase-Metrics2-1]
> impl.MetricsConfig(124): Cannot locate configuration: tried
> hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
> 119114 2015-08-20 15:31:10,710 ERROR [HBase-Metrics2-1]
> lib.MethodMetric$2(118): Error invoking method getBlocksTotal
> 119115 java.lang.reflect.InvocationTargetException
> 119116 › at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
> 119117 › at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 119118 › at java.lang.reflect.Method.invoke(Method.java:606)
> 119119 › at
> org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
> 119120 › at
> org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
> 119121 › at
> org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:387)
> 119122 › at
> org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
> 119123 › at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
> 119124 › at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
> 119125 › at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
> 119126 › at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
> 119127 › at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
> 119128 › at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> 119129 › at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
> 119130 › at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:221)
> 119131 › at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:96)
> 119132 › at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:245)
> 119133 › at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:229)
> 119134 › at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
> 119135 › at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 119136 › at java.lang.reflect.Method.invoke(Method.java:606)
> 119137 › at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:290)
> 119138 › at com.sun.proxy.$Proxy13.postStart(Unknown Source)
> 119139 › at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:185)
> 119140 › at
> org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:81)
> 119141 › at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 119142 › at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 119143 › at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> 119144 › at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> 119145 › at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 119146 › at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 119147 › at java.lang.Thread.run(
[jira] [Updated] (HBASE-14265) we should forbid creating table using 'hbase' namespace except by superuser
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Attachment: HBASE-14265_v4.patch
Thanks for your review.
Replace {{SystemNamespaceAccessException}} with {{AccessDeniedException}} as
suggestions.
> we should forbid creating table using 'hbase' namespace except by superuser
> ---
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch, HBASE-14265_v2.patch,
> HBASE-14265_v3.patch, HBASE-14265_v4.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbid creating table using 'hbase' namespace except by superuser
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Status: Patch Available (was: Open)
> we should forbid creating table using 'hbase' namespace except by superuser
> ---
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch, HBASE-14265_v2.patch,
> HBASE-14265_v3.patch, HBASE-14265_v4.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: HBASE-14230.patch
retry
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: (was: HBASE-14230.patch)
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14189:
--
Attachment: (was: HBASE-14189_v2.patch)
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch
>
>
> While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
> represents for whether we should cache block after read(write) block from(to)
> hdfs. We should honour BC setting and CF Level cache setting while using
> BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14189:
--
Attachment: HBASE-14189_v2.patch
QA not run. retry
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch
>
>
> While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
> represents for whether we should cache block after read(write) block from(to)
> hdfs. We should honour BC setting and CF Level cache setting while using
> BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbid creating table using 'hbase' namespace except by superuser
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Status: Patch Available (was: Open)
> we should forbid creating table using 'hbase' namespace except by superuser
> ---
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch, HBASE-14265_v2.patch,
> HBASE-14265_v3.patch, HBASE-14265_v4.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbid creating table using 'hbase' namespace except by superuser
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Status: Open (was: Patch Available)
> we should forbid creating table using 'hbase' namespace except by superuser
> ---
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14265.patch, HBASE-14265_v2.patch,
> HBASE-14265_v3.patch, HBASE-14265_v4.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-14265) we should forbid creating table using 'hbase' namespace except by superuser
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen reassigned HBASE-14265:
-
Assignee: Heng Chen
> we should forbid creating table using 'hbase' namespace except by superuser
> ---
>
> Key: HBASE-14265
> URL: https://issues.apache.org/jira/browse/HBASE-14265
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14265.patch, HBASE-14265_v2.patch,
> HBASE-14265_v3.patch, HBASE-14265_v4.patch
>
>
> Now, there is no limit for users who can create table under 'hbase'
> NameSpace. I think it has some risk.
> Because we use {{TableName.systemTable}} to decide whether this table is
> System or not.
> But as code, {{TableName.systemTable}} will be true, if NS equals "hbase'
> {code}
> if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
> this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
> this.namespaceAsString =
> NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
> this.systemTable = true;
> }
> {code}
>
> And we treat system table and normal table differently.
> For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
> fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710691#comment-14710691
]
Heng Chen commented on HBASE-14279:
---
May I upload a patch ?
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Priority: Minor
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14279:
--
Attachment: HBASE-14279.patch
first try for the issue.
Could you help me review the patch, [~ikeda]? Thanks
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710930#comment-14710930
]
Heng Chen commented on HBASE-14230:
---
Any suggestions? [~yuzhih...@gmail.com]
> replace reflection in FSHlog with
> HdfsDataOutputStream#getCurrentBlockReplication()
> ---
>
> Key: HBASE-14230
> URL: https://issues.apache.org/jira/browse/HBASE-14230
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14230.patch
>
>
> As comment TODO said, we use
> {{HdfsDataOutputStream#getCurrentBlockReplication}} and
> {{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting
[
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14189:
--
Attachment: HBASE-14189_v3.patch
Fix bug which cause testcase failed
> BlockCache options should consider CF Level BlockCacheEnabled setting
> -
>
> Key: HBASE-14189
> URL: https://issues.apache.org/jira/browse/HBASE-14189
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14189.patch, HBASE-14189_v1.patch,
> HBASE-14189_v2.patch, HBASE-14189_v3.patch
>
>
> While using BlockCache, we use {{cacheDataOnRead}}({{cacheDataOnWrite}})
> represents for whether we should cache block after read(write) block from(to)
> hdfs. We should honour BC setting and CF Level cache setting while using
> BlockCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
