[jira] [Updated] (HBASE-20192) RedirectServlet not getting registered in HMaster.putUpJettyServer() in local mode
[ https://issues.apache.org/jira/browse/HBASE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-20192: Resolution: Fixed Status: Resolved (was: Patch Available) I also can not reproduce this issue any more. HBASE-20224 fixed this closing this issue [~uagashe]. > RedirectServlet not getting registered in HMaster.putUpJettyServer() in local > mode > -- > > Key: HBASE-20192 > URL: https://issues.apache.org/jira/browse/HBASE-20192 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.0.0-beta-2 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Major > Attachments: HBASE-20192.branch-2.0.01.patch, Screen Shot 2018-03-14 > at 09.23.34.png, Screen Shot 2018-03-14 at 09.24.06.png > > > Jetty is returning 404 when trying to open master-status page from RS status > page in local cluster mode. > After some debugging it looks like request never hits jetty RedirectServlet > so i assume RedirectServlet is not properly registered in jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20192) RedirectServlet not getting registered in HMaster.putUpJettyServer() in local mode
[ https://issues.apache.org/jira/browse/HBASE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-20192: Status: Patch Available (was: Open) > RedirectServlet not getting registered in HMaster.putUpJettyServer() in local > mode > -- > > Key: HBASE-20192 > URL: https://issues.apache.org/jira/browse/HBASE-20192 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.0.0-beta-2 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20192.branch-2.0.01.patch, Screen Shot 2018-03-14 > at 09.23.34.png, Screen Shot 2018-03-14 at 09.24.06.png > > > Jetty is returning 404 when trying to open master-status page from RS status > page in local cluster mode. > After some debugging it looks like request never hits jetty RedirectServlet > so i assume RedirectServlet is not properly registered in jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20192) RedirectServlet not getting registered in HMaster.putUpJettyServer() in local mode
[ https://issues.apache.org/jira/browse/HBASE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398299#comment-16398299 ] Samir Ahmic commented on HBASE-20192: - Here is a patch for issue. > RedirectServlet not getting registered in HMaster.putUpJettyServer() in local > mode > -- > > Key: HBASE-20192 > URL: https://issues.apache.org/jira/browse/HBASE-20192 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.0.0-beta-2 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20192.branch-2.0.01.patch, Screen Shot 2018-03-14 > at 09.23.34.png, Screen Shot 2018-03-14 at 09.24.06.png > > > Jetty is returning 404 when trying to open master-status page from RS status > page in local cluster mode. > After some debugging it looks like request never hits jetty RedirectServlet > so i assume RedirectServlet is not properly registered in jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20192) RedirectServlet not getting registered in HMaster.putUpJettyServer() in local mode
[ https://issues.apache.org/jira/browse/HBASE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-20192: Attachment: HBASE-20192.branch-2.0.01.patch > RedirectServlet not getting registered in HMaster.putUpJettyServer() in local > mode > -- > > Key: HBASE-20192 > URL: https://issues.apache.org/jira/browse/HBASE-20192 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.0.0-beta-2 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20192.branch-2.0.01.patch, Screen Shot 2018-03-14 > at 09.23.34.png, Screen Shot 2018-03-14 at 09.24.06.png > > > Jetty is returning 404 when trying to open master-status page from RS status > page in local cluster mode. > After some debugging it looks like request never hits jetty RedirectServlet > so i assume RedirectServlet is not properly registered in jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20192) RedirectServlet not getting registered in HMaster.putUpJettyServer() in local mode
Samir Ahmic created HBASE-20192: --- Summary: RedirectServlet not getting registered in HMaster.putUpJettyServer() in local mode Key: HBASE-20192 URL: https://issues.apache.org/jira/browse/HBASE-20192 Project: HBase Issue Type: Bug Components: UI Affects Versions: 2.0.0-beta-2 Reporter: Samir Ahmic Assignee: Samir Ahmic Fix For: 2.0.0 Attachments: Screen Shot 2018-03-14 at 09.23.34.png, Screen Shot 2018-03-14 at 09.24.06.png Jetty is returning 404 when trying to open master-status page from RS status page in local cluster mode. After some debugging it looks like request never hits jetty RedirectServlet so i assume RedirectServlet is not properly registered in jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[
https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397786#comment-16397786
]
Samir Ahmic commented on HBASE-15121:
-
[~stack] it was some time ago :), but as i can recall i was using cluster setup
and i'm pretty sure i was running
{code}
hbase org.apache.hadoop.hbase.IntegrationTestsDriver -r IntegrationTestMTTR
{code}
[~uagashe] agree, WARN sounds more appropriate ,
> ConnectionImplementation#locateRegionInMeta() issue when master is restarted
>
>
> Key: HBASE-15121
> URL: https://issues.apache.org/jira/browse/HBASE-15121
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-15121-v0.patch, HBASE-15121-v0.patch
>
>
> I notice this issue while i was running
> IntegrationTestMTTR#testRestartMaster() test was failing on put operation.
> Here is sequence of events from logs leading to failed put operation:
> Master restart
> {code}
> INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh
> hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' |
> cut -d ' ' -f2 | xargs kill -s SIGKILL"]
> {code}
> Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17
> (this is additional logging inspecting metaKey which is used to search
> hbase:meta )
> {code}
> 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9]
> client.ConnectionImplementation: metaKey inspection:
> table=IntegrationTestMTTRLoadTestTool row=
> 70efdf2ec9b086079795c442636b55fb-17 metaKey=
> IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99
> {code}
> Client throwing TableNotFoundException (hbase:meta scan returned null)
> {code}
> 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5]
> client.ConnectionImplementation: regionInfo result is null:
> HBaseWriterThread_5 throwing TableNotFoundException logging details
> table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17
> metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99
> 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess:
> Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException:
> IntegrationTestMTTRLoadTestTool
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
> {code}
> And as result we have failed insert operation:
> {code}
> 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter:
> Failed to insert: 17 after 60046ms; region information: cached:
> region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7.,
> hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors:
> exception from null for 70efdf2ec9b086079795c442636b55fb-17
> org.apache.hadoop.hbase.TableNotFoundException:
> IntegrationTestMTTRLoadTestTool
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
> at
[jira] [Commented] (HBASE-18615) hbase-rest tests fail in hbase-2.0.0-alpha2
[ https://issues.apache.org/jira/browse/HBASE-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136303#comment-16136303 ] Samir Ahmic commented on HBASE-18615: - +1 lgtm. > hbase-rest tests fail in hbase-2.0.0-alpha2 > --- > > Key: HBASE-18615 > URL: https://issues.apache.org/jira/browse/HBASE-18615 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-alpha-3 > > Attachments: 18615.2.txt, 18615.txt, 18615.v3.txt, > HBASE-18615.branch-2.001.patch, HBASE-18615.branch-2.002.patch > > > Pointed out by Andrew on VOTE mail for hbase-2.0.0-alpha2 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18615) hbase-rest tests fail in hbase-2.0.0-alpha2
[
https://issues.apache.org/jira/browse/HBASE-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134062#comment-16134062
]
Samir Ahmic commented on HBASE-18615:
-
[~stack] small tip for test failures we probably need this line in
HBaseRESTTestingUtility.java also:
{code}
@@ -62,6 +62,7 @@ public class HBaseRESTTestingUtility {
// set up the Jersey servlet container for Jetty
ResourceConfig app = new ResourceConfig();
+
app.packages("org.apache.hadoop.hbase.rest").register(Jackson1Feature.class);
ServletHolder sh = new ServletHolder(new ServletContainer(app));
{code}
> hbase-rest tests fail in hbase-2.0.0-alpha2
> ---
>
> Key: HBASE-18615
> URL: https://issues.apache.org/jira/browse/HBASE-18615
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
> Attachments: 18615.2.txt, 18615.txt, 18615.v3.txt,
> HBASE-18615.branch-2.001.patch
>
>
> Pointed out by Andrew on VOTE mail for hbase-2.0.0-alpha2
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HBASE-18506) java.lang.AbstractMethodError in hbase REST server
[
https://issues.apache.org/jira/browse/HBASE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic resolved HBASE-18506.
-
Resolution: Fixed
Fix Version/s: 3.0.0
This is fixed for for master branch. For branch-2 there will be different
solution under HBASE-18615
> java.lang.AbstractMethodError in hbase REST server
> --
>
> Key: HBASE-18506
> URL: https://issues.apache.org/jira/browse/HBASE-18506
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Just run it this one while testing some scripts. Basically any call to
> service will end up with 500 error. After some checking it looks like we have
> some issues with dependencies incompatibility.
> Here is more details:
> {code}
> Stack trace:
> 2017-08-02 20:46:25,407 WARN [qtp422330142-30] servlet.ServletHandler: Error
> for /status/cluster
> java.lang.AbstractMethodError:
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:119)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:298)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
> at
> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:77)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:518)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> and here are suspects from lib dir
> {code}
> $ grep -r "UriBuilder" .
> Binary file ./javax.ws.rs-api-2.0.1.jar matches
> Binary file ./jersey-common-2.25.1.jar matches
> Binary file ./jersey-core-1.9.jar matches
> {code}
> I have also checked hbase-1.2.6 we have only jersey-core-1.9.jar there
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18518) Remove jersey1* dependencies from project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-18518: Fix Version/s: (was: 2.0.0-alpha-3) 3.0.0 > Remove jersey1* dependencies from project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 3.0.0 > > Attachments: HBASE-18518-master-01.patch, HBASE-18518-master-02.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18518) Remove jersey1* dependencies from project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133212#comment-16133212 ] Samir Ahmic commented on HBASE-18518: - Cool, thanks [~stack] i was running test and starting from tar ball. That explains inconsistency. If you need assistant on HBASE-18615 feel free to ping me, i have fresh wounds fighting jersey1 + jersey2 on jetty war :). > Remove jersey1* dependencies from project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18518-master-01.patch, HBASE-18518-master-02.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18518) Remove jersey1* dependencies from project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131833#comment-16131833 ] Samir Ahmic edited comment on HBASE-18518 at 8/18/17 7:13 AM: -- Ignore last one :) you have already committed this on branch-2 i will do some testing with rest. was (Author: asamir): Ignore last one :) you have already committed this on branch-2. > Remove jersey1* dependencies from project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18518-master-01.patch, HBASE-18518-master-02.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18518) Remove jersey1* dependencies from project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131833#comment-16131833 ] Samir Ahmic edited comment on HBASE-18518 at 8/18/17 7:12 AM: -- Ignore last one :) you have already committed this on branch-2. was (Author: asamir): Ignore last one :) i have already committed this on branch-2. > Remove jersey1* dependencies from project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18518-master-01.patch, HBASE-18518-master-02.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18518) Remove jersey1* dependencies from project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131833#comment-16131833 ] Samir Ahmic commented on HBASE-18518: - Ignore last one :) i have already committed this on branch-2. > Remove jersey1* dependencies from project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18518-master-01.patch, HBASE-18518-master-02.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18518) Remove jersey1* dependencies from project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131825#comment-16131825 ] Samir Ahmic commented on HBASE-18518: - [~stack] interesting... i made this this patch for master and have tested full flow there, tests + running rest server locally and as i remember rest was working fine. Let me try apply patch on top of 2.0.0-alpha and check what will happening :) > Remove jersey1* dependencies from project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18518-master-01.patch, HBASE-18518-master-02.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18615) hbase-rest tests fail in hbase-2.0.0-alpha2
[ https://issues.apache.org/jira/browse/HBASE-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131715#comment-16131715 ] Samir Ahmic commented on HBASE-18615: - [~stack] please take look at HBASE-18518 i was replacing jersey1 with jersey2 in hbase-rest on master maybe we can do same on hbase-2.0.0-alpha2. > hbase-rest tests fail in hbase-2.0.0-alpha2 > --- > > Key: HBASE-18615 > URL: https://issues.apache.org/jira/browse/HBASE-18615 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack > > Pointed out by Andrew on VOTE mail -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18506) java.lang.AbstractMethodError in hbase REST server
[
https://issues.apache.org/jira/browse/HBASE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116998#comment-16116998
]
Samir Ahmic commented on HBASE-18506:
-
HBASE-18518 will resolve this issue
> java.lang.AbstractMethodError in hbase REST server
> --
>
> Key: HBASE-18506
> URL: https://issues.apache.org/jira/browse/HBASE-18506
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
>
> Just run it this one while testing some scripts. Basically any call to
> service will end up with 500 error. After some checking it looks like we have
> some issues with dependencies incompatibility.
> Here is more details:
> {code}
> Stack trace:
> 2017-08-02 20:46:25,407 WARN [qtp422330142-30] servlet.ServletHandler: Error
> for /status/cluster
> java.lang.AbstractMethodError:
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:119)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:298)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
> at
> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:77)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:518)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> and here are suspects from lib dir
> {code}
> $ grep -r "UriBuilder" .
> Binary file ./javax.ws.rs-api-2.0.1.jar matches
> Binary file ./jersey-common-2.25.1.jar matches
> Binary file ./jersey-core-1.9.jar matches
> {code}
> I have also checked hbase-1.2.6 we have only jersey-core-1.9.jar there
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18518) Remove jersey1* dependencies from project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-18518: Attachment: HBASE-18518-master-02.patch Here is new patch addressing test failure. > Remove jersey1* dependencies from project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 3.0.0 > > Attachments: HBASE-18518-master-01.patch, HBASE-18518-master-02.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18518) Remove jersey1* dependencies form project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115512#comment-16115512 ] Samir Ahmic edited comment on HBASE-18518 at 8/5/17 8:58 PM: - [~mdrob] bq. Also, do we need to make changes in the hbase-rest/pom.xml to specify the correct version of jersey? This is already specified in parent pom.xml in root dir. bq. Where do we need the transitive hadoop jersey available? We need it in every module when we run tests which requires starting hadoop mini cluster. Without it staring hadoop mini cluster will fail and consequently our tests will fail. Do you have any suggestions how may improve current dependency management ? was (Author: asamir): [~mdrob] bq. Also, do we need to make changes in the hbase-rest/pom.xml to specify the correct version of jersey? This is already specified in parent pom.xml in root dir. bq. Where do we need the transitive hadoop jersey available? We need it every module when we run tests which requires starting hadoop mini cluster. Without it staring hadoop mini cluster will fail and consequently our tests will fail. Do you have any suggestions how may improve current dependency management ? > Remove jersey1* dependencies form project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 3.0.0 > > Attachments: HBASE-18518-master-01.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18518) Remove jersey1* dependencies form project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115512#comment-16115512 ] Samir Ahmic commented on HBASE-18518: - [~mdrob] bq. Also, do we need to make changes in the hbase-rest/pom.xml to specify the correct version of jersey? This is already specified in parent pom.xml in root dir. bq. Where do we need the transitive hadoop jersey available? We need it every module when we run tests which requires starting hadoop mini cluster. Without it staring hadoop mini cluster will fail and consequently our tests will fail. Do you have any suggestions how may improve current dependency management ? > Remove jersey1* dependencies form project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 3.0.0 > > Attachments: HBASE-18518-master-01.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18518) Remove jersey1* dependencies form project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115496#comment-16115496 ] Samir Ahmic commented on HBASE-18518: - Sure thing [~mdrob]. I will revisit tests to check how change impact them. > Remove jersey1* dependencies form project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 3.0.0 > > Attachments: HBASE-18518-master-01.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18518) Remove jersey1* dependencies form project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-18518: Fix Version/s: 3.0.0 Status: Patch Available (was: Open) > Remove jersey1* dependencies form project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 2.0.0-alpha-1, 3.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Fix For: 3.0.0 > > Attachments: HBASE-18518-master-01.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18518) Remove jersey1* dependencies form project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115419#comment-16115419 ] Samir Ahmic commented on HBASE-18518: - Here is patch fixing issue with mixed jersey1 and jersey2 classes on REST project. After some digging i have find that we were using jersey1 classes only on few places in hbase code, main source of jersey1 dependency is hadoop and can not be removed completely since we are using it during testing in HBaseTestingUtility#startMiniDFSCluster() and similar functions which requires starting hadoop processes. This patch also resolves https://issues.apache.org/jira/browse/HBASE-18506. And i will resolve it once we finish work here > Remove jersey1* dependencies form project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Attachments: HBASE-18518-master-01.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18518) Remove jersey1* dependencies form project and jersey1* jars from lib dir
[ https://issues.apache.org/jira/browse/HBASE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-18518: Attachment: HBASE-18518-master-01.patch > Remove jersey1* dependencies form project and jersey1* jars from lib dir > > > Key: HBASE-18518 > URL: https://issues.apache.org/jira/browse/HBASE-18518 > Project: HBase > Issue Type: Task > Components: dependencies, pom, REST >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Labels: cleanup > Attachments: HBASE-18518-master-01.patch > > > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and > it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. > With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced > jersey2 to project, and we also have bunch of transitive dependencies > (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 > and jersey2 under same classpath can case runtime issues as it was case with > rest. > This task will have following steps > * Clean code and replace jersey1 constructs with jersey2 versions(there > should not be much of this) > * Add exclusions for transitive jersey1 dependencies in pom.xml > * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18506) java.lang.AbstractMethodError in hbase REST server
[
https://issues.apache.org/jira/browse/HBASE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic reassigned HBASE-18506:
---
Assignee: Samir Ahmic
> java.lang.AbstractMethodError in hbase REST server
> --
>
> Key: HBASE-18506
> URL: https://issues.apache.org/jira/browse/HBASE-18506
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
>
> Just run it this one while testing some scripts. Basically any call to
> service will end up with 500 error. After some checking it looks like we have
> some issues with dependencies incompatibility.
> Here is more details:
> {code}
> Stack trace:
> 2017-08-02 20:46:25,407 WARN [qtp422330142-30] servlet.ServletHandler: Error
> for /status/cluster
> java.lang.AbstractMethodError:
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:119)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:298)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
> at
> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:77)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:518)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> and here are suspects from lib dir
> {code}
> $ grep -r "UriBuilder" .
> Binary file ./javax.ws.rs-api-2.0.1.jar matches
> Binary file ./jersey-common-2.25.1.jar matches
> Binary file ./jersey-core-1.9.jar matches
> {code}
> I have also checked hbase-1.2.6 we have only jersey-core-1.9.jar there
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18518) Remove jersey1* dependencies form project and jersey1* jars from lib dir
Samir Ahmic created HBASE-18518: --- Summary: Remove jersey1* dependencies form project and jersey1* jars from lib dir Key: HBASE-18518 URL: https://issues.apache.org/jira/browse/HBASE-18518 Project: HBase Issue Type: Task Components: dependencies, pom, REST Affects Versions: 2.0.0-alpha-1, 3.0.0 Reporter: Samir Ahmic Assignee: Samir Ahmic Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and it is clear that is caused by mixing jersey1 and jersey2 jars in classpath. With https://issues.apache.org/jira/browse/HBASE-12894 we have introduced jersey2 to project, and we also have bunch of transitive dependencies (mainly from hadoop) on jersey1 which is not happiest situation since jersey1 and jersey2 under same classpath can case runtime issues as it was case with rest. This task will have following steps * Clean code and replace jersey1 constructs with jersey2 versions(there should not be much of this) * Add exclusions for transitive jersey1 dependencies in pom.xml * Add exclusions in hadoop-two-compat.xml to prevent jersey1 jars in lib dir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18506) java.lang.AbstractMethodError in hbase REST server
Samir Ahmic created HBASE-18506:
---
Summary: java.lang.AbstractMethodError in hbase REST server
Key: HBASE-18506
URL: https://issues.apache.org/jira/browse/HBASE-18506
Project: HBase
Issue Type: Bug
Components: REST
Affects Versions: 2.0.0-alpha-1, 3.0.0
Reporter: Samir Ahmic
Priority: Blocker
Just run it this one while testing some scripts. Basically any call to service
will end up with 500 error. After some checking it looks like we have some
issues with dependencies incompatibility.
Here is more details:
{code}
Stack trace:
2017-08-02 20:46:25,407 WARN [qtp422330142-30] servlet.ServletHandler: Error
for /status/cluster
java.lang.AbstractMethodError:
javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:119)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:298)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
at
org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:77)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
{code}
and here are suspects from lib dir
{code}
$ grep -r "UriBuilder" .
Binary file ./javax.ws.rs-api-2.0.1.jar matches
Binary file ./jersey-common-2.25.1.jar matches
Binary file ./jersey-core-1.9.jar matches
{code}
I have also checked hbase-1.2.6 we have only jersey-core-1.9.jar there
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106090#comment-16106090 ] Samir Ahmic commented on HBASE-7386: [~stack] did you maybe have chance to take scripts on test drive ? Any suggestions how we can improve this chief ? > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: stack >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HBASE-7386-bin.patch, HBASE-7386-bin-v2.patch, > HBASE-7386-bin-v3.patch, HBASE-7386-conf.patch, HBASE-7386-conf-v2.patch, > HBASE-7386-conf-v3.patch, HBASE-7386-master-00.patch, > HBASE-7386-master-01.patch, HBASE-7386-src.patch, HBASE-7386-v0.patch, > supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18476) HTable#put should call RS#mutate rather than RS#multi
[ https://issues.apache.org/jira/browse/HBASE-18476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic reassigned HBASE-18476: --- Assignee: (was: Samir Ahmic) > HTable#put should call RS#mutate rather than RS#multi > - > > Key: HBASE-18476 > URL: https://issues.apache.org/jira/browse/HBASE-18476 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai > Fix For: 3.0.0, 1.4.0, 1.5.0, 2.0.0-alpha-2 > > > The HBASE-18374 separates the metric for a single put and batched puts so the > HTable#put shouldn't use the RS#mulit. It messes up the metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18476) HTable#put should call RS#mutate rather than RS#multi
[ https://issues.apache.org/jira/browse/HBASE-18476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic reassigned HBASE-18476: --- Assignee: Chia-Ping Tsai > HTable#put should call RS#mutate rather than RS#multi > - > > Key: HBASE-18476 > URL: https://issues.apache.org/jira/browse/HBASE-18476 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 3.0.0, 1.4.0, 1.5.0, 2.0.0-alpha-2 > > > The HBASE-18374 separates the metric for a single put and batched puts so the > HTable#put shouldn't use the RS#mulit. It messes up the metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18476) HTable#put should call RS#mutate rather than RS#multi
[ https://issues.apache.org/jira/browse/HBASE-18476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic reassigned HBASE-18476: --- Assignee: Samir Ahmic (was: Chia-Ping Tsai) > HTable#put should call RS#mutate rather than RS#multi > - > > Key: HBASE-18476 > URL: https://issues.apache.org/jira/browse/HBASE-18476 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Samir Ahmic > Fix For: 3.0.0, 1.4.0, 1.5.0, 2.0.0-alpha-2 > > > The HBASE-18374 separates the metric for a single put and batched puts so the > HTable#put shouldn't use the RS#mulit. It messes up the metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-7386: --- Attachment: HBASE-7386-master-01.patch Here is new patch addressing shellcheck issues where is possible, can not comply to all suggestions because some of them will break scripts :) Also patch includes some minor fixes for issues detected while testing and adding licence header to config files. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: stack >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HBASE-7386-bin.patch, HBASE-7386-bin-v2.patch, > HBASE-7386-bin-v3.patch, HBASE-7386-conf.patch, HBASE-7386-conf-v2.patch, > HBASE-7386-conf-v3.patch, HBASE-7386-master-00.patch, > HBASE-7386-master-01.patch, HBASE-7386-src.patch, HBASE-7386-v0.patch, > supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096776#comment-16096776 ] Samir Ahmic commented on HBASE-7386: Thanks [~stack]. Why python supervisor? Well we originally started this story around it, and after some time testing it, at least for me, choosing mature and well proven process control system instead of writing custom bash scripts has multiple advantages. To be honest work here extends original issue of just removing stale znodes to creating watchdog over hbase processes and making alternative option for managing cluster but when we started tackling supervisor approach why not offer folks chance to less worry when rs process dies because it will be automatically restarted :) Also python supervisor has set of very cool futures like, auto-restart, event listeners (that may execute arbitrary code based on process state) an so on, and folks may start creating they own listeners for different proposes. Btw i will address shellcheck and pylint issues in next patch. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: stack >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HBASE-7386-bin.patch, HBASE-7386-bin-v2.patch, > HBASE-7386-bin-v3.patch, HBASE-7386-conf.patch, HBASE-7386-conf-v2.patch, > HBASE-7386-conf-v3.patch, HBASE-7386-master-00.patch, HBASE-7386-src.patch, > HBASE-7386-v0.patch, supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-7386: --- Fix Version/s: 3.0.0 Status: Patch Available (was: Open) > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: stack >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HBASE-7386-bin.patch, HBASE-7386-bin-v2.patch, > HBASE-7386-bin-v3.patch, HBASE-7386-conf.patch, HBASE-7386-conf-v2.patch, > HBASE-7386-conf-v3.patch, HBASE-7386-master-00.patch, HBASE-7386-src.patch, > HBASE-7386-v0.patch, supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-7386: --- Attachment: HBASE-7386-master-00.patch Here is first fully operational patch for running hbase processes under python supervisor control. Patch creates new bin/supervisord and conf/supervisord directories first dir contains supporting scripts for managing cluster (start/stop/restart/check) and second one supervisor config files. In order to test this you will need to instal python supervisor (3.3.2) usually with "pip install supervisor==3.3.2". No additional steps are required, configure your hbase as usual and go to /bin/supervisor and run ./start-supervisord-hbase.sh. There is also python script zk_cleaner.py which acts as process event listener in charge to remove mater/rs znode when process is in stoping or exit state. This is first version of patch and whole idea will need more testing and code polishing, all comments and suggestions are welcome. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: stack >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HBASE-7386-bin.patch, HBASE-7386-bin-v2.patch, > HBASE-7386-bin-v3.patch, HBASE-7386-conf.patch, HBASE-7386-conf-v2.patch, > HBASE-7386-conf-v3.patch, HBASE-7386-master-00.patch, HBASE-7386-src.patch, > HBASE-7386-v0.patch, supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18392) Add default value of ----movetimeout to rolling-restart.sh
[
https://issues.apache.org/jira/browse/HBASE-18392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091873#comment-16091873
]
Samir Ahmic commented on HBASE-18392:
-
Thank you [~tedyu].
> Add default value of movetimeout to rolling-restart.sh
> --
>
> Key: HBASE-18392
> URL: https://issues.apache.org/jira/browse/HBASE-18392
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 3.0.0
>
> Attachments: HBASE-18392-master-001.patch
>
>
> We are calling graceful_stop.sh in rolling-restart.sh with following line
> {code}
> "$bin"/graceful_stop.sh --config ${HBASE_CONF_DIR} --restart --reload -nob
> --maxthreads \
> ${RR_MAXTHREADS} ${RR_NOACK} --movetimeout ${RR_MOVE_TIMEOUT}
> $hostname
> {code}
> and if we not specified --movetimeout option while calling rolling-restart.sh
> --graceful script will not work. My propose is to add default value for this
> parameter same way we are doing in graceful_stop.sh
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18392) Add default value of ----movetimeout to rolling-restart.sh
[
https://issues.apache.org/jira/browse/HBASE-18392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091402#comment-16091402
]
Samir Ahmic commented on HBASE-18392:
-
[~tedyu] mind committing this i need it in place for
https://issues.apache.org/jira/browse/HBASE-7386 ?
> Add default value of movetimeout to rolling-restart.sh
> --
>
> Key: HBASE-18392
> URL: https://issues.apache.org/jira/browse/HBASE-18392
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 3.0.0
>
> Attachments: HBASE-18392-master-001.patch
>
>
> We are calling graceful_stop.sh in rolling-restart.sh with following line
> {code}
> "$bin"/graceful_stop.sh --config ${HBASE_CONF_DIR} --restart --reload -nob
> --maxthreads \
> ${RR_MAXTHREADS} ${RR_NOACK} --movetimeout ${RR_MOVE_TIMEOUT}
> $hostname
> {code}
> and if we not specified --movetimeout option while calling rolling-restart.sh
> --graceful script will not work. My propose is to add default value for this
> parameter same way we are doing in graceful_stop.sh
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18393) hbase shell non-interactive broken
[
https://issues.apache.org/jira/browse/HBASE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091238#comment-16091238
]
Samir Ahmic commented on HBASE-18393:
-
[~mdrob] fix for hirb.rb looks fine and regarding test it may fail also because
of "hbase" scripts itself for example if JAVA_HOME is not found in env where
test is running, but i think it is OK since this way we also test "hbase"
script for potential issues, we just need to ensure env is setup correctly when
running test.
> hbase shell non-interactive broken
>
>
> Key: HBASE-18393
> URL: https://issues.apache.org/jira/browse/HBASE-18393
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Mike Drob
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18393.patch, HBASE-18393.v2.patch
>
>
> Here is error for command line:
> {code}
> $ echo "list" | ./hbase shell -n
> 2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ERROR NoMethodError: undefined method `encoding' for #
> Did you mean? set_encoding
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18392) Add default value of ----movetimeout to rolling-restart.sh
[
https://issues.apache.org/jira/browse/HBASE-18392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091094#comment-16091094
]
Samir Ahmic commented on HBASE-18392:
-
[~tedyu] I have inherited that value from graceful_stop.sh it is MAX_INT.
> Add default value of movetimeout to rolling-restart.sh
> --
>
> Key: HBASE-18392
> URL: https://issues.apache.org/jira/browse/HBASE-18392
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 3.0.0
>
> Attachments: HBASE-18392-master-001.patch
>
>
> We are calling graceful_stop.sh in rolling-restart.sh with following line
> {code}
> "$bin"/graceful_stop.sh --config ${HBASE_CONF_DIR} --restart --reload -nob
> --maxthreads \
> ${RR_MAXTHREADS} ${RR_NOACK} --movetimeout ${RR_MOVE_TIMEOUT}
> $hostname
> {code}
> and if we not specified --movetimeout option while calling rolling-restart.sh
> --graceful script will not work. My propose is to add default value for this
> parameter same way we are doing in graceful_stop.sh
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18392) Add default value of ----movetimeout to rolling-restart.sh
[
https://issues.apache.org/jira/browse/HBASE-18392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-18392:
Status: Patch Available (was: Open)
> Add default value of movetimeout to rolling-restart.sh
> --
>
> Key: HBASE-18392
> URL: https://issues.apache.org/jira/browse/HBASE-18392
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 3.0.0
>
> Attachments: HBASE-18392-master-001.patch
>
>
> We are calling graceful_stop.sh in rolling-restart.sh with following line
> {code}
> "$bin"/graceful_stop.sh --config ${HBASE_CONF_DIR} --restart --reload -nob
> --maxthreads \
> ${RR_MAXTHREADS} ${RR_NOACK} --movetimeout ${RR_MOVE_TIMEOUT}
> $hostname
> {code}
> and if we not specified --movetimeout option while calling rolling-restart.sh
> --graceful script will not work. My propose is to add default value for this
> parameter same way we are doing in graceful_stop.sh
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18392) Add default value of ----movetimeout to rolling-restart.sh
[
https://issues.apache.org/jira/browse/HBASE-18392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-18392:
Attachment: HBASE-18392-master-001.patch
Here is very simple patch.
> Add default value of movetimeout to rolling-restart.sh
> --
>
> Key: HBASE-18392
> URL: https://issues.apache.org/jira/browse/HBASE-18392
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 3.0.0
>
> Attachments: HBASE-18392-master-001.patch
>
>
> We are calling graceful_stop.sh in rolling-restart.sh with following line
> {code}
> "$bin"/graceful_stop.sh --config ${HBASE_CONF_DIR} --restart --reload -nob
> --maxthreads \
> ${RR_MAXTHREADS} ${RR_NOACK} --movetimeout ${RR_MOVE_TIMEOUT}
> $hostname
> {code}
> and if we not specified --movetimeout option while calling rolling-restart.sh
> --graceful script will not work. My propose is to add default value for this
> parameter same way we are doing in graceful_stop.sh
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18393) hbase shell non-interactive broken
[
https://issues.apache.org/jira/browse/HBASE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090439#comment-16090439
]
Samir Ahmic edited comment on HBASE-18393 at 7/17/17 7:49 PM:
--
Thanks for testing [~mdrob]. Then let us stick to RubyLex. Can i reassign issue
to
you to submit patch ?
was (Author: asamir):
Thanks for testing [~mdrob]. Then let us stick to RubyLex. Can i reassign issue
you to submit patch ?
> hbase shell non-interactive broken
>
>
> Key: HBASE-18393
> URL: https://issues.apache.org/jira/browse/HBASE-18393
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> Here is error for command line:
> {code}
> $ echo "list" | ./hbase shell -n
> 2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ERROR NoMethodError: undefined method `encoding' for #
> Did you mean? set_encoding
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18393) hbase shell non-interactive broken
[
https://issues.apache.org/jira/browse/HBASE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090439#comment-16090439
]
Samir Ahmic commented on HBASE-18393:
-
Thanks for testing [~mdrob]. Then let us stick to RubyLex. Can i reassign issue
you to submit patch ?
> hbase shell non-interactive broken
>
>
> Key: HBASE-18393
> URL: https://issues.apache.org/jira/browse/HBASE-18393
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> Here is error for command line:
> {code}
> $ echo "list" | ./hbase shell -n
> 2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ERROR NoMethodError: undefined method `encoding' for #
> Did you mean? set_encoding
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18393) hbase shell non-interactive broken
[
https://issues.apache.org/jira/browse/HBASE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090414#comment-16090414
]
Samir Ahmic edited comment on HBASE-18393 at 7/17/17 7:25 PM:
--
[~mdrob] I have tested ARGF fix with:
{code}
echo "scan" \ "cluster_test," \ {LIMIT=>10} | ./hbase shell -n
{code}
and line continuation is working. If we want minimal fix ARGF fix even reduces
code :) and removes deps to RubyLex. Your approach is also acceptable if you
think is more robust we also can go that way.
was (Author: asamir):
[~mdrob] I have tested ARGF fix with:
{code}
echo "scan" \ "cluster_test," \ {LIMIT=>10} | ./hbase shell -n
{code}
and line continuation is working. If we want minimal fix ARGF fix even reduces
code :) and removes deps to RubyLex. Your approach is also acceptable if you
think it more robust we also can go that way.
> hbase shell non-interactive broken
>
>
> Key: HBASE-18393
> URL: https://issues.apache.org/jira/browse/HBASE-18393
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> Here is error for command line:
> {code}
> $ echo "list" | ./hbase shell -n
> 2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ERROR NoMethodError: undefined method `encoding' for #
> Did you mean? set_encoding
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18393) hbase shell non-interactive broken
[
https://issues.apache.org/jira/browse/HBASE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090414#comment-16090414
]
Samir Ahmic commented on HBASE-18393:
-
[~mdrob] I have tested ARGF fix with:
{code}
echo "scan" \ "cluster_test," \ {LIMIT=>10} | ./hbase shell -n
{code}
and line continuation is working. If we want minimal fix ARGF fix even reduces
code :) and removes deps to RubyLex. Your approach is also acceptable if you
think it more robust we also can go that way.
> hbase shell non-interactive broken
>
>
> Key: HBASE-18393
> URL: https://issues.apache.org/jira/browse/HBASE-18393
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> Here is error for command line:
> {code}
> $ echo "list" | ./hbase shell -n
> 2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ERROR NoMethodError: undefined method `encoding' for #
> Did you mean? set_encoding
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18393) hbase shell non-interactive broken
[
https://issues.apache.org/jira/browse/HBASE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090303#comment-16090303
]
Samir Ahmic edited comment on HBASE-18393 at 7/17/17 6:38 PM:
--
[~mdrob] looks like issue is on relation RubyLex an ruby io, maybe we can fix
issue by not using RubyLex and replace it with something else. For me
non-interactive started working with this change:
{code}
diff --git a/bin/hirb.rb b/bin/hirb.rb
index d0295d6..94391cc 100644
--- a/bin/hirb.rb
+++ b/bin/hirb.rb
@@ -211,12 +211,9 @@ else
# in order to maintain compatibility with previous behavior where
# a user could pass in script2run and then still pipe commands on
# stdin.
-require "irb/ruby-lex"
require "irb/workspace"
workspace = IRB::WorkSpace.new(binding())
-scanner = RubyLex.new
-scanner.set_input(STDIN)
-scanner.each_top_level_statement do |statement, linenum|
+ ARGF.each_with_index do |statement, linenum|
puts(workspace.evaluate(nil, statement, 'stdin', linenum))
end
# XXX We're catching Exception on purpose, because we want to include
{code}
if you think this is acceptable solution i can create patch ?
was (Author: asamir):
[~mdrob] looks like issue is on relation RubyLex an ruby io, maybe we can fix
issue by not using RubyLex and replace it with something else. For me
non-interactive started working with this change:
{code}
diff --git a/bin/hirb.rb b/bin/hirb.rb
index d0295d6..94391cc 100644
--- a/bin/hirb.rb
+++ b/bin/hirb.rb
@@ -211,12 +211,9 @@ else
# in order to maintain compatibility with previous behavior where
# a user could pass in script2run and then still pipe commands on
# stdin.
-require "irb/ruby-lex"
require "irb/workspace"
workspace = IRB::WorkSpace.new(binding())
-scanner = RubyLex.new
-scanner.set_input(STDIN)
-scanner.each_top_level_statement do |statement, linenum|
+ ARGF.each_with_index do |statement, linenum|
puts(workspace.evaluate(nil, statement, 'stdin', linenum))
end
# XXX We're catching Exception on purpose, because we want to include
{code}
if you think this is acceptable solution a can create patch ?
> hbase shell non-interactive broken
>
>
> Key: HBASE-18393
> URL: https://issues.apache.org/jira/browse/HBASE-18393
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> Here is error for command line:
> {code}
> $ echo "list" | ./hbase shell -n
> 2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ERROR NoMethodError: undefined method `encoding' for #
> Did you mean? set_encoding
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18393) hbase shell non-interactive broken
[
https://issues.apache.org/jira/browse/HBASE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090303#comment-16090303
]
Samir Ahmic commented on HBASE-18393:
-
[~mdrob] looks like issue is on relation RubyLex an ruby io, maybe we can fix
issue by not using RubyLex and replace it with something else. For me
non-interactive started working with this change:
{code}
diff --git a/bin/hirb.rb b/bin/hirb.rb
index d0295d6..94391cc 100644
--- a/bin/hirb.rb
+++ b/bin/hirb.rb
@@ -211,12 +211,9 @@ else
# in order to maintain compatibility with previous behavior where
# a user could pass in script2run and then still pipe commands on
# stdin.
-require "irb/ruby-lex"
require "irb/workspace"
workspace = IRB::WorkSpace.new(binding())
-scanner = RubyLex.new
-scanner.set_input(STDIN)
-scanner.each_top_level_statement do |statement, linenum|
+ ARGF.each_with_index do |statement, linenum|
puts(workspace.evaluate(nil, statement, 'stdin', linenum))
end
# XXX We're catching Exception on purpose, because we want to include
{code}
if you think this is acceptable solution a can create patch ?
> hbase shell non-interactive broken
>
>
> Key: HBASE-18393
> URL: https://issues.apache.org/jira/browse/HBASE-18393
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> Here is error for command line:
> {code}
> $ echo "list" | ./hbase shell -n
> 2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ERROR NoMethodError: undefined method `encoding' for #
> Did you mean? set_encoding
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18393) hbase shell non-interactive broken
[
https://issues.apache.org/jira/browse/HBASE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089555#comment-16089555
]
Samir Ahmic commented on HBASE-18393:
-
[~tedyu] i'm checking issue at moment if i find solution i will submit patch,
on first glance looks like some ruby modules internal issue, because it is
failing in hirb.rb on :
{code}
scanner.each_top_level_statement
{code}
If some of ruby experts can pinpoint exact issue i can reassign issue.
> hbase shell non-interactive broken
>
>
> Key: HBASE-18393
> URL: https://issues.apache.org/jira/browse/HBASE-18393
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 3.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>
> Here is error for command line:
> {code}
> $ echo "list" | ./hbase shell -n
> 2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ERROR NoMethodError: undefined method `encoding' for #
> Did you mean? set_encoding
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18393) hbase shell non-interactive broken
Samir Ahmic created HBASE-18393:
---
Summary: hbase shell non-interactive broken
Key: HBASE-18393
URL: https://issues.apache.org/jira/browse/HBASE-18393
Project: HBase
Issue Type: Bug
Components: scripts, shell
Affects Versions: 3.0.0
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Here is error for command line:
{code}
$ echo "list" | ./hbase shell -n
2017-07-17 08:01:09,442 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
ERROR NoMethodError: undefined method `encoding' for #
Did you mean? set_encoding
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18392) Add default value of ----movetimeout to rolling-restart.sh
Samir Ahmic created HBASE-18392:
---
Summary: Add default value of movetimeout to rolling-restart.sh
Key: HBASE-18392
URL: https://issues.apache.org/jira/browse/HBASE-18392
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Fix For: 3.0.0
We are calling graceful_stop.sh in rolling-restart.sh with following line
{code}
"$bin"/graceful_stop.sh --config ${HBASE_CONF_DIR} --restart --reload -nob
--maxthreads \
${RR_MAXTHREADS} ${RR_NOACK} --movetimeout ${RR_MOVE_TIMEOUT}
$hostname
{code}
and if we not specified --movetimeout option while calling rolling-restart.sh
--graceful script will not work. My propose is to add default value for this
parameter same way we are doing in graceful_stop.sh
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084730#comment-16084730 ] Samir Ahmic commented on HBASE-7386: [~stack] i have done some testing with last patches against master branch and good news is that most of code(with small changes) and functionality works fine. So original idea to improve MTTR by removing stale master and rs znodes plus watchdog which will restart process in case of unexpected failure is still valid. My original scripts here are written with idea to be optional route in managing hbase processes using supervisor, and that approach opens couple of questions which i would like to discuss: # Amount of code added and options to reduce it (i will anyway try to reduce it to minimum) probably some code can be integrated in exiting scripts to avoid copying # Where are we going to add new scripts supervisord folder inside bin dir was may original idea and same thing goes for config files supervisord folder in conf dir # Testing: i will cover supervisor 3.3.2 version(last stable) and some older version that are installed trough system packet manages # And finally would it be better to implement our own Java supervisor which would do similar thing as python implementation Based on what we decide i will continue work here, if we go with python supervisor i can have patch ready for testing in couple of days. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: stack >Priority: Blocker > Attachments: HBASE-7386-bin.patch, HBASE-7386-bin-v2.patch, > HBASE-7386-bin-v3.patch, HBASE-7386-conf.patch, HBASE-7386-conf-v2.patch, > HBASE-7386-conf-v3.patch, HBASE-7386-src.patch, HBASE-7386-v0.patch, > supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18310) LoadTestTool unable to write data
[
https://issues.apache.org/jira/browse/HBASE-18310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074015#comment-16074015
]
Samir Ahmic commented on HBASE-18310:
-
Thanks for review [~tedyu]. Build failure looks unrelated to patch.
> LoadTestTool unable to write data
> -
>
> Key: HBASE-18310
> URL: https://issues.apache.org/jira/browse/HBASE-18310
> Project: HBase
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 3.0.0
>
> Attachments: HBASE-18310-master-01.patch
>
>
> I was trying to write some data to cluster with hbase ltt but i have this one:
> {code}
> $ hbase ltt -write 3:30:10 -num_keys 100
> 2017-07-03 14:40:47,372 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> java.lang.UnsupportedOperationException: HColumnDescriptor is read-only
> at
> org.apache.hadoop.hbase.client.ImmutableHColumnDescriptor.getDelegateeForModification(ImmutableHColumnDescriptor.java:44)
> at
> org.apache.hadoop.hbase.HColumnDescriptor.setBloomFilterType(HColumnDescriptor.java:475)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.applyColumnFamilyOptions(LoadTestTool.java:288)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.initTestTable(LoadTestTool.java:557)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.loadTable(LoadTestTool.java:584)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:565)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:262)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:831)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18310) LoadTestTool unable to write data
[
https://issues.apache.org/jira/browse/HBASE-18310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-18310:
Fix Version/s: 3.0.0
Status: Patch Available (was: Open)
> LoadTestTool unable to write data
> -
>
> Key: HBASE-18310
> URL: https://issues.apache.org/jira/browse/HBASE-18310
> Project: HBase
> Issue Type: Bug
> Components: util
>Affects Versions: 2.0.0-alpha-1, 3.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 3.0.0
>
> Attachments: HBASE-18310-master-01.patch
>
>
> I was trying to write some data to cluster with hbase ltt but i have this one:
> {code}
> $ hbase ltt -write 3:30:10 -num_keys 100
> 2017-07-03 14:40:47,372 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> java.lang.UnsupportedOperationException: HColumnDescriptor is read-only
> at
> org.apache.hadoop.hbase.client.ImmutableHColumnDescriptor.getDelegateeForModification(ImmutableHColumnDescriptor.java:44)
> at
> org.apache.hadoop.hbase.HColumnDescriptor.setBloomFilterType(HColumnDescriptor.java:475)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.applyColumnFamilyOptions(LoadTestTool.java:288)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.initTestTable(LoadTestTool.java:557)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.loadTable(LoadTestTool.java:584)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:565)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:262)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:831)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18310) LoadTestTool unable to write data
[
https://issues.apache.org/jira/browse/HBASE-18310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-18310:
Attachment: HBASE-18310-master-01.patch
Here is simple patch replacing HTableDescriptor which is removed with
TableDescriptor.
> LoadTestTool unable to write data
> -
>
> Key: HBASE-18310
> URL: https://issues.apache.org/jira/browse/HBASE-18310
> Project: HBase
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 2.0.0-alpha-1
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 3.0.0
>
> Attachments: HBASE-18310-master-01.patch
>
>
> I was trying to write some data to cluster with hbase ltt but i have this one:
> {code}
> $ hbase ltt -write 3:30:10 -num_keys 100
> 2017-07-03 14:40:47,372 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> java.lang.UnsupportedOperationException: HColumnDescriptor is read-only
> at
> org.apache.hadoop.hbase.client.ImmutableHColumnDescriptor.getDelegateeForModification(ImmutableHColumnDescriptor.java:44)
> at
> org.apache.hadoop.hbase.HColumnDescriptor.setBloomFilterType(HColumnDescriptor.java:475)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.applyColumnFamilyOptions(LoadTestTool.java:288)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.initTestTable(LoadTestTool.java:557)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.loadTable(LoadTestTool.java:584)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:565)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:262)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:831)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072789#comment-16072789 ] Samir Ahmic commented on HBASE-7386: [~stack] anything improving MTTR and cluster operability deserves decent retry :), i will take look at work done here and check if same or similar concept can be applied at current hbase clusters. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: stack >Priority: Blocker > Attachments: HBASE-7386-bin.patch, HBASE-7386-bin-v2.patch, > HBASE-7386-bin-v3.patch, HBASE-7386-conf.patch, HBASE-7386-conf-v2.patch, > HBASE-7386-conf-v3.patch, HBASE-7386-src.patch, HBASE-7386-v0.patch, > supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072769#comment-16072769 ] Samir Ahmic commented on HBASE-15943: - Failures look unrelated to patch. [~stack] i have also tested this patch against branch-2 and applies correctly, ping me if any additional work is required. > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: Operability, UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Attachments: HBASE-15943-master-03.patch, HBASE-15943_v0.patch, > HBASE-15943_v1.patch, processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18310) LoadTestTool unable to write data
Samir Ahmic created HBASE-18310:
---
Summary: LoadTestTool unable to write data
Key: HBASE-18310
URL: https://issues.apache.org/jira/browse/HBASE-18310
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 2.0.0-alpha-1, 3.0.0
Reporter: Samir Ahmic
Assignee: Samir Ahmic
I was trying to write some data to cluster with hbase ltt but i have this one:
{code}
$ hbase ltt -write 3:30:10 -num_keys 100
2017-07-03 14:40:47,372 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.UnsupportedOperationException: HColumnDescriptor is read-only
at
org.apache.hadoop.hbase.client.ImmutableHColumnDescriptor.getDelegateeForModification(ImmutableHColumnDescriptor.java:44)
at
org.apache.hadoop.hbase.HColumnDescriptor.setBloomFilterType(HColumnDescriptor.java:475)
at
org.apache.hadoop.hbase.util.LoadTestTool.applyColumnFamilyOptions(LoadTestTool.java:288)
at
org.apache.hadoop.hbase.util.LoadTestTool.initTestTable(LoadTestTool.java:557)
at
org.apache.hadoop.hbase.util.LoadTestTool.loadTable(LoadTestTool.java:584)
at
org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:565)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:262)
at org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:831)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15943: Attachment: HBASE-15943-master-03.patch New patch for master branch. > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: Operability, UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Attachments: HBASE-15943-master-03.patch, HBASE-15943_v0.patch, > HBASE-15943_v1.patch, processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070368#comment-16070368 ] Samir Ahmic commented on HBASE-15943: - Sure thing i will take a look and check how things looks now and submit new patch against master in few days, also i can add patch agains branch-2. > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: Operability, UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Attachments: HBASE-15943_v0.patch, HBASE-15943_v1.patch, > processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679335#comment-15679335 ] Samir Ahmic commented on HBASE-15943: - [~stack] it is correct you will get 404 but only in case when you try to display RS process metrics on regionserver that is collocated with master process. For rest of regionservers page works. Now i guess future of regionserver that lives in master process is still not decided. We can have simple fix by adding same page processRS.jsp to hbase-server/src/main/resources/hbase-webapps/master ? > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15943_v0.patch, HBASE-15943_v1.patch, > processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666580#comment-15666580 ] Samir Ahmic commented on HBASE-15943: - Thanks for checking [~stack], patch is little bit old probably something changed in meantime. I will reiterate trough changes an post new patch with fix in couple of days. > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15943_v0.patch, HBASE-15943_v1.patch, > processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16317) revert all ESAPI changes
[ https://issues.apache.org/jira/browse/HBASE-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404131#comment-15404131 ] Samir Ahmic commented on HBASE-16317: - Sure thing i will check list and re-test UI to see is all there regarding JIRA-s i was working on. > revert all ESAPI changes > > > Key: HBASE-16317 > URL: https://issues.apache.org/jira/browse/HBASE-16317 > Project: HBase > Issue Type: Sub-task > Components: dependencies, security >Reporter: Sean Busbey >Assignee: Nick Dimiduk >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3 > > Attachments: HBASE-16317.v00.branch-1.1.patch, > HBASE-16317.v00.branch-1.2.patch, HBASE-16317.v00.branch-1.3.patch, > HBASE-16317.v00.branch-1.patch, HBASE-16317.v00.master.patch > > > to unblock releases, we'll start cleaning up the category-x problem by > reverting all the ESAPI changes. > we should try to include a release note with what this means we'll be > vulnerable to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16317) revert all ESAPI changes
[ https://issues.apache.org/jira/browse/HBASE-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403406#comment-15403406 ] Samir Ahmic commented on HBASE-16317: - [~ndimiduk], [~busbey] feel free to revert and them we can reopen this issues and find solution without ESAPI deps. This are mainly UI minor fixes that we can live with. > revert all ESAPI changes > > > Key: HBASE-16317 > URL: https://issues.apache.org/jira/browse/HBASE-16317 > Project: HBase > Issue Type: Sub-task > Components: dependencies, security >Reporter: Sean Busbey >Assignee: Nick Dimiduk >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3 > > Attachments: HBASE-16317.v00.branch-1.3.patch, > HBASE-16317.v00.branch-1.patch, HBASE-16317.v00.master.patch > > > to unblock releases, we'll start cleaning up the category-x problem by > reverting all the ESAPI changes. > we should try to include a release note with what this means we'll be > vulnerable to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16078) Create java cli tool for managing balancer states for scripts usage
[ https://issues.apache.org/jira/browse/HBASE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-16078: Resolution: Won't Fix Status: Resolved (was: Patch Available) > Create java cli tool for managing balancer states for scripts usage > --- > > Key: HBASE-16078 > URL: https://issues.apache.org/jira/browse/HBASE-16078 > Project: HBase > Issue Type: Improvement > Components: scripts, util >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-16078_v1.patch, HBASE-16078_v2.patch > > > This ticket is result of discussion in > [HBASE16044|https://issues.apache.org/jira/browse/HBASE-16044] to avoid > "hbase shell" output parsing hacks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16044) Fix 'hbase shell' output parsing in graceful_stop.sh
[
https://issues.apache.org/jira/browse/HBASE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-16044:
Assignee: Appy (was: Samir Ahmic)
> Fix 'hbase shell' output parsing in graceful_stop.sh
>
>
> Key: HBASE-16044
> URL: https://issues.apache.org/jira/browse/HBASE-16044
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Appy
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16044.master.001.patch
>
>
> In some of our bash scripts we are piping command in hbase shell and then
> parsing response to define variables. Since 'hbase shell' output format is
> changed we are picking wrong values from output Here is example form
> gracful_stop.sh:
> {code}
> HBASE_BALANCER_STATE=$(echo 'balance_switch false' | "$bin"/hbase --config
> "${HBASE_CONF_DIR}" shell | tail -3 | head -1)
> {code}
> this will return "balance_switch true" instead of previous balancer state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16044) Fix 'hbase shell' output parsing in graceful_stop.sh
[
https://issues.apache.org/jira/browse/HBASE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358563#comment-15358563
]
Samir Ahmic commented on HBASE-16044:
-
+1. [~appy] mind putting same fix in rolling-restart.sh ?
> Fix 'hbase shell' output parsing in graceful_stop.sh
>
>
> Key: HBASE-16044
> URL: https://issues.apache.org/jira/browse/HBASE-16044
> Project: HBase
> Issue Type: Bug
> Components: scripts, shell
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16044.master.001.patch
>
>
> In some of our bash scripts we are piping command in hbase shell and then
> parsing response to define variables. Since 'hbase shell' output format is
> changed we are picking wrong values from output Here is example form
> gracful_stop.sh:
> {code}
> HBASE_BALANCER_STATE=$(echo 'balance_switch false' | "$bin"/hbase --config
> "${HBASE_CONF_DIR}" shell | tail -3 | head -1)
> {code}
> this will return "balance_switch true" instead of previous balancer state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16078) Create java cli tool for managing balancer states for scripts usage
[ https://issues.apache.org/jira/browse/HBASE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352558#comment-15352558 ] Samir Ahmic commented on HBASE-16078: - Eliminating fromatter in non-interactive mode will probably do the job. I'm +1 for any solution that will define better compat guarantees and avoid breaking stuff silently like we have case with bash scripts using hbase shell commands. > Create java cli tool for managing balancer states for scripts usage > --- > > Key: HBASE-16078 > URL: https://issues.apache.org/jira/browse/HBASE-16078 > Project: HBase > Issue Type: Improvement > Components: scripts, util >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-16078_v1.patch, HBASE-16078_v2.patch > > > This ticket is result of discussion in > [HBASE16044|https://issues.apache.org/jira/browse/HBASE-16044] to avoid > "hbase shell" output parsing hacks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16078) Create java cli tool for managing balancer states for scripts usage
[ https://issues.apache.org/jira/browse/HBASE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-16078: Attachment: HBASE-16078_v2.patch Here is patch v2 adding missing licence in test file. > Create java cli tool for managing balancer states for scripts usage > --- > > Key: HBASE-16078 > URL: https://issues.apache.org/jira/browse/HBASE-16078 > Project: HBase > Issue Type: Improvement > Components: scripts, util >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-16078_v1.patch, HBASE-16078_v2.patch > > > This ticket is result of discussion in > [HBASE16044|https://issues.apache.org/jira/browse/HBASE-16044] to avoid > "hbase shell" output parsing hacks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16078) Create java cli tool for managing balancer states for scripts usage
[ https://issues.apache.org/jira/browse/HBASE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15346066#comment-15346066 ] Samir Ahmic commented on HBASE-16078: - [~tedyu], [~appy] thanks for review. I will include licence in next patch. Here is my opinion regarding usefulness of this and similar tools: - execution time is 3-4 seconds faster then piping command to shell - it is simple single propose tool intend for usage in bash scripts which needs to manage balancer, and i believe is useful for critical operations scripts like graceful_stop.sh and rolling-restart.sh where we need to do things precise and simple as possible - it is independent and probably will be less touched then hbase shell code and that way will decrease possibility of scripts get broken (i should probably put note in code saying that it is intended for internal use only, used by graceful_stop.sh and rolling-restart.sh and that changes should be propagated to scripts) - if we replace our scripts with something else and we find this tool unused it can be easily removed > Create java cli tool for managing balancer states for scripts usage > --- > > Key: HBASE-16078 > URL: https://issues.apache.org/jira/browse/HBASE-16078 > Project: HBase > Issue Type: Improvement > Components: scripts, util >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-16078_v1.patch > > > This ticket is result of discussion in > [HBASE16044|https://issues.apache.org/jira/browse/HBASE-16044] to avoid > "hbase shell" output parsing hacks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16044) Fix 'hbase shell' output parsing in bash scripts
[
https://issues.apache.org/jira/browse/HBASE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344185#comment-15344185
]
Samir Ahmic commented on HBASE-16044:
-
Here is ticket for adding java based tool
[HBASE-16078|https://issues.apache.org/jira/browse/HBASE-16078]
> Fix 'hbase shell' output parsing in bash scripts
>
>
> Key: HBASE-16044
> URL: https://issues.apache.org/jira/browse/HBASE-16044
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
>
> In some of our bash scripts we are piping command in hbase shell and then
> parsing response to define variables. Since 'hbase shell' output format is
> changed we are picking wrong values from output Here is example form
> gracful_stop.sh:
> {code}
> HBASE_BALANCER_STATE=$(echo 'balance_switch false' | "$bin"/hbase --config
> "${HBASE_CONF_DIR}" shell | tail -3 | head -1)
> {code}
> this will return "balance_switch true" instead of previous balancer state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16078) Create java cli tool for managing blancer states for scripts usage.
[ https://issues.apache.org/jira/browse/HBASE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-16078: Status: Patch Available (was: Open) > Create java cli tool for managing blancer states for scripts usage. > --- > > Key: HBASE-16078 > URL: https://issues.apache.org/jira/browse/HBASE-16078 > Project: HBase > Issue Type: Improvement > Components: scripts, util >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-16078_v1.patch > > > This ticket is result of discussion in > [HBASE16044|https://issues.apache.org/jira/browse/HBASE-16044] to avoid > "hbase shell" output parsing hacks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16078) Create java cli tool for managing blancer states for scripts usage.
[ https://issues.apache.org/jira/browse/HBASE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-16078: Attachment: HBASE-16078_v1.patch Here is patch adding cli tool for managing balancer and adding changes in scripts to use it instead of hbase shell. > Create java cli tool for managing blancer states for scripts usage. > --- > > Key: HBASE-16078 > URL: https://issues.apache.org/jira/browse/HBASE-16078 > Project: HBase > Issue Type: Improvement > Components: scripts, util >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-16078_v1.patch > > > This ticket is result of discussion in > [HBASE16044|https://issues.apache.org/jira/browse/HBASE-16044] to avoid > "hbase shell" output parsing hacks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16078) Create java cli tool for managing blancer states for scripts usage.
Samir Ahmic created HBASE-16078: --- Summary: Create java cli tool for managing blancer states for scripts usage. Key: HBASE-16078 URL: https://issues.apache.org/jira/browse/HBASE-16078 Project: HBase Issue Type: Improvement Components: scripts, util Affects Versions: 2.0.0 Reporter: Samir Ahmic Assignee: Samir Ahmic Fix For: 2.0.0 This ticket is result of discussion in [HBASE16044|https://issues.apache.org/jira/browse/HBASE-16044] to avoid "hbase shell" output parsing hacks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16044) Fix 'hbase shell' output parsing in bash scripts
[
https://issues.apache.org/jira/browse/HBASE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341712#comment-15341712
]
Samir Ahmic commented on HBASE-16044:
-
If nobody objects i will create ticket for creating simple java command line
tool that will manage balancer states to avoid this type of bugs in future ? I
have checked our scripts and we are parsing 'hbase shell' output only in
graceful_stop.sh and rolling-restart.sh in order to manage balancer.
> Fix 'hbase shell' output parsing in bash scripts
>
>
> Key: HBASE-16044
> URL: https://issues.apache.org/jira/browse/HBASE-16044
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
>
> In some of our bash scripts we are piping command in hbase shell and then
> parsing response to define variables. Since 'hbase shell' output format is
> changed we are picking wrong values from output Here is example form
> gracful_stop.sh:
> {code}
> HBASE_BALANCER_STATE=$(echo 'balance_switch false' | "$bin"/hbase --config
> "${HBASE_CONF_DIR}" shell | tail -3 | head -1)
> {code}
> this will return "balance_switch true" instead of previous balancer state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15722) Size. WAL Files (bytes) in regionserver status page displays negative values
[
https://issues.apache.org/jira/browse/HBASE-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335982#comment-15335982
]
Samir Ahmic commented on HBASE-15722:
-
[~enis], [~tedyu] is there anything else that needs to be done regarding this
issue ?
> Size. WAL Files (bytes) in regionserver status page displays negative values
>
>
> Key: HBASE-15722
> URL: https://issues.apache.org/jira/browse/HBASE-15722
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Minor
> Attachments: HBASE-15722_v0.patch, HBASE-15722_v1.patch,
> HBASE-15722_v2.patch, WALs.png
>
>
> Here is the line from ServerMetricTmpl.jamon
> {code}
> TraditionalBinaryPrefix.long2String(mWrap.getWALFileSize(), "B", 1)
> {code}
> I will change this to StringUtils.humanSize()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15722) Size. WAL Files (bytes) in regionserver status page displays negative values
[
https://issues.apache.org/jira/browse/HBASE-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280043#comment-15280043
]
Samir Ahmic edited comment on HBASE-15722 at 6/17/16 12:38 PM:
---
Failed test is {{TestRegionServerMetrics#testMobMetrics()}}: i do not think
failed test is related with this patch.
was (Author: asamir):
Filed test is {{TestRegionServerMetrics#testMobMetrics()}} i do not think
failed test is related with this patch.
> Size. WAL Files (bytes) in regionserver status page displays negative values
>
>
> Key: HBASE-15722
> URL: https://issues.apache.org/jira/browse/HBASE-15722
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Minor
> Attachments: HBASE-15722_v0.patch, HBASE-15722_v1.patch,
> HBASE-15722_v2.patch, WALs.png
>
>
> Here is the line from ServerMetricTmpl.jamon
> {code}
> TraditionalBinaryPrefix.long2String(mWrap.getWALFileSize(), "B", 1)
> {code}
> I will change this to StringUtils.humanSize()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16044) Fix 'hbase shell' output parsing in bash scripts
[
https://issues.apache.org/jira/browse/HBASE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334203#comment-15334203
]
Samir Ahmic edited comment on HBASE-16044 at 6/16/16 5:20 PM:
--
I think it was [HBASE-15849 |
https://issues.apache.org/jira/browse/HBASE-15849] by adding this line:
{code}
+formatter.output_str("Took %.4f seconds" % [@end_time - @start_time])
{code}
It is simple fix in graceful_stop.sh and rolling-restart.sh just "tail -2"
instead fo "tail -3".
was (Author: asamir):
I think it was [ HBASE-15849 |
https://issues.apache.org/jira/browse/HBASE-15849] by adding this line:
{code}
+formatter.output_str("Took %.4f seconds" % [@end_time - @start_time])
{code}
It is simple fix in graceful_stop.sh and rolling-restart.sh just "tail -2"
instead fo "tail -3".
> Fix 'hbase shell' output parsing in bash scripts
>
>
> Key: HBASE-16044
> URL: https://issues.apache.org/jira/browse/HBASE-16044
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
>
> In some of our bash scripts we are piping command in hbase shell and then
> parsing response to define variables. Since 'hbase shell' output format is
> changed we are picking wrong values from output Here is example form
> gracful_stop.sh:
> {code}
> HBASE_BALANCER_STATE=$(echo 'balance_switch false' | "$bin"/hbase --config
> "${HBASE_CONF_DIR}" shell | tail -3 | head -1)
> {code}
> this will return "balance_switch true" instead of previous balancer state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16044) Fix 'hbase shell' output parsing in bash scripts
[
https://issues.apache.org/jira/browse/HBASE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334203#comment-15334203
]
Samir Ahmic commented on HBASE-16044:
-
I think it was [ HBASE-15849 |
https://issues.apache.org/jira/browse/HBASE-15849] by adding this line:
{code}
+formatter.output_str("Took %.4f seconds" % [@end_time - @start_time])
{code}
It is simple fix in graceful_stop.sh and rolling-restart.sh just "tail -2"
instead fo "tail -3".
> Fix 'hbase shell' output parsing in bash scripts
>
>
> Key: HBASE-16044
> URL: https://issues.apache.org/jira/browse/HBASE-16044
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
>
> In some of our bash scripts we are piping command in hbase shell and then
> parsing response to define variables. Since 'hbase shell' output format is
> changed we are picking wrong values from output Here is example form
> gracful_stop.sh:
> {code}
> HBASE_BALANCER_STATE=$(echo 'balance_switch false' | "$bin"/hbase --config
> "${HBASE_CONF_DIR}" shell | tail -3 | head -1)
> {code}
> this will return "balance_switch true" instead of previous balancer state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16044) Fix 'hbase shell' output parsing in bash scripts
Samir Ahmic created HBASE-16044:
---
Summary: Fix 'hbase shell' output parsing in bash scripts
Key: HBASE-16044
URL: https://issues.apache.org/jira/browse/HBASE-16044
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 2.0.0
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Fix For: 2.0.0
In some of our bash scripts we are piping command in hbase shell and then
parsing response to define variables. Since 'hbase shell' output format is
changed we are picking wrong values from output Here is example form
gracful_stop.sh:
{code}
HBASE_BALANCER_STATE=$(echo 'balance_switch false' | "$bin"/hbase --config
"${HBASE_CONF_DIR}" shell | tail -3 | head -1)
{code}
this will return "balance_switch true" instead of previous balancer state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15908) Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
[
https://issues.apache.org/jira/browse/HBASE-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333743#comment-15333743
]
Samir Ahmic commented on HBASE-15908:
-
I have just test 1.3+hadoop-2.5.2+native libs loaded all looks clear issue is
fixed.
Also i have tested master+hadoop-2.5.2+native libs loaded, issue is fixed.
Not sure why fix was not present in master branch 2 days ago when i was
building code.
[~mantonov] sorry for inconvenience.
> Checksum verification is broken due to incorrect passing of ByteBuffers in
> DataChecksum
> ---
>
> Key: HBASE-15908
> URL: https://issues.apache.org/jira/browse/HBASE-15908
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: master.v1.patch
>
>
> It looks like HBASE-11625 (cc [~stack], [~appy]) has broken checksum
> verification? I'm seeing the following on my cluster (1.3.0, Hadoop 2.7).
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1135)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.lang.IllegalArgumentException: input ByteBuffers must be
> direct buffers
> at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native
> Method)
> at
> org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301)
> at
> org.apache.hadoop.hbase.io.hfile.ChecksumUtil.validateChecksum(ChecksumUtil.java:120)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.validateChecksum(HFileBlock.java:1785)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1728)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:151)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV3.(HFileReaderV3.java:78)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:487)
> ... 16 more
> Prior this change we won't use use native crc32 checksum verification as in
> Hadoop's DataChecksum#verifyChunkedSums we would go this codepath
> if (data.hasArray() && checksums.hasArray()) {
>
> }
> So we were fine. However, now we're dropping below and try to use the
> slightly different variant of native crc32 (if one is available) taking
> ByteBuffer instead of byte[], which expects DirectByteBuffer, not Heap BB.
> I think easiest fix working on all Hadoops would be to remove asReadonly()
> conversion here:
> !validateChecksum(offset, onDiskBlockByteBuffer.asReadOnlyBuffer(), hdrSize))
> {
> I don't see why do we need it. Let me test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15908) Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
[
https://issues.apache.org/jira/browse/HBASE-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333562#comment-15333562
]
Samir Ahmic commented on HBASE-15908:
-
Sure thing. Let me try this with 1.3 and will report back.
> Checksum verification is broken due to incorrect passing of ByteBuffers in
> DataChecksum
> ---
>
> Key: HBASE-15908
> URL: https://issues.apache.org/jira/browse/HBASE-15908
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: master.v1.patch
>
>
> It looks like HBASE-11625 (cc [~stack], [~appy]) has broken checksum
> verification? I'm seeing the following on my cluster (1.3.0, Hadoop 2.7).
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1135)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.lang.IllegalArgumentException: input ByteBuffers must be
> direct buffers
> at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native
> Method)
> at
> org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301)
> at
> org.apache.hadoop.hbase.io.hfile.ChecksumUtil.validateChecksum(ChecksumUtil.java:120)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.validateChecksum(HFileBlock.java:1785)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1728)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:151)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV3.(HFileReaderV3.java:78)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:487)
> ... 16 more
> Prior this change we won't use use native crc32 checksum verification as in
> Hadoop's DataChecksum#verifyChunkedSums we would go this codepath
> if (data.hasArray() && checksums.hasArray()) {
>
> }
> So we were fine. However, now we're dropping below and try to use the
> slightly different variant of native crc32 (if one is available) taking
> ByteBuffer instead of byte[], which expects DirectByteBuffer, not Heap BB.
> I think easiest fix working on all Hadoops would be to remove asReadonly()
> conversion here:
> !validateChecksum(offset, onDiskBlockByteBuffer.asReadOnlyBuffer(), hdrSize))
> {
> I don't see why do we need it. Let me test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15908) Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
[
https://issues.apache.org/jira/browse/HBASE-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331590#comment-15331590
]
Samir Ahmic edited comment on HBASE-15908 at 6/15/16 12:05 PM:
---
I did not try other hadoop/hbase versions but i'm pretty sure that this will be
issue whenever hadoop native libs are loaded. I have checked hadoop-2.7.1 and
hadoop-2.5.2 DataChecksum#verifyChunkedSums() and in both branches there is
this condition:
{code}
if (NativeCrc32.isAvailable()) {
NativeCrc32.verifyChunkedSums(bytesPerChecksum, type.id, checksums, data,
fileName, basePos);
return;
}
{code}
I was able to workaround this issue by making direct ByteBuffers and use them
if native libs are loaded. Here is diff:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
index a47cc12..48310c9 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
@@ -27,6 +27,7 @@ import org.apache.hadoop.fs.ChecksumException;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.util.ChecksumType;
import org.apache.hadoop.util.DataChecksum;
+import org.apache.hadoop.util.NativeCodeLoader;
/**
* Utility methods to compute and validate checksums.
@@ -117,12 +118,27 @@ public class ChecksumUtil {
ByteBuffer data = (ByteBuffer)
buffer.duplicate().position(0).limit(onDiskDataSizeWithHeader);
ByteBuffer checksums = (ByteBuffer)
buffer.duplicate().position(onDiskDataSizeWithHeader)
.limit(buffer.capacity());
+ if(NativeCodeLoader.isNativeCodeLoaded()) {
+dataChecksum.verifyChunkedSums(directify(data), directify(checksums),
pathName, 0);
+ } else {
dataChecksum.verifyChunkedSums(data, checksums, pathName, 0);
+ }
} catch (ChecksumException e) {
return false;
}
return true; // checksum is valid
}
+
+ private static ByteBuffer directify(ByteBuffer dataBuf) {
+ByteBuffer newBuf = ByteBuffer.allocateDirect(dataBuf.capacity());
+newBuf.position(dataBuf.position());
+newBuf.mark();
+newBuf.put(dataBuf);
+newBuf.reset();
+newBuf.limit(dataBuf.limit());
+return newBuf;
+ }
+
/**
* Returns the number of bytes needed to store the checksums for
{code}
was (Author: asamir):
I did not try other hadoop/hbase versions but i'm pretty sure that this will be
issue whenever hadoop native libs are loaded. I have checked hadoop-2.7.1 and
hadoop-2.5.2 DataChecksum#verifyChunkedSums() and in both branches there is
this condition:
{code}
if (NativeCrc32.isAvailable()) {
NativeCrc32.verifyChunkedSums(bytesPerChecksum, type.id, checksums, data,
fileName, basePos);
return;
}
{code}
I was able to workaround this issue by making direct ByteBuffers and use them
if native libs are loaded. I will attach diff.
> Checksum verification is broken due to incorrect passing of ByteBuffers in
> DataChecksum
> ---
>
> Key: HBASE-15908
> URL: https://issues.apache.org/jira/browse/HBASE-15908
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: master.v1.patch
>
>
> It looks like HBASE-11625 (cc [~stack], [~appy]) has broken checksum
> verification? I'm seeing the following on my cluster (1.3.0, Hadoop 2.7).
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1135)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>
[jira] [Commented] (HBASE-15908) Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
[
https://issues.apache.org/jira/browse/HBASE-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331590#comment-15331590
]
Samir Ahmic commented on HBASE-15908:
-
I did not try other hadoop/hbase versions but i'm pretty sure that this will be
issue whenever hadoop native libs are loaded. I have checked hadoop-2.7.1 and
hadoop-2.5.2 DataChecksum#verifyChunkedSums() and in both branches there is
this condition:
{code}
if (NativeCrc32.isAvailable()) {
NativeCrc32.verifyChunkedSums(bytesPerChecksum, type.id, checksums, data,
fileName, basePos);
return;
}
{code}
I was able to workaround this issue by making direct ByteBuffers and use them
if native libs are loaded. I will attach diff.
> Checksum verification is broken due to incorrect passing of ByteBuffers in
> DataChecksum
> ---
>
> Key: HBASE-15908
> URL: https://issues.apache.org/jira/browse/HBASE-15908
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: master.v1.patch
>
>
> It looks like HBASE-11625 (cc [~stack], [~appy]) has broken checksum
> verification? I'm seeing the following on my cluster (1.3.0, Hadoop 2.7).
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1135)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.lang.IllegalArgumentException: input ByteBuffers must be
> direct buffers
> at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native
> Method)
> at
> org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301)
> at
> org.apache.hadoop.hbase.io.hfile.ChecksumUtil.validateChecksum(ChecksumUtil.java:120)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.validateChecksum(HFileBlock.java:1785)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1728)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:151)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV3.(HFileReaderV3.java:78)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:487)
> ... 16 more
> Prior this change we won't use use native crc32 checksum verification as in
> Hadoop's DataChecksum#verifyChunkedSums we would go this codepath
> if (data.hasArray() && checksums.hasArray()) {
>
> }
> So we were fine. However, now we're dropping below and try to use the
> slightly different variant of native crc32 (if one is available) taking
> ByteBuffer instead of byte[], which expects DirectByteBuffer, not Heap BB.
> I think easiest fix working on all Hadoops would be to remove asReadonly()
> conversion here:
> !validateChecksum(offset, onDiskBlockByteBuffer.asReadOnlyBuffer(), hdrSize))
> {
> I don't see why do we need it. Let me test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16021) graceful_stop.sh: Wrap variables in double quote to avoid "[: too many arguments" error
[ https://issues.apache.org/jira/browse/HBASE-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-16021: Attachment: HBASE-16021_v1.patch Here is simple patch. > graceful_stop.sh: Wrap variables in double quote to avoid "[: too many > arguments" error > > > Key: HBASE-16021 > URL: https://issues.apache.org/jira/browse/HBASE-16021 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16021_v1.patch > > > On few places in graceful_stop.sh there is if conditions where variables are > not double quoted which may cause error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16021) graceful_stop.sh: Wrap variables in double quote to avoid "[: too many arguments" error
[ https://issues.apache.org/jira/browse/HBASE-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-16021: Status: Patch Available (was: Open) > graceful_stop.sh: Wrap variables in double quote to avoid "[: too many > arguments" error > > > Key: HBASE-16021 > URL: https://issues.apache.org/jira/browse/HBASE-16021 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16021_v1.patch > > > On few places in graceful_stop.sh there is if conditions where variables are > not double quoted which may cause error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16021) graceful_stop.sh: Wrap variables in double quote to avoid "[: too many arguments" error
[ https://issues.apache.org/jira/browse/HBASE-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-16021: Affects Version/s: 2.0.0 > graceful_stop.sh: Wrap variables in double quote to avoid "[: too many > arguments" error > > > Key: HBASE-16021 > URL: https://issues.apache.org/jira/browse/HBASE-16021 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > > On few places in graceful_stop.sh there is if conditions where variables are > not double quoted which may cause error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16021) graceful_stop.sh: Wrap variables in double quote to avoid "[: too many arguments" error
[ https://issues.apache.org/jira/browse/HBASE-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-16021: Fix Version/s: 2.0.0 > graceful_stop.sh: Wrap variables in double quote to avoid "[: too many > arguments" error > > > Key: HBASE-16021 > URL: https://issues.apache.org/jira/browse/HBASE-16021 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > > On few places in graceful_stop.sh there is if conditions where variables are > not double quoted which may cause error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16021) graceful_stop.sh: Wrap variables in double quote to avoid "[: too many arguments" error
Samir Ahmic created HBASE-16021: --- Summary: graceful_stop.sh: Wrap variables in double quote to avoid "[: too many arguments" error Key: HBASE-16021 URL: https://issues.apache.org/jira/browse/HBASE-16021 Project: HBase Issue Type: Bug Components: scripts Reporter: Samir Ahmic Assignee: Samir Ahmic Priority: Minor On few places in graceful_stop.sh there is if conditions where variables are not double quoted which may cause error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15908) Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
[
https://issues.apache.org/jira/browse/HBASE-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329026#comment-15329026
]
Samir Ahmic commented on HBASE-15908:
-
Few more information about my environment:
- Cent OS 6.7 (deployed in lxc container)
- java version 1.7.0_80
- hadoop-2.5.2 with native libs loaded
- hbase master branch 2.0.0-SNAPSHOT,
revision=2d0448fa84745991d2447ff78600866873b1fec0
> Checksum verification is broken due to incorrect passing of ByteBuffers in
> DataChecksum
> ---
>
> Key: HBASE-15908
> URL: https://issues.apache.org/jira/browse/HBASE-15908
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: master.v1.patch
>
>
> It looks like HBASE-11625 (cc [~stack], [~appy]) has broken checksum
> verification? I'm seeing the following on my cluster (1.3.0, Hadoop 2.7).
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1135)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.lang.IllegalArgumentException: input ByteBuffers must be
> direct buffers
> at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native
> Method)
> at
> org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301)
> at
> org.apache.hadoop.hbase.io.hfile.ChecksumUtil.validateChecksum(ChecksumUtil.java:120)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.validateChecksum(HFileBlock.java:1785)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1728)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:151)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV3.(HFileReaderV3.java:78)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:487)
> ... 16 more
> Prior this change we won't use use native crc32 checksum verification as in
> Hadoop's DataChecksum#verifyChunkedSums we would go this codepath
> if (data.hasArray() && checksums.hasArray()) {
>
> }
> So we were fine. However, now we're dropping below and try to use the
> slightly different variant of native crc32 (if one is available) taking
> ByteBuffer instead of byte[], which expects DirectByteBuffer, not Heap BB.
> I think easiest fix working on all Hadoops would be to remove asReadonly()
> conversion here:
> !validateChecksum(offset, onDiskBlockByteBuffer.asReadOnlyBuffer(), hdrSize))
> {
> I don't see why do we need it. Let me test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15908) Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
[
https://issues.apache.org/jira/browse/HBASE-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328040#comment-15328040
]
Samir Ahmic commented on HBASE-15908:
-
Here is exception i'm seeing
{code}
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
hdfs://P3cluster/hbase/data/default/cluster_test/37b19126a6455b5efd454b7774e22298/test_cf/390bef6889a042d6a08a1a386f29314d
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:518)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:547)
at
org.apache.hadoop.hbase.regionserver.StoreFileReader.(StoreFileReader.java:94)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:270)
at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:419)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:526)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:516)
at
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:614)
at
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:115)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:481)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:478)
... 6 more
---> Caused by: java.lang.IllegalArgumentException: input ByteBuffers must
be direct buffers
at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native
Method)
at
org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57)
at
org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:299)
at
org.apache.hadoop.hbase.io.hfile.ChecksumUtil.validateChecksum(ChecksumUtil.java:120)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.validateChecksum(HFileBlock.java:1775)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1714)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1547)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$2.nextBlock(HFileBlock.java:1447)
{code}
I have compiled master branch against hadoop-2.5.2 and deployed in distributed
mode.
> Checksum verification is broken due to incorrect passing of ByteBuffers in
> DataChecksum
> ---
>
> Key: HBASE-15908
> URL: https://issues.apache.org/jira/browse/HBASE-15908
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: master.v1.patch
>
>
> It looks like HBASE-11625 (cc [~stack], [~appy]) has broken checksum
> verification? I'm seeing the following on my cluster (1.3.0, Hadoop 2.7).
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1135)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.lang.IllegalArgumentException: input ByteBuffers must be
> direct buffers
> at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native
> Method)
> at
> org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301)
> at
> org.apache.hadoop.hbase.io.hfile.ChecksumUtil.validateChecksum(ChecksumUtil.java:120)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.validateChecksum(HFileBlock.java:1785)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1728)
> at
>
[jira] [Commented] (HBASE-15908) Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum
[
https://issues.apache.org/jira/browse/HBASE-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327781#comment-15327781
]
Samir Ahmic commented on HBASE-15908:
-
I still see this issue on master branch + hadoop-2.5.2. Should we reopen this
ticket or crate new one ?
> Checksum verification is broken due to incorrect passing of ByteBuffers in
> DataChecksum
> ---
>
> Key: HBASE-15908
> URL: https://issues.apache.org/jira/browse/HBASE-15908
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: master.v1.patch
>
>
> It looks like HBASE-11625 (cc [~stack], [~appy]) has broken checksum
> verification? I'm seeing the following on my cluster (1.3.0, Hadoop 2.7).
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1135)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
> ... 6 more
> Caused by: java.lang.IllegalArgumentException: input ByteBuffers must be
> direct buffers
> at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native
> Method)
> at
> org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301)
> at
> org.apache.hadoop.hbase.io.hfile.ChecksumUtil.validateChecksum(ChecksumUtil.java:120)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.validateChecksum(HFileBlock.java:1785)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1728)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:151)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV3.(HFileReaderV3.java:78)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:487)
> ... 16 more
> Prior this change we won't use use native crc32 checksum verification as in
> Hadoop's DataChecksum#verifyChunkedSums we would go this codepath
> if (data.hasArray() && checksums.hasArray()) {
>
> }
> So we were fine. However, now we're dropping below and try to use the
> slightly different variant of native crc32 (if one is available) taking
> ByteBuffer instead of byte[], which expects DirectByteBuffer, not Heap BB.
> I think easiest fix working on all Hadoops would be to remove asReadonly()
> conversion here:
> !validateChecksum(offset, onDiskBlockByteBuffer.asReadOnlyBuffer(), hdrSize))
> {
> I don't see why do we need it. Let me test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15272) Add error handling for split and compact actions in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15272: Resolution: Fixed Status: Resolved (was: Patch Available) This was fixed in meantime. > Add error handling for split and compact actions in table.jsp > - > > Key: HBASE-15272 > URL: https://issues.apache.org/jira/browse/HBASE-15272 > Project: HBase > Issue Type: Sub-task > Components: UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-15272_v0.patch > > > Split and compact actions in table.jsp throwing 500 error in case when > non-existing key is passed as parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15943: Status: Patch Available (was: Open) > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15943_v0.patch, HBASE-15943_v1.patch, > processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15943: Attachment: HBASE-15943_v1.patch Here is new patch getting values from MBeans avoiding dumping to JSON and parsing back. > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15943_v0.patch, HBASE-15943_v1.patch, > processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15312950#comment-15312950 ] Samir Ahmic commented on HBASE-15943: - Thanks for review [~enis]. bq. Can you obtain the metrics without dumping everything to json then parsing back? Probably yes by searching MBean object for specific attribute. Do you have any other idea ? bq. What is different in processRS versus processMaster ? At this point only one thing there is "JvmPauseMonitor Count" on processRS page. Since we have separated webapps for master and rs i have created two separated pages. > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15943_v0.patch, processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15943) Add page displaying JVM process metrics
[ https://issues.apache.org/jira/browse/HBASE-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15943: Attachment: HBASE-15943_v0.patch processInfo.png Here is patch v0 with screenshot how this pages look. If you think this is something that will be useful comments and critics are welcome:). > Add page displaying JVM process metrics > --- > > Key: HBASE-15943 > URL: https://issues.apache.org/jira/browse/HBASE-15943 > Project: HBase > Issue Type: New Feature > Components: UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15943_v0.patch, processInfo.png > > > It would be useful to have page displaying some JVM metrics like PID, process > owner. threads info, GC info, etc. This ticked will create two jsp pages (for > master and rs) displaying stats listed above. > - -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15943) Add page displaying JVM process metrics
Samir Ahmic created HBASE-15943: --- Summary: Add page displaying JVM process metrics Key: HBASE-15943 URL: https://issues.apache.org/jira/browse/HBASE-15943 Project: HBase Issue Type: New Feature Components: UI Affects Versions: 2.0.0 Reporter: Samir Ahmic Assignee: Samir Ahmic Priority: Minor Fix For: 2.0.0 It would be useful to have page displaying some JVM metrics like PID, process owner. threads info, GC info, etc. This ticked will create two jsp pages (for master and rs) displaying stats listed above. - -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15722) Size. WAL Files (bytes) in regionserver status page displays negative values
[
https://issues.apache.org/jira/browse/HBASE-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280043#comment-15280043
]
Samir Ahmic commented on HBASE-15722:
-
Filed test is {{TestRegionServerMetrics#testMobMetrics()}} i do not think
failed test is related with this patch.
> Size. WAL Files (bytes) in regionserver status page displays negative values
>
>
> Key: HBASE-15722
> URL: https://issues.apache.org/jira/browse/HBASE-15722
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Minor
> Attachments: HBASE-15722_v0.patch, HBASE-15722_v1.patch,
> HBASE-15722_v2.patch, WALs.png
>
>
> Here is the line from ServerMetricTmpl.jamon
> {code}
> TraditionalBinaryPrefix.long2String(mWrap.getWALFileSize(), "B", 1)
> {code}
> I will change this to StringUtils.humanSize()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15722) Size. WAL Files (bytes) in regionserver status page displays negative values
[
https://issues.apache.org/jira/browse/HBASE-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-15722:
Attachment: HBASE-15722_v2.patch
[~enis] here is new patch fixing issue in {{doReplaceWriter()}} and using this
function for getting WALs file size. Issue was cause because we were getting
writer length before writer was closed resulting that return value was less
then actual file size on HDFS.
> Size. WAL Files (bytes) in regionserver status page displays negative values
>
>
> Key: HBASE-15722
> URL: https://issues.apache.org/jira/browse/HBASE-15722
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Minor
> Attachments: HBASE-15722_v0.patch, HBASE-15722_v1.patch,
> HBASE-15722_v2.patch, WALs.png
>
>
> Here is the line from ServerMetricTmpl.jamon
> {code}
> TraditionalBinaryPrefix.long2String(mWrap.getWALFileSize(), "B", 1)
> {code}
> I will change this to StringUtils.humanSize()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15722) Size. WAL Files (bytes) in regionserver status page displays negative values
[
https://issues.apache.org/jira/browse/HBASE-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276646#comment-15276646
]
Samir Ahmic commented on HBASE-15722:
-
Sure thing. I assume you mean {{doReplaceWriter()}} implementation from
{{FSHLog.java}} ?
> Size. WAL Files (bytes) in regionserver status page displays negative values
>
>
> Key: HBASE-15722
> URL: https://issues.apache.org/jira/browse/HBASE-15722
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Minor
> Attachments: HBASE-15722_v0.patch, HBASE-15722_v1.patch, WALs.png
>
>
> Here is the line from ServerMetricTmpl.jamon
> {code}
> TraditionalBinaryPrefix.long2String(mWrap.getWALFileSize(), "B", 1)
> {code}
> I will change this to StringUtils.humanSize()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
