Re: [DISCUSS] Changing hadoop check versions in our hbase-personality?
+1 Definitely valuable for HBase users to be aware of Hadoop CVEs. Peter On Tue, Nov 6, 2018 at 6:16 AM Stack wrote: > On Mon, Nov 5, 2018 at 10:09 AM Sean Busbey wrote: > > > As of last week the Apache Hadoop project now keeps a public list of CVEs > > that are public. (although it's currently limited to things from the last > > year) > > > > https://hadoop.apache.org/cve_list.html > > > > Folks okay with linking to that page and updating Hadoop requirements for > > the next minor releases in 1.y and 2.y to be versions without something > > listed there? > > > > > Sounds good. > > > > What about dropping all the Hadoop 2's for HBase 3? (maybe a new thread? > > :smile:) > > > > > I wish. > > Yeah, another thread. > > S > > > > On 2018/10/24 15:21:32, Sean Busbey wrote: > > > FYI HADOOP-15878 is tracking publishing impacted versions for the CVE > > > > > > On Tue, Oct 23, 2018 at 11:37 AM Sean Busbey > wrote: > > > > > > > > I can get behind more aggressively updating our dependencies to avoid > > > > CVEs. I don't think we should do this in maintenance releases though. > > > > Maintenance releases are meant to be extremely low risk for > downstream > > > > users. Despite our efforts to date upgrading a dependency is still > > > > disruptive, especially when it's Hadoop. CVEs carry with them a > needed > > > > context for something to be an issue. That context almost never > covers > > > > all possible deployment scenarios and we should leave it to > downstream > > > > users to decide if the risk / reward trade off of justifies the > > > > dependency update. Asking folks who think the risk is worth it to > bump > > > > up a minor HBase version or patch their deployment locally is a > > > > reasonable trade off IMHO. > > > > > > > > I think I have the Hadoop PMC on board for publicizing impacted > > > > versions on CVE-2018-8009 specifically. Give me a couple of days to > > > > get that out in whatever form so that everyone in this discussion has > > > > a more level field? > > > > On Mon, Oct 22, 2018 at 9:07 PM 张铎(Duo Zhang) > > > wrote: > > > > > > > > > > I believe if there is a CVE for any of the dependencies we should > > try to > > > > > upgrade it, and IIRC there is an issue about finding these > > dependencies out > > > > > automatically. We haven't done this before does not mean ignoring a > > CVE is > > > > > the correct way, it is just because no one takes care of it... > > > > > > > > > > And the hadoop team has stated the versions which are vulnerable, > all > > > > > versions before 2.7.7, 2.8.5, 2.9.2(not released yet?), 3.0.3 and > > 3.1.1. > > > > > Not sure if they have published this out to the public, but as you > > can see > > > > > the url provided by me above, it is already public, so it does not > > matter > > > > > whether the hadoop team has published or not... > > > > > > > > > > Sean Busbey 于2018年10月23日周二 上午9:50写道： > > > > > > > > > > > Has the Hadoop PMC put out a public notice on the impact of that > > CVE yet? > > > > > > Specifically have they stated what versions are vulnerable? Are > we > > flagging > > > > > > all versions impacted by it as "HBase says keep away"? > > > > > > > > > > > > Is there some reason this particular CVE especially impacts users > > of HBase? > > > > > > I presume not since we're talking about this on dev@ and in JIRA > > instead > > > > > > of > > > > > > on private@. > > > > > > > > > > > > Why are we reacting to this CVE when we don't seem to react to > any > > other > > > > > > Hadoop CVEs? Or is this the start of a change wrt that? > > > > > > > > > > > > What about other dependencies with open CVEs? > > > > > > > > > > > > On Mon, Oct 22, 2018, 20:33 张铎(Duo Zhang) > > > wrote: > > > > > > > > > > > > > See here: > > > > > > > > > > > > > > https://access.redhat.com/security/cve/cve-2018-8009 > > > > > > > > > > > > > > All 2.7.x releases before 2.7.7 have the problem. And for > 2.6.x, > > the > > > > > > hadoop > > > > > > > team seems to drop the support as there is no release about two > > years, so > > > > > > > either we keep the original support versions, or we just drop > > the support > > > > > > > for the 2.6.x release line. > > > > > > > > > > > > > > Zach York 于2018年10月23日周二 > > 上午8:51写道： > > > > > > > > > > > > > > > What is the main reason for the change? Build time speedup? > > > > > > > > > > > > > > > > Any reason for testing all of the 2.6.x line, but not the > > 2.7.x line? > > > > > > We > > > > > > > > don't check at all for 2.8.x? > > > > > > > > > > > > > > > > Can we be more consistent with how we test compatibility? (Do > > we only > > > > > > > care > > > > > > > > about the latest patch release in a line?) > > > > > > > > > > > > > > > > Sorry If I'm missing some of the reasoning, but at a surface > > level it > > > > > > > seems > > > > > > > > fairly arbitrary which releases we are cutting. > > > > > > > > > > > > > > > > On Mon, Oct 22, 2018 at 5:44 PM Sean Busbey < > bus...@apache.org> > > wrote: > > >
[jira] [Created] (HBASE-21441) NPE if RS crashes between REFRESH_PEER_SYNC_REPLICATION_STATE_ON_RS_BEGIN and TRANSIT_PEER_NEW_SYNC_REPLICATION_STATE
Duo Zhang created HBASE-21441: - Summary: NPE if RS crashes between REFRESH_PEER_SYNC_REPLICATION_STATE_ON_RS_BEGIN and TRANSIT_PEER_NEW_SYNC_REPLICATION_STATE Key: HBASE-21441 URL: https://issues.apache.org/jira/browse/HBASE-21441 Project: HBase Issue Type: Sub-task Components: Replication Reporter: Duo Zhang Fix For: 3.0.0 2018-11-06,12:55:25,980 WARN [RpcServer.default.FPBQ.Fifo.handler=251,queue=11,port=17100] org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure: Refresh peer TestPeer for TRANSIT_SYNC_REPLICATION_STATE on c4-hadoop-tst-st54.bj,17200,1541479922465 failed java.lang.NullPointerException via c4-hadoop-tst-st54.bj,17200,1541479922465:java.lang.NullPointerException: at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:124) at org.apache.hadoop.hbase.master.MasterRpcServices.lambda$reportProcedureDone$4(MasterRpcServices.java:2303) at java.util.ArrayList.forEach(ArrayList.java:1249) at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) at org.apache.hadoop.hbase.master.MasterRpcServices.reportProcedureDone(MasterRpcServices.java:2298) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13149) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) Caused by: java.lang.NullPointerException: at org.apache.hadoop.hbase.wal.SyncReplicationWALProvider.peerSyncReplicationStateChange(SyncReplicationWALProvider.java:303) at org.apache.hadoop.hbase.replication.regionserver.PeerProcedureHandlerImpl.transitSyncReplicationPeerState(PeerProcedureHandlerImpl.java:216) at org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:74) at org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:34) at org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:47) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [DISCUSS] Changing hadoop check versions in our hbase-personality?
On Mon, Nov 5, 2018 at 10:09 AM Sean Busbey wrote: > As of last week the Apache Hadoop project now keeps a public list of CVEs > that are public. (although it's currently limited to things from the last > year) > > https://hadoop.apache.org/cve_list.html > > Folks okay with linking to that page and updating Hadoop requirements for > the next minor releases in 1.y and 2.y to be versions without something > listed there? > > Sounds good. > What about dropping all the Hadoop 2's for HBase 3? (maybe a new thread? > :smile:) > > I wish. Yeah, another thread. S > On 2018/10/24 15:21:32, Sean Busbey wrote: > > FYI HADOOP-15878 is tracking publishing impacted versions for the CVE > > > > On Tue, Oct 23, 2018 at 11:37 AM Sean Busbey wrote: > > > > > > I can get behind more aggressively updating our dependencies to avoid > > > CVEs. I don't think we should do this in maintenance releases though. > > > Maintenance releases are meant to be extremely low risk for downstream > > > users. Despite our efforts to date upgrading a dependency is still > > > disruptive, especially when it's Hadoop. CVEs carry with them a needed > > > context for something to be an issue. That context almost never covers > > > all possible deployment scenarios and we should leave it to downstream > > > users to decide if the risk / reward trade off of justifies the > > > dependency update. Asking folks who think the risk is worth it to bump > > > up a minor HBase version or patch their deployment locally is a > > > reasonable trade off IMHO. > > > > > > I think I have the Hadoop PMC on board for publicizing impacted > > > versions on CVE-2018-8009 specifically. Give me a couple of days to > > > get that out in whatever form so that everyone in this discussion has > > > a more level field? > > > On Mon, Oct 22, 2018 at 9:07 PM 张铎(Duo Zhang) > wrote: > > > > > > > > I believe if there is a CVE for any of the dependencies we should > try to > > > > upgrade it, and IIRC there is an issue about finding these > dependencies out > > > > automatically. We haven't done this before does not mean ignoring a > CVE is > > > > the correct way, it is just because no one takes care of it... > > > > > > > > And the hadoop team has stated the versions which are vulnerable, all > > > > versions before 2.7.7, 2.8.5, 2.9.2(not released yet?), 3.0.3 and > 3.1.1. > > > > Not sure if they have published this out to the public, but as you > can see > > > > the url provided by me above, it is already public, so it does not > matter > > > > whether the hadoop team has published or not... > > > > > > > > Sean Busbey 于2018年10月23日周二 上午9:50写道： > > > > > > > > > Has the Hadoop PMC put out a public notice on the impact of that > CVE yet? > > > > > Specifically have they stated what versions are vulnerable? Are we > flagging > > > > > all versions impacted by it as "HBase says keep away"? > > > > > > > > > > Is there some reason this particular CVE especially impacts users > of HBase? > > > > > I presume not since we're talking about this on dev@ and in JIRA > instead > > > > > of > > > > > on private@. > > > > > > > > > > Why are we reacting to this CVE when we don't seem to react to any > other > > > > > Hadoop CVEs? Or is this the start of a change wrt that? > > > > > > > > > > What about other dependencies with open CVEs? > > > > > > > > > > On Mon, Oct 22, 2018, 20:33 张铎(Duo Zhang) > wrote: > > > > > > > > > > > See here: > > > > > > > > > > > > https://access.redhat.com/security/cve/cve-2018-8009 > > > > > > > > > > > > All 2.7.x releases before 2.7.7 have the problem. And for 2.6.x, > the > > > > > hadoop > > > > > > team seems to drop the support as there is no release about two > years, so > > > > > > either we keep the original support versions, or we just drop > the support > > > > > > for the 2.6.x release line. > > > > > > > > > > > > Zach York 于2018年10月23日周二 > 上午8:51写道： > > > > > > > > > > > > > What is the main reason for the change? Build time speedup? > > > > > > > > > > > > > > Any reason for testing all of the 2.6.x line, but not the > 2.7.x line? > > > > > We > > > > > > > don't check at all for 2.8.x? > > > > > > > > > > > > > > Can we be more consistent with how we test compatibility? (Do > we only > > > > > > care > > > > > > > about the latest patch release in a line?) > > > > > > > > > > > > > > Sorry If I'm missing some of the reasoning, but at a surface > level it > > > > > > seems > > > > > > > fairly arbitrary which releases we are cutting. > > > > > > > > > > > > > > On Mon, Oct 22, 2018 at 5:44 PM Sean Busbey > wrote: > > > > > > > > > > > > > > > Please leave me time to review before it is committed. > > > > > > > > > > > > > > > > On Mon, Oct 22, 2018, 13:58 Stack wrote: > > > > > > > > > > > > > > > > > Duo has a patch up on HBASE-20970 that changes the Hadoop > versions > > > > > we > > > > > > > > check > > > > > > > > > at build time. Any objections to committing to branch-2.1+? > > > > > > > > > > > > > > > > > > It makes
[jira] [Created] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted
Ankit Singhal created HBASE-21440: - Summary: Assign procedure on the crashed server is not properly interrupted Key: HBASE-21440 URL: https://issues.apache.org/jira/browse/HBASE-21440 Project: HBase Issue Type: Bug Reporter: Ankit Singhal Assignee: Ankit Singhal Fix For: 2.0.2 When the server crashes and it's SCP checks if there is already a procedure assigning the region on this crashed server. If we found one, SCP will just interrupt the already running AssignProcedure by calling remoteCallFailed which just changes the region node state to OFFLINE and send the procedure back with transition queue state for assignment with a new plan. But, due to the race condition between the calling of the remoteCallFailed and current state of the already running assign procedure(REGION_TRANSITION_FINISH: where the region is already opened), it is possible that assign procedure goes ahead in updating the regionStateNode to OPEN on a crashed server. As SCP had already skipped this region for assignment as it was relying on existing assign procedure to do the right thing, this whole confusion leads region to a not accessible state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21439) StochasticLoadBalancer RegionLoads aren’t being used in RegionLoad cost functions
Ben Lau created HBASE-21439: --- Summary: StochasticLoadBalancer RegionLoads aren’t being used in RegionLoad cost functions Key: HBASE-21439 URL: https://issues.apache.org/jira/browse/HBASE-21439 Project: HBase Issue Type: Bug Components: Balancer Affects Versions: 2.0.2, 1.3.2.1 Reporter: Ben Lau Assignee: Ben Lau In StochasticLoadBalancer.updateRegionLoad() the region loads are being put into the map with Bytes.toString(regionName). First, this is a problem because Bytes.toString() assumes that the byte array is a UTF8 encoded String but there is no guarantee that regionName bytes are legal UTF8. Secondly, in BaseLoadBalancer.registerRegion, we are reading the region loads out of the load map not using Bytes.toString() but using region.getRegionNameAsString() and region.getEncodedName(). So the load balancer will not see or use any of the cluster's RegionLoad history. There are 2 primary ways to solve this issue, assuming we want to stay with String keys for the load map (seems reasonable to aid debugging). We can either fix updateRegionLoad to store the regionName as a string properly or we can update both the reader & writer to use a new common valid String representation. Will post a patch assuming we want to pursue the original intention, i.e. store regionNameAsAString for the loadmap key, but I'm open to fixing this a different way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [DISCUSS] Gitbox<-> JIRA (WAS: [DISCUSS] Kafka Connection, HBASE-15320)
+1, please do On Mon, Nov 5, 2018 at 9:55 AM Stack wrote: > I was going to file an issue w/ INFRA this evening asking that they disable > github comment/commits being added as JIRA comments (See HBASE-21435 for an > innocuous example of this feature in action and HBASE-21430 for an > illustration of how it can quickly turn silly). While I could see > github-comments-appearing-as-JIRA comments working if the changesets were > small, it blows away JIRA if the change has any weight at all. > > I was going to ask that the github/gitbox commentary instead go to issues@ > mailing list (Sean suggestion. Duo and our Peter asked off-list suggested > we not fill JIRA comments). Shout if you are against or have any > preferences. > > Yours, > S > > > > On Fri, Nov 2, 2018 at 11:43 AM Stack wrote: > > > Related, interesting thread on how various projects are doing the > > github<->asf connection has just started[1]. > > > > A few have a notifications list that gets the github notification emails. > > Email clients make parseable threads of the firehose apparently. We might > > do same. > > > > S > > 1. > > > http://mail-archives.apache.org/mod_mbox/community-dev/201811.mbox/browser > > > > On Fri, Nov 2, 2018 at 10:37 AM Josh Elser wrote: > > > >> > >> > >> On 11/2/18 11:20 AM, Stack wrote: > >> > On Fri, Nov 2, 2018 at 7:30 AM Josh Elser wrote: > >> > > >> >> Nice stuff, Stack! > >> >> > >> >> > >> > Not me. It Mike's work. > >> > > >> > >> Sorry, didn't mean it that way. Was happy to see you pushing the > >> hbase-connectors repo forward :) > >> > >> Obviously kudos to MikeW for the code itself! > >> > >> >> Two quick questions: > >> >> > >> >> First, on provenance: this codebase primarily came from Mike Wingert > on > >> >> https://issues.apache.org/jira/browse/HBASE-15320? Just saw that the > >> >> commit came from your email addr -- wasn't sure if that Mike was > still > >> >> involved (or you took it to completion). > >> >> > >> >> > >> > You talking about the merge done over on the hbase-connectors? > >> > > >> > Looks like I get blamed for the merge -- if I do a show on the merge > >> > commit, it is nothing but the merge note "Merge pull request #3 from > >> > hbasejanitor/master" where hbasejanitor is Mike's handle -- but then > the > >> > merge note is followed by Mikes' work, properly attributed to him. > >> > > >> > I did not pay close attention to this aspect of how git boxing does > it. > >> > Seems fine to me. What you think? > >> > >> Ahh, no, this was just me. I think I only looked at the first commit(s), > >> and not far enough down the list. No concerns. > >> > >> >> Second, I assume this new Git repo had all of the normal email-hooks > >> set > >> >> up. Do you know where they are being sent (dev, commit, or issues)? > I'm > >> >> also assuming that this is a Gitbox repo -- are we OK with > >> pull-requests > >> >> to this repo (as well as operator-tools) but still create a Jira > issue? > >> >> > >> >> > >> > Yep, gitbox. It has whatever infra set it up as. > >> > > >> > Back and forth was dumped into HBASE-21002 (We changed this config on > >> > hbase-operator-tools config to not do this? I should look). > >> > > >> > Regards pull requests, etc., email configs., etc., we are in > >> experimental > >> > mode around all this stuff trying to figure it out so > >> > suggestions/help/exercising the possibilities are all welcome. > >> > > >> > Thanks, > >> > S > >> > >> Grand. Thanks for the reminder. Had a random question in Slack the other > >> day about contributions -- will keep the "pave your own road" mindset in > >> the fore-front :) > >> > >> >> - Josh > >> >> > >> >> On 10/31/18 6:43 PM, Stack wrote: > >> >>> To tie-off this thread, this nice feature was just pushed on > >> >>> hbase-connector. See > >> >>> https://github.com/apache/hbase-connectors/tree/master/kafka for > >> how-to. > >> >>> Review and commentary welcome. > >> >>> > >> >>> Thanks, > >> >>> S > >> >>> > >> >>> On Fri, Aug 3, 2018 at 6:32 AM Hbase Janitor < > hbasejani...@gmail.com> > >> >> wrote: > >> >>> > >> I opened hbase-21002 to start the scripts and assembly. > >> > >> Mike > >> > >> On Thu, Aug 2, 2018, 19:29 Stack wrote: > >> > >> > Up in https://issues.apache.org/jira/browse/HBASE-20934 I created > >> an > >> > hbase-connectors repo. I put some form on it using the v19 patch > >> from > >> > HBASE-15320 "HBase connector for Kafka Connect". It builds and > tests > >> > pass. Here are some remaining TODOs: > >> > > >> >* Figure how to do start scripts: e.g. we need to start up the > >> kafka > >> > proxy. It wants some hbase jars, conf dir, and others on the > >> CLASSPATH > >> > (Depend on an HBASE_HOME and then source bin/hbase?) > >> >* Can any of the connectors make-do with the shaded client? > >> >* Make connectors standalone or have them share conf, bin, etc? > >> >* Need to do an assembly. Not done. > >> >*
[jira] [Created] (HBASE-21438) TestAdmin2#testGetProcedures fails due to FailedProcedure inaccessible
Ted Yu created HBASE-21438:
--
Summary: TestAdmin2#testGetProcedures fails due to FailedProcedure
inaccessible
Key: HBASE-21438
URL: https://issues.apache.org/jira/browse/HBASE-21438
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
>From
>https://builds.apache.org/job/HBase-Flaky-Tests/job/master/1863/testReport/org.apache.hadoop.hbase.client/TestAdmin2/testGetProcedures/
> :
{code}
Mon Nov 05 04:52:13 UTC 2018, RpcRetryingCaller{globalStartTime=1541393533029,
pause=250, maxAttempts=7},
org.apache.hadoop.hbase.procedure2.BadProcedureException:
org.apache.hadoop.hbase.procedure2.BadProcedureException: The procedure class
org.apache.hadoop.hbase.procedure2.FailedProcedure must be accessible and have
an empty constructor
at
org.apache.hadoop.hbase.procedure2.ProcedureUtil.validateClass(ProcedureUtil.java:82)
at
org.apache.hadoop.hbase.procedure2.ProcedureUtil.convertToProtoProcedure(ProcedureUtil.java:162)
at
org.apache.hadoop.hbase.master.MasterRpcServices.getProcedures(MasterRpcServices.java:1249)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DISCUSS] Changing hadoop check versions in our hbase-personality?
As of last week the Apache Hadoop project now keeps a public list of CVEs that are public. (although it's currently limited to things from the last year) https://hadoop.apache.org/cve_list.html Folks okay with linking to that page and updating Hadoop requirements for the next minor releases in 1.y and 2.y to be versions without something listed there? What about dropping all the Hadoop 2's for HBase 3? (maybe a new thread? :smile:) On 2018/10/24 15:21:32, Sean Busbey wrote: > FYI HADOOP-15878 is tracking publishing impacted versions for the CVE > > On Tue, Oct 23, 2018 at 11:37 AM Sean Busbey wrote: > > > > I can get behind more aggressively updating our dependencies to avoid > > CVEs. I don't think we should do this in maintenance releases though. > > Maintenance releases are meant to be extremely low risk for downstream > > users. Despite our efforts to date upgrading a dependency is still > > disruptive, especially when it's Hadoop. CVEs carry with them a needed > > context for something to be an issue. That context almost never covers > > all possible deployment scenarios and we should leave it to downstream > > users to decide if the risk / reward trade off of justifies the > > dependency update. Asking folks who think the risk is worth it to bump > > up a minor HBase version or patch their deployment locally is a > > reasonable trade off IMHO. > > > > I think I have the Hadoop PMC on board for publicizing impacted > > versions on CVE-2018-8009 specifically. Give me a couple of days to > > get that out in whatever form so that everyone in this discussion has > > a more level field? > > On Mon, Oct 22, 2018 at 9:07 PM 张铎(Duo Zhang) wrote: > > > > > > I believe if there is a CVE for any of the dependencies we should try to > > > upgrade it, and IIRC there is an issue about finding these dependencies > > > out > > > automatically. We haven't done this before does not mean ignoring a CVE is > > > the correct way, it is just because no one takes care of it... > > > > > > And the hadoop team has stated the versions which are vulnerable, all > > > versions before 2.7.7, 2.8.5, 2.9.2(not released yet?), 3.0.3 and 3.1.1. > > > Not sure if they have published this out to the public, but as you can see > > > the url provided by me above, it is already public, so it does not matter > > > whether the hadoop team has published or not... > > > > > > Sean Busbey 于2018年10月23日周二 上午9:50写道： > > > > > > > Has the Hadoop PMC put out a public notice on the impact of that CVE > > > > yet? > > > > Specifically have they stated what versions are vulnerable? Are we > > > > flagging > > > > all versions impacted by it as "HBase says keep away"? > > > > > > > > Is there some reason this particular CVE especially impacts users of > > > > HBase? > > > > I presume not since we're talking about this on dev@ and in JIRA instead > > > > of > > > > on private@. > > > > > > > > Why are we reacting to this CVE when we don't seem to react to any other > > > > Hadoop CVEs? Or is this the start of a change wrt that? > > > > > > > > What about other dependencies with open CVEs? > > > > > > > > On Mon, Oct 22, 2018, 20:33 张铎(Duo Zhang) wrote: > > > > > > > > > See here: > > > > > > > > > > https://access.redhat.com/security/cve/cve-2018-8009 > > > > > > > > > > All 2.7.x releases before 2.7.7 have the problem. And for 2.6.x, the > > > > hadoop > > > > > team seems to drop the support as there is no release about two > > > > > years, so > > > > > either we keep the original support versions, or we just drop the > > > > > support > > > > > for the 2.6.x release line. > > > > > > > > > > Zach York 于2018年10月23日周二 上午8:51写道： > > > > > > > > > > > What is the main reason for the change? Build time speedup? > > > > > > > > > > > > Any reason for testing all of the 2.6.x line, but not the 2.7.x > > > > > > line? > > > > We > > > > > > don't check at all for 2.8.x? > > > > > > > > > > > > Can we be more consistent with how we test compatibility? (Do we > > > > > > only > > > > > care > > > > > > about the latest patch release in a line?) > > > > > > > > > > > > Sorry If I'm missing some of the reasoning, but at a surface level > > > > > > it > > > > > seems > > > > > > fairly arbitrary which releases we are cutting. > > > > > > > > > > > > On Mon, Oct 22, 2018 at 5:44 PM Sean Busbey > > > > > > wrote: > > > > > > > > > > > > > Please leave me time to review before it is committed. > > > > > > > > > > > > > > On Mon, Oct 22, 2018, 13:58 Stack wrote: > > > > > > > > > > > > > > > Duo has a patch up on HBASE-20970 that changes the Hadoop > > > > > > > > versions > > > > we > > > > > > > check > > > > > > > > at build time. Any objections to committing to branch-2.1+? > > > > > > > > > > > > > > > > It makes following changes: > > > > > > > > > > > > > > > > 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 > > > > > > > > > > > > > > > > becomes > > > > > > > > > > > > > > > > 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.7 > > > > >
[DISCUSS] Gitbox<-> JIRA (WAS: [DISCUSS] Kafka Connection, HBASE-15320)
I was going to file an issue w/ INFRA this evening asking that they disable github comment/commits being added as JIRA comments (See HBASE-21435 for an innocuous example of this feature in action and HBASE-21430 for an illustration of how it can quickly turn silly). While I could see github-comments-appearing-as-JIRA comments working if the changesets were small, it blows away JIRA if the change has any weight at all. I was going to ask that the github/gitbox commentary instead go to issues@ mailing list (Sean suggestion. Duo and our Peter asked off-list suggested we not fill JIRA comments). Shout if you are against or have any preferences. Yours, S On Fri, Nov 2, 2018 at 11:43 AM Stack wrote: > Related, interesting thread on how various projects are doing the > github<->asf connection has just started[1]. > > A few have a notifications list that gets the github notification emails. > Email clients make parseable threads of the firehose apparently. We might > do same. > > S > 1. > http://mail-archives.apache.org/mod_mbox/community-dev/201811.mbox/browser > > On Fri, Nov 2, 2018 at 10:37 AM Josh Elser wrote: > >> >> >> On 11/2/18 11:20 AM, Stack wrote: >> > On Fri, Nov 2, 2018 at 7:30 AM Josh Elser wrote: >> > >> >> Nice stuff, Stack! >> >> >> >> >> > Not me. It Mike's work. >> > >> >> Sorry, didn't mean it that way. Was happy to see you pushing the >> hbase-connectors repo forward :) >> >> Obviously kudos to MikeW for the code itself! >> >> >> Two quick questions: >> >> >> >> First, on provenance: this codebase primarily came from Mike Wingert on >> >> https://issues.apache.org/jira/browse/HBASE-15320? Just saw that the >> >> commit came from your email addr -- wasn't sure if that Mike was still >> >> involved (or you took it to completion). >> >> >> >> >> > You talking about the merge done over on the hbase-connectors? >> > >> > Looks like I get blamed for the merge -- if I do a show on the merge >> > commit, it is nothing but the merge note "Merge pull request #3 from >> > hbasejanitor/master" where hbasejanitor is Mike's handle -- but then the >> > merge note is followed by Mikes' work, properly attributed to him. >> > >> > I did not pay close attention to this aspect of how git boxing does it. >> > Seems fine to me. What you think? >> >> Ahh, no, this was just me. I think I only looked at the first commit(s), >> and not far enough down the list. No concerns. >> >> >> Second, I assume this new Git repo had all of the normal email-hooks >> set >> >> up. Do you know where they are being sent (dev, commit, or issues)? I'm >> >> also assuming that this is a Gitbox repo -- are we OK with >> pull-requests >> >> to this repo (as well as operator-tools) but still create a Jira issue? >> >> >> >> >> > Yep, gitbox. It has whatever infra set it up as. >> > >> > Back and forth was dumped into HBASE-21002 (We changed this config on >> > hbase-operator-tools config to not do this? I should look). >> > >> > Regards pull requests, etc., email configs., etc., we are in >> experimental >> > mode around all this stuff trying to figure it out so >> > suggestions/help/exercising the possibilities are all welcome. >> > >> > Thanks, >> > S >> >> Grand. Thanks for the reminder. Had a random question in Slack the other >> day about contributions -- will keep the "pave your own road" mindset in >> the fore-front :) >> >> >> - Josh >> >> >> >> On 10/31/18 6:43 PM, Stack wrote: >> >>> To tie-off this thread, this nice feature was just pushed on >> >>> hbase-connector. See >> >>> https://github.com/apache/hbase-connectors/tree/master/kafka for >> how-to. >> >>> Review and commentary welcome. >> >>> >> >>> Thanks, >> >>> S >> >>> >> >>> On Fri, Aug 3, 2018 at 6:32 AM Hbase Janitor >> >> wrote: >> >>> >> I opened hbase-21002 to start the scripts and assembly. >> >> Mike >> >> On Thu, Aug 2, 2018, 19:29 Stack wrote: >> >> > Up in https://issues.apache.org/jira/browse/HBASE-20934 I created >> an >> > hbase-connectors repo. I put some form on it using the v19 patch >> from >> > HBASE-15320 "HBase connector for Kafka Connect". It builds and tests >> > pass. Here are some remaining TODOs: >> > >> >* Figure how to do start scripts: e.g. we need to start up the >> kafka >> > proxy. It wants some hbase jars, conf dir, and others on the >> CLASSPATH >> > (Depend on an HBASE_HOME and then source bin/hbase?) >> >* Can any of the connectors make-do with the shaded client? >> >* Make connectors standalone or have them share conf, bin, etc? >> >* Need to do an assembly. Not done. >> >* Move over REST and thrift next. Mapreduce after? >> > >> > The poms could do w/ a review. Hacked them over from >> hbase-thirdparty. >> > >> > File issues and apply patches up in JIRA if your up for any of the >> >> above. >> > >> > Thanks, >> > S >> > >> > On Wed, Jul 25, 2018 at 10:46 PM Stack wrote: >> >> >> >>
[jira] [Created] (HBASE-21437) Bypassed procedure throw IllegalArgumentException when its state is WAITING_TIMEOUT
Jingyun Tian created HBASE-21437:
Summary: Bypassed procedure throw IllegalArgumentException when
its state is WAITING_TIMEOUT
Key: HBASE-21437
URL: https://issues.apache.org/jira/browse/HBASE-21437
Project: HBase
Issue Type: Bug
Reporter: Jingyun Tian
Assignee: Jingyun Tian
{code}
2018-11-05,18:25:52,735 WARN
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker terminating
UNNATURALLY null
java.lang.IllegalArgumentException: NOT RUNNABLE! pid=3,
state=WAITING_TIMEOUT:REGION_STATE_TRANSITION_CLOSE, hasLock=true, bypass=true;
TransitRegionStateProcedure table=test_fail
over, region=1bb029ba4ec03b92061be5c4329d2096, UNASSIGN
at
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:134)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1620)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1384)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1948)
2018-11-05,18:25:52,736 TRACE
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker terminated.
{code}
Since when we bypassed a WAITING_TIMEOUT procedure and resubmit it, its state
is still WAITING_TIMEOUT, then when executor run this procedure, it will throw
exception and cause worker terminated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21436) Getting OOM frequently if hold many regions
Zephyr Guo created HBASE-21436: -- Summary: Getting OOM frequently if hold many regions Key: HBASE-21436 URL: https://issues.apache.org/jira/browse/HBASE-21436 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 2.0.2, 1.4.8, 3.0.0 Reporter: Zephyr Guo Recently, some feedback reached me from a customer which complains about NotServingRegionException thrown out at intevals. I examined his cluster and found there were quite a lot of OOM logs there but metric "readDataPerSecondKB/writeDataPerSecondKB" is in quite low level. In this customer's case, each RS has 3k regions and heap size of 4G. I dumped heap when OOM took place, and found that a lot of Chunk objects (counts as much as 1700) was there. Eventually, piecing all these evidences together, I came to the conclusion that: 1. The root cause is that global flush is triggered by size of all memstores, rather than size of all chunks. 2. A chunk is always allocated for each region, even we only write a few data to the region. And in this case, a total of 3.4G memory was consumed by 1700 chunks, although throughput is very low. Although 3K regions is too much for RS with 4G memory, it is still wise to improve RS stability in such scenario (In fact, most customers buy a small size HBase on cloud side). I provide a patch (only contain UT) to reproduce this case (just send a batch). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
