[jira] [Created] (HBASE-21254) Need to find a way to limit the number of proc wal files
Duo Zhang created HBASE-21254: - Summary: Need to find a way to limit the number of proc wal files Key: HBASE-21254 URL: https://issues.apache.org/jira/browse/HBASE-21254 Project: HBase Issue Type: Sub-task Components: proc-v2 Reporter: Duo Zhang Fix For: 3.0.0, 2.2.0 For regionserver, we have a max wal file limitation, if we reach the limitation, we will trigger a flush on specific regions so that we can delete old wal files. But for proc wals, we do not have this mechanism, and it will be worse after HBASE-21233, as if there is an old procedure which can not make progress and do not persist its state, we need to keep the old proc wal file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[ https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-19418. Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.1 1.4.8 2.2.0 1.3.3 1.5.0 3.0.0 Pushed up, thanks for the contribution [~ramatronics] > RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable. > - > > Key: HBASE-19418 > URL: https://issues.apache.org/jira/browse/HBASE-19418 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: Jean-Marc Spaggiari >Assignee: Ramie Raufdeen >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1 > > Attachments: HBASE-19418.master.000.patch > > > When RSs have a LOT of regions and CFs, flushing everything within 5 minutes > is not always doable. It might be interesting to be able to increase the > RANGE_OF_DELAY. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
HBase developer meetup
Hi all, I'm planning to put together an HBase developer meetup at the Salesforce office (with video conference for those who cannot attend in person) in the next few weeks. If you're interested please put your name in this spreadsheet:https://docs.google.com/spreadsheets/d/13eIMItFbM35K_lfn9woGGObW-12tfyOVLqZi8X4p5PM/edit#gid=0 This is a chance to get all those who contribute to HBase together. There will also be food. :)I will leave the spreadsheet up for one week - until Friday October 5th. Possible agenda:- Round-table- Status of branch-1, branch-2, and master- Current challenges (operations?, public cloud?, availability?, performance?, community?)- Future direction. Where do we want HBase to be in 1 years, 2 years, 5 years?- more... Thanks. -- Lars
Re: Are we accepting empty commits?
I think it's a bad precident but can't argue given the constraints presented. I agree a line in the commit for explaining the situation would be good. That could be an s.apache.org short URL to this thread even. > On Sep 28, 2018, at 12:08 AM, 张铎(Duo Zhang) wrote: > > OK if the empty commit itself is the way to close PRs then I think it is > fine. Maybe we could add a line in the commit message to mention this > explicitly that the commit message is the key to close the PRs, so that we > will not start discussion thread like again in the future? > > Thanks. > > Sean Busbey 于2018年9月28日周五 下午2:38写道： > >> The commit itself is the mechanism used to close PRs. If you'd like >> detailed information, please read the explanation I gave on the thread >> "move to github flow" earlier this month[1]. >> >> We already have a section in our committer guide asking folks who >> review PRs to make sure the needed language is in the commit that gets >> merged[2]. But even if mistakes didn't happen, the vast majority of >> opened PRs are closed out as stale because the contributor never >> follows up after we ask them to open a JIRA. >> >> the tl;dr: given our current project set up our options are >> >> 1) Ignore that there are github PRs present (and thus look like an >> inactive or poorly maintained project) >> >> 2) Switch to the newer "Dual Master" git setup (which will require a >> nontrivial amount of tooling changes) >> >> 3) Push commits that close PRs (as we are doing) >> >> As the person who's done the vast majority of this maintenance work >> over the last 3 years I'd be happy to see myself replaced with a bot. >> But for reasons explained in the excellent thread on a github workflow >> mentioned above, I don't think that should be a priority for the >> project. >> >> I've only had to push 8 empty commits to keep our GitHub PR list >> cleaned up[2] in this whole time. On each of those some committer as >> signed off on the idea that we need a commit to master that indicates >> the PR is done. >> >> [1]: https://s.apache.org/dBXC >> >> [2]: http://hbase.apache.org/book.html#_close_related_github_prs >> >> [3]: https://s.apache.org/hgCv >> On Thu, Sep 27, 2018 at 7:19 PM 张铎(Duo Zhang) >> wrote: >>> >>> I think record the information on jira is enough, so I prefer no empty >>> commit. >>> >>> Andrew Purtell 于2018年9月28日周五 上午9:56写道： >>> There is an odd commit on master that has 0 file changes. commit 98b1feac771d7cc5778fb14089834c99642a3533 Author: Sean Busbey Date: Wed Sep 26 15:00:05 2018 -0700 HBASE-21241 Close stale PRs * Closes #86 - referenced JIRA has already been merged * Closes #90 - no response from contributor for 24 days Signed-off-by: Peter Somogyi When did we agree to allow empty commits? I don't recall the >> discussion. For one I am opposed to this practice. Isn't the JIRA HBASE-21241 sufficient to account for this activity? -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk >>
[jira] [Created] (HBASE-21253) Backport HBASE-21244 Skip persistence when retrying for assignment related procedures to branch-2.0 and branch-2.1
Allan Yang created HBASE-21253: -- Summary: Backport HBASE-21244 Skip persistence when retrying for assignment related procedures to branch-2.0 and branch-2.1 Key: HBASE-21253 URL: https://issues.apache.org/jira/browse/HBASE-21253 Project: HBase Issue Type: Bug Affects Versions: 2.0.2, 2.1.0 Reporter: Allan Yang Assignee: Allan Yang See HBASE-21244 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Are we accepting empty commits?
There was discussion on the (incubator?) lists about allowing probot/stale to close inactive PRs which I understood to not need enough commits. Can't find the discussion now from mobile, but I think the option could be available to us if we make enough noise. Mike On Fri, Sep 28, 2018, 2:09 AM 张铎(Duo Zhang) wrote: > OK if the empty commit itself is the way to close PRs then I think it is > fine. Maybe we could add a line in the commit message to mention this > explicitly that the commit message is the key to close the PRs, so that we > will not start discussion thread like again in the future? > > Thanks. > > Sean Busbey 于2018年9月28日周五 下午2:38写道： > > > The commit itself is the mechanism used to close PRs. If you'd like > > detailed information, please read the explanation I gave on the thread > > "move to github flow" earlier this month[1]. > > > > We already have a section in our committer guide asking folks who > > review PRs to make sure the needed language is in the commit that gets > > merged[2]. But even if mistakes didn't happen, the vast majority of > > opened PRs are closed out as stale because the contributor never > > follows up after we ask them to open a JIRA. > > > > the tl;dr: given our current project set up our options are > > > > 1) Ignore that there are github PRs present (and thus look like an > > inactive or poorly maintained project) > > > > 2) Switch to the newer "Dual Master" git setup (which will require a > > nontrivial amount of tooling changes) > > > > 3) Push commits that close PRs (as we are doing) > > > > As the person who's done the vast majority of this maintenance work > > over the last 3 years I'd be happy to see myself replaced with a bot. > > But for reasons explained in the excellent thread on a github workflow > > mentioned above, I don't think that should be a priority for the > > project. > > > > I've only had to push 8 empty commits to keep our GitHub PR list > > cleaned up[2] in this whole time. On each of those some committer as > > signed off on the idea that we need a commit to master that indicates > > the PR is done. > > > > [1]: https://s.apache.org/dBXC > > > > [2]: http://hbase.apache.org/book.html#_close_related_github_prs > > > > [3]: https://s.apache.org/hgCv > > On Thu, Sep 27, 2018 at 7:19 PM 张铎(Duo Zhang) > > wrote: > > > > > > I think record the information on jira is enough, so I prefer no empty > > > commit. > > > > > > Andrew Purtell 于2018年9月28日周五 上午9:56写道： > > > > > > > There is an odd commit on master that has 0 file changes. > > > > > > > > commit 98b1feac771d7cc5778fb14089834c99642a3533 > > > > Author: Sean Busbey > > > > Date: Wed Sep 26 15:00:05 2018 -0700 > > > > > > > > HBASE-21241 Close stale PRs > > > > > > > > * Closes #86 - referenced JIRA has already been merged > > > > * Closes #90 - no response from contributor for 24 days > > > > > > > > Signed-off-by: Peter Somogyi > > > > > > > > When did we agree to allow empty commits? I don't recall the > > discussion. > > > > For one I am opposed to this practice. Isn't the JIRA HBASE-21241 > > > > sufficient to account for this activity? > > > > > > > > -- > > > > Best regards, > > > > Andrew > > > > > > > > Words like orphans lost among the crosstalk, meaning torn from > truth's > > > > decrepit hands > > > >- A23, Crosstalk > > > > > > >
[jira] [Resolved] (HBASE-21252) Collect loadavg when gathering machine environment
[ https://issues.apache.org/jira/browse/HBASE-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-21252. --- Resolution: Not A Problem OK, uptime already contains the load information. Let me check. > Collect loadavg when gathering machine environment > -- > > Key: HBASE-21252 > URL: https://issues.apache.org/jira/browse/HBASE-21252 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > > Skimmed several flaky test report, it seems that, for a successful run, the > SystemLoadAverage is usually less than 500, and for a failed run, the > SystemLoadAverage is usually greater than 1000. > Let's collect the loadavg in the script to see if the machine itself has > already overloaded before we actually execute the tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21252) Collect loadavg when gathering machine environment
Duo Zhang created HBASE-21252: - Summary: Collect loadavg when gathering machine environment Key: HBASE-21252 URL: https://issues.apache.org/jira/browse/HBASE-21252 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang Skimmed several flaky test report, it seems that, for a successful run, the SystemLoadAverage is usually less than 500, and for a failed run, the SystemLoadAverage is usually greater than 1000. Let's collect the loadavg in the script to see if the machine itself has already overloaded before we actually execute the tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21251) Refactor RegionMover
Guanghao Zhang created HBASE-21251: -- Summary: Refactor RegionMover Key: HBASE-21251 URL: https://issues.apache.org/jira/browse/HBASE-21251 Project: HBase Issue Type: Improvement Reporter: Guanghao Zhang Assignee: Guanghao Zhang 1. Move connection and admin to RegionMover's member variables. No need create connection many times. 2. use try-with-resource to reduce code 3. use ServerName instead of String 4. don't use Deprecated method 5. remove duplicate code .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21250) Refactor WALProcedureStore and add more comments for better understanding the implementation
Duo Zhang created HBASE-21250:
-
Summary: Refactor WALProcedureStore and add more comments for
better understanding the implementation
Key: HBASE-21250
URL: https://issues.apache.org/jira/browse/HBASE-21250
Project: HBase
Issue Type: Sub-task
Reporter: Duo Zhang
Fix For: 3.0.0, 2.2.0
The implementation is complicated and lack of comments to say how it works.
{code}
/**
* WAL implementation of the ProcedureStore.
* @see ProcedureWALPrettyPrinter for printing content of a single WAL.
* @see #main(String[]) to parse a directory of MasterWALProcs.
*/
{code}
I think at least we can move sub classes to separated files to make the class
smaller, and add more comments to describe what is going on here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Are we accepting empty commits?
OK if the empty commit itself is the way to close PRs then I think it is fine. Maybe we could add a line in the commit message to mention this explicitly that the commit message is the key to close the PRs, so that we will not start discussion thread like again in the future? Thanks. Sean Busbey 于2018年9月28日周五 下午2:38写道： > The commit itself is the mechanism used to close PRs. If you'd like > detailed information, please read the explanation I gave on the thread > "move to github flow" earlier this month[1]. > > We already have a section in our committer guide asking folks who > review PRs to make sure the needed language is in the commit that gets > merged[2]. But even if mistakes didn't happen, the vast majority of > opened PRs are closed out as stale because the contributor never > follows up after we ask them to open a JIRA. > > the tl;dr: given our current project set up our options are > > 1) Ignore that there are github PRs present (and thus look like an > inactive or poorly maintained project) > > 2) Switch to the newer "Dual Master" git setup (which will require a > nontrivial amount of tooling changes) > > 3) Push commits that close PRs (as we are doing) > > As the person who's done the vast majority of this maintenance work > over the last 3 years I'd be happy to see myself replaced with a bot. > But for reasons explained in the excellent thread on a github workflow > mentioned above, I don't think that should be a priority for the > project. > > I've only had to push 8 empty commits to keep our GitHub PR list > cleaned up[2] in this whole time. On each of those some committer as > signed off on the idea that we need a commit to master that indicates > the PR is done. > > [1]: https://s.apache.org/dBXC > > [2]: http://hbase.apache.org/book.html#_close_related_github_prs > > [3]: https://s.apache.org/hgCv > On Thu, Sep 27, 2018 at 7:19 PM 张铎(Duo Zhang) > wrote: > > > > I think record the information on jira is enough, so I prefer no empty > > commit. > > > > Andrew Purtell 于2018年9月28日周五 上午9:56写道： > > > > > There is an odd commit on master that has 0 file changes. > > > > > > commit 98b1feac771d7cc5778fb14089834c99642a3533 > > > Author: Sean Busbey > > > Date: Wed Sep 26 15:00:05 2018 -0700 > > > > > > HBASE-21241 Close stale PRs > > > > > > * Closes #86 - referenced JIRA has already been merged > > > * Closes #90 - no response from contributor for 24 days > > > > > > Signed-off-by: Peter Somogyi > > > > > > When did we agree to allow empty commits? I don't recall the > discussion. > > > For one I am opposed to this practice. Isn't the JIRA HBASE-21241 > > > sufficient to account for this activity? > > > > > > -- > > > Best regards, > > > Andrew > > > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > > decrepit hands > > >- A23, Crosstalk > > > >
Re: Are we accepting empty commits?
The commit itself is the mechanism used to close PRs. If you'd like detailed information, please read the explanation I gave on the thread "move to github flow" earlier this month[1]. We already have a section in our committer guide asking folks who review PRs to make sure the needed language is in the commit that gets merged[2]. But even if mistakes didn't happen, the vast majority of opened PRs are closed out as stale because the contributor never follows up after we ask them to open a JIRA. the tl;dr: given our current project set up our options are 1) Ignore that there are github PRs present (and thus look like an inactive or poorly maintained project) 2) Switch to the newer "Dual Master" git setup (which will require a nontrivial amount of tooling changes) 3) Push commits that close PRs (as we are doing) As the person who's done the vast majority of this maintenance work over the last 3 years I'd be happy to see myself replaced with a bot. But for reasons explained in the excellent thread on a github workflow mentioned above, I don't think that should be a priority for the project. I've only had to push 8 empty commits to keep our GitHub PR list cleaned up[2] in this whole time. On each of those some committer as signed off on the idea that we need a commit to master that indicates the PR is done. [1]: https://s.apache.org/dBXC [2]: http://hbase.apache.org/book.html#_close_related_github_prs [3]: https://s.apache.org/hgCv On Thu, Sep 27, 2018 at 7:19 PM 张铎(Duo Zhang) wrote: > > I think record the information on jira is enough, so I prefer no empty > commit. > > Andrew Purtell 于2018年9月28日周五 上午9:56写道： > > > There is an odd commit on master that has 0 file changes. > > > > commit 98b1feac771d7cc5778fb14089834c99642a3533 > > Author: Sean Busbey > > Date: Wed Sep 26 15:00:05 2018 -0700 > > > > HBASE-21241 Close stale PRs > > > > * Closes #86 - referenced JIRA has already been merged > > * Closes #90 - no response from contributor for 24 days > > > > Signed-off-by: Peter Somogyi > > > > When did we agree to allow empty commits? I don't recall the discussion. > > For one I am opposed to this practice. Isn't the JIRA HBASE-21241 > > sufficient to account for this activity? > > > > -- > > Best regards, > > Andrew > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > decrepit hands > >- A23, Crosstalk > >
