[jira] [Created] (HBASE-15843) Replace RegionState.getRegionInTransition() Map with a Set

[Matteo Bertozzi (JIRA)]

Matteo Bertozzi created HBASE-15843:
---

 Summary: Replace RegionState.getRegionInTransition() Map with a Set
 Key: HBASE-15843
 URL: https://issues.apache.org/jira/browse/HBASE-15843
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Affects Versions: 1.2.1, 2.0.0, 1.3.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
 Fix For: 2.0.0


RegionState.getRegionInTransition() is always used as a Set.
replace the Map with a Set, avoid some allocation and extra code.

also ClusterStatus.RegionInTransition has duplicated information.
The spec field contains the regionName (not encoded). 
but we have the same info as part of the region_state with the HRegionInfo 
serialized.
unfortunately I don't think we can get rid of 'spec' that being a required 
field.
{noformat}
message RegionInTransition {
  required RegionSpecifier spec = 1;
  required RegionState region_state = 2;
}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Using the new 'help wanted' tool from comdev

[Daniel Vimont]

Andrew -- It's encouraging to hear about your positive take on the overall
situation at Salesforce. I agree that one of the questions to consider is
that of how to encourage people of different genders and ethnicities (and
perhaps even ages!) to join us. To me, the easiest way to get an inkling of
how to do that is to look around and ask ourselves "what drew US into this
project"? It seems to me that what made US want to be here might similarly
entice OTHERS to want to join. That's why, in my original contribution to
this conversation, I stated why *I* am here. Unfortunately, I don't know
that there are any cookbook answers regarding how to recruit for HBase (and
other hadoop-related) technologies in specific workplaces. In workplaces
where people are *assigned* to work with HBase, the person to talk with
would probably be the assigner; in places where people more freely *choose*
their own roles (sounds like Salesforce is in this category), then focused
one-on-one evangelizing/recruiting might be appropriate.

But I imagine that getting people initially intrigued at the prospect of
working with leading-edge technology is likely much easier than asking them
to join a team in which they are to play a Jackie Robinson-like role. A few
days ago, I traded some messages with a woman who recently left the entire
IT field a few months back (and had, in fact, most recently been working on
teams using hadoop-ecosystem technologies). She said she LOVED what she was
doing, but that 20 years of (1) often being the only woman in the room, (2)
having to constantly prove herself "worthy" of even being in the room, and
(3) having to deal with explicit threats, doxing, etc. in online work
situations -- all that wore her down until she finally quit a few months
ago.

So now, Andrew, to get back to your question of what's keeping non-male
people from opting in to your HBase team, all I can do is guess that
perhaps nobody wants to be the sole woman sitting in the roomful of guys
(playing that potentially very uncomfortable "Jackie Robinson" role). But
your best bet is to approach some of the women that you think might make
the best candidates for your HBase team and simply ask them. That's sure to
get you much better data than my wild-assed guesses!!

(If it turns out that my guess is correct and that nobody feels comfortable
being that trail-blazing "first woman" on the team, then try to arrange for
three or more ladies to be transferred in at the same time. After that, the
challenge would be to make sure that the environment is safe and supportive
for them, so they'll stick around. If that can be maintained, then I would
hope that they would become your next team of evangelists, bringing others
in and continuing a virtuous and organic cycle of diversification and
inclusion.)

On Tue, May 17, 2016 at 2:30 AM, Andrew Purtell 
wrote:

> Salesforce has a diverse workforce. I went there directly from Intel and
> have had occasion to participate in many more mixed gender work situations
> since moving over. I could go on about how Salesforce has a great culture
> but I'll spare you the tangent. (Smile) However those of us working in the
> various storage and compute platform groups here still have to make the
> case we're a good team to work on in an open internal hiring market. There
> are some direct to position external recruitments but they're in the
> minority. Two observations:
> - Even with an overall diverse workforce in our org storage is
> overwhelming staffed with males. I can count females on one hand and none
> are working on the HBase team. So what is it about Storage (or the people
> working on it) that do not make or counter an appeal?
> - If we want to increase team diversity - and we do, internally and out in
> the greater community - then we should be actively recruiting. How should
> that pitch go?
>
>
> > On May 16, 2016, at 12:49 AM, Daniel Vimont 
> wrote:
> >
> > Stack & Sean -- Thanks for your positive responses to all this (and the
> > intriguing Mother Jones link).
> >
> > Will any of the HBaseCon2016 events be live-streamed? (I can't make it --
> > an Osaka/SF round-trip was not in the budget this spring!)
> >
> >> On Sun, May 15, 2016 at 2:05 PM, Stack  wrote:
> >>
> >> On Sat, May 14, 2016 at 7:18 PM, Daniel Vimont 
> >> wrote:
> >>
> >>> The first step in working through a problem is acknowledging that there
> >> IS
> >>> a problem, so in that respect, this conversation is going well so far!!
> >>> Here are some thoughts (some of which are barely half-baked, so pardon
> >> the
> >>> "raw dough").
> >>>
> >>> SHORT-TERM BRAINSTORMING
> >>> I know that HBaseCon2016 is very close, and all the speaking slots and
> >>> topic sessions are locked into place, but I do have a few suggestions:
> >>>
> >>> (1) At *several* points (not just once!) throughout the plenary
> sessions,
> >>> make a statement that this conversation has come up on hbase-dev, and
> >> it's
> >>> an issue that you think is worthy of our 

Re: Smart Flaky Handler

[Stack]

Sweet!

On Mon, May 16, 2016 at 4:38 PM, Apekshit Sharma  wrote:

> This mail is to introduce the work to tackle the flaky tests in our build.
>
> *Why is it important?*
> - Our build history sucks, last 175 post-commit runs failed. We need to
> make it useful.
> - To better understand our code’s testing status, more importantly it’s
> weak points.
> - We know those 2-3 tests which keep failing every now and then, but not
> those ~10 nasty ones which fail like 1 out of 50 times, and screw our build.
> - This isn’t something that can be done manually on a daily basis. We need
> automation.
>
> *Changes made so far:*
> Code changes: HBASE-15839
>   (Umbrella issue)
>
> *Jenkins changes:*
>
>
> [Diagram link:
> https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%204.02.46%20PM.png
> ]
> ​
> *(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent builds of
> post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see below) to
> find flaky tests. Frequency of run determines how fast we catch test
> regressions. So if we run it every 4 hours, any test which started failing
> in post-commit job (TRUNK_matrix) in last 4 hour will be blacklisted.
>
> *(new job) HBase-Flaky-Tests*: This job runs only the flaky tests. The
> aim is to run this job back-to-back to collect as many runs as we can.
> Higher the run rate, the better will be our system at catching the flaky
> tests. We currently run it hourly. so we’ll be able to keep track of flaky
> tests with ~5% failure rate or more.
>
> *Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these flaky
> tests.
>
>
> *So what if a bad commit makes a good test bad?*
> Since the test is not bad, it’ll run in next post-commit and will fail.
> Next run of HBase-Find-Flaky-Tests will  pick it up and blacklist it.
> Blacklisting will help keep the post-commit job and more importantly
> pre-commit job clean, a problem we face quite often.
>
> *Are we just tucking away are shit?*
> Nope, this will help us:
> - first, Maintain a list of bad test (we lack that today).
> - second, make our build greener to the point that a failed/red build is
> something we worry about seriously.
>
> Once we are confident that the system is working fine, we’ll setup up
> HBase-Find-Flaky-Tests job to send reports to dev@hbase so that devs know
> about the bad tests. If it remains hidden somewhere in a jenkins job’s
> archive, it’s unlike that we’ll actively work on getting them fixed :).
>
> I'll keep posting further updates on this thread.
>
> -- Appy
>

Smart Flaky Handler

[Apekshit Sharma]

 This mail is to introduce the work to tackle the flaky tests in our build.

*Why is it important?*
- Our build history sucks, last 175 post-commit runs failed. We need to
make it useful.
- To better understand our code’s testing status, more importantly it’s
weak points.
- We know those 2-3 tests which keep failing every now and then, but not
those ~10 nasty ones which fail like 1 out of 50 times, and screw our build.
- This isn’t something that can be done manually on a daily basis. We need
automation.

*Changes made so far:*
Code changes: HBASE-15839
  (Umbrella issue)

*Jenkins changes:*


[Diagram link:
https://issues.apache.org/jira/secure/attachment/12804292/Screen%20Shot%202016-05-16%20at%204.02.46%20PM.png
]
​
*(new job) HBase-Find-Flaky-Tests*: Gets test reports of recent builds of
post-commit job (TRUNK_matrix) and HBase-Flaky-Tests job (see below) to
find flaky tests. Frequency of run determines how fast we catch test
regressions. So if we run it every 4 hours, any test which started failing
in post-commit job (TRUNK_matrix) in last 4 hour will be blacklisted.

*(new job) HBase-Flaky-Tests*: This job runs only the flaky tests. The aim
is to run this job back-to-back to collect as many runs as we can. Higher
the run rate, the better will be our system at catching the flaky tests. We
currently run it hourly. so we’ll be able to keep track of flaky tests with
~5% failure rate or more.

*Post-commit (TRUNK_matrix) and pre-commit jobs*: Exclude these flaky tests.


*So what if a bad commit makes a good test bad?*
Since the test is not bad, it’ll run in next post-commit and will fail.
Next run of HBase-Find-Flaky-Tests will  pick it up and blacklist it.
Blacklisting will help keep the post-commit job and more importantly
pre-commit job clean, a problem we face quite often.

*Are we just tucking away are shit?*
Nope, this will help us:
- first, Maintain a list of bad test (we lack that today).
- second, make our build greener to the point that a failed/red build is
something we worry about seriously.

Once we are confident that the system is working fine, we’ll setup up
HBase-Find-Flaky-Tests job to send reports to dev@hbase so that devs know
about the bad tests. If it remains hidden somewhere in a jenkins job’s
archive, it’s unlike that we’ll actively work on getting them fixed :).

I'll keep posting further updates on this thread.

-- Appy

[jira] [Created] (HBASE-15842) SnapshotInfo should display ownership information

[Ted Yu (JIRA)]

Ted Yu created HBASE-15842:
--

 Summary: SnapshotInfo should display ownership information
 Key: HBASE-15842
 URL: https://issues.apache.org/jira/browse/HBASE-15842
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu


Currently SnapshotInfo doesn't show snapshot owner:
{code}
hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot 
snapshot_table_qm0uxvk19x -stats -schema
...
Snapshot Info

   Name: snapshot_table_qm0uxvk19x
   Type: FLUSH
  Table: table_qm0uxvk19x
 Format: 2
Created: 2016-05-16T20:54:08
...
{code}
This JIRA is to add ownership information to the display.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-15651) Add report-flakies.py to use jenkins api to get failing tests

[Appy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Appy resolved HBASE-15651.
--
Resolution: Fixed

> Add report-flakies.py to use jenkins api to get failing tests
> -
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Appy
>Assignee: Appy
> Fix For: master
>
> Attachments: HBASE-15651-master-v2.patch, 
> HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch, 
> HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the 
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] 
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We 
> can either directly pull these results into other jobs using curl on this 
> job's artifacts, or commit the list of flaky test to repo (idk if it's 
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and 
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using 
> --exclude maven flag. Someone familiar with yetus might be able to do it 
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Using Secure HTTP (HTTPS) for the Web UI

[Talat Uyarer]

Hi,

Is there any way to learn what does cluster use either http or https
in ui without checking configuration object ?

Thanks
--
Talat

[jira] [Created] (HBASE-15841) Performance Evaluation tool total rows may not be set correctly

[Jerry He (JIRA)]

Jerry He created HBASE-15841:


 Summary: Performance Evaluation tool total rows may not be set 
correctly
 Key: HBASE-15841
 URL: https://issues.apache.org/jira/browse/HBASE-15841
 Project: HBase
  Issue Type: Bug
Reporter: Jerry He
Priority: Minor


Carried my comment on HBASE-15403 to here:

Recently when I ran PerformanceEvaluation, I did notice some problem with the 
number of rows.

hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestTable1 
randomWrite 1
hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestTable5 
randomWrite 5
hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestTable10 
randomWrite 10
hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestTable10 
randomWrite 20

All produced similar number of rows, and on the file system, they look like in 
similar size as well:

hadoop fs -du -h /apps/hbase/data/data/default
786.5 M /apps/hbase/data/data/default/TestTable1
786.0 M /apps/hbase/data/data/default/TestTable10
782.0 M /apps/hbase/data/data/default/TestTable20
713.4 M /apps/hbase/data/data/default/TestTable5

HBase is 1.2.0. Looks like a regression somewhere.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-15651) Track our flaky tests and use them to improve our build environment

[Appy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Appy reopened HBASE-15651:
--

Reopening to edit title and re-organize the jiras related to flaky work 
together.

> Track our flaky tests and use them to improve our build environment
> ---
>
> Key: HBASE-15651
> URL: https://issues.apache.org/jira/browse/HBASE-15651
> Project: HBase
>  Issue Type: New Feature
>Reporter: Appy
>Assignee: Appy
> Fix For: master
>
> Attachments: HBASE-15651-master-v2.patch, 
> HBASE-15651-master-v3.patch, HBASE-15651-master-v4.patch, 
> HBASE-15651-master.patch, flakies.py, hbase-personality-master.patch
>
>
> So i have written this simple script (attached) which looks at history of the 
> [post commit build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] 
> and outputs a list of flaky tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We 
> can either directly pull these results into other jobs using curl on this 
> job's artifacts, or commit the list of flaky test to repo (idk if it's 
> possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and 
> *flaky-tests* job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using 
> --exclude maven flag. Someone familiar with yetus might be able to do it 
> easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15840) WAL.proto compilation broken for cpp

[Elliott Clark (JIRA)]

Elliott Clark created HBASE-15840:
-

 Summary: WAL.proto compilation broken for cpp
 Key: HBASE-15840
 URL: https://issues.apache.org/jira/browse/HBASE-15840
 Project: HBase
  Issue Type: Bug
Reporter: Elliott Clark
Assignee: Elliott Clark


HBASE-15669 seems to have broken generating code for WAL.proto in cpp.
{code}
In file included from buck-out/gen/if/WAL.pb.cc/WAL.pb.cc:5:0:
buck-out/gen/if/WAL.pb.cc/WAL.pb.h:1103:37: error: 'google::protobuf::uint64 
hbase::pb::StoreDescriptor::store_file_size() const' cannot be overloaded
   inline ::google::protobuf::uint64 store_file_size() const;
 ^
buck-out/gen/if/WAL.pb.cc/WAL.pb.h:1084:14: error: with 'int 
hbase::pb::StoreDescriptor::store_file_size() const'
   inline int store_file_size() const;
  ^
In file included from buck-out/gen/if/WAL.pb.cc/WAL.pb.cc:5:0:
buck-out/gen/if/WAL.pb.cc/WAL.pb.h:3527:35: error: prototype for 
'google::protobuf::uint64 hbase::pb::StoreDescriptor::store_file_size() const' 
does not match any in class 'hbase::pb::StoreDescriptor'
 inline ::google::protobuf::uint64 StoreDescriptor::store_file_size() const {
   ^
buck-out/gen/if/WAL.pb.cc/WAL.pb.h:3460:12: error: candidate is: int 
hbase::pb::StoreDescriptor::store_file_size() const
 inline int StoreDescriptor::store_file_size() const {
^

BUILD FAILED: //if:if#compile-WAL.pb.cc.oa0460dda,default failed with exit code 
1:
c++-cpp-output compile
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15839) Track our flaky tests and use them to improve our build environment

[Appy (JIRA)]

Appy created HBASE-15839:


 Summary: Track our flaky tests and use them to improve our build 
environment
 Key: HBASE-15839
 URL: https://issues.apache.org/jira/browse/HBASE-15839
 Project: HBase
  Issue Type: Improvement
Reporter: Appy
Assignee: Appy






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15838) Fix broken external links across hbase.apache.org website and docs

[Misty Stanley-Jones (JIRA)]

Misty Stanley-Jones created HBASE-15838:
---

 Summary: Fix broken external links across hbase.apache.org website 
and docs
 Key: HBASE-15838
 URL: https://issues.apache.org/jira/browse/HBASE-15838
 Project: HBase
  Issue Type: Bug
Reporter: Misty Stanley-Jones


See 
https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/43/artifact/link_report/urlfailX.html.
 One hint is that all the links that reference www.cloudera.com/blog should be 
changed to blog.cloudera.com/blog.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Successful: hbase.apache.org HTML Checker

[Apache Jenkins Server]

Successful

If successful, the HTML and link-checking report for http://hbase.apache.org is 
available at 
https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/43/artifact/link_report/index.html.

If failed, see 
https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/43/console.

[jira] [Created] (HBASE-15837) More gracefully handle a negative memstoreSize

[Josh Elser (JIRA)]

Josh Elser created HBASE-15837:
--

 Summary: More gracefully handle a negative memstoreSize
 Key: HBASE-15837
 URL: https://issues.apache.org/jira/browse/HBASE-15837
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0


Over in PHOENIX-2883, I've been trying to figure out how to track down the root 
cause of an issue we were seeing where a negative memstoreSize was ultimately 
causing an RS to abort. The tl;dr version is

* Something causes memstoreSize to be negative (not sure what is doing this yet)
* All subsequent flushes short-circuit and don't run because they think there 
is no data to flush
* The region is eventually closed (commonly, for a move).
* A final flush is attempted on each store before closing (which also 
short-circuit for the same reason), leaving unflushed data in each store.
* The sanity check that each store's size is zero fails and the RS aborts.

I have a little patch which I think should improve our failure case around 
this, preventing the RS abort safely (forcing a flush when memstoreSize is 
negative) and logging a calltrace when an update to memstoreSize make it 
negative (to find culprits in the future).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Using the new 'help wanted' tool from comdev

[Andrew Purtell]

Salesforce has a diverse workforce. I went there directly from Intel and have 
had occasion to participate in many more mixed gender work situations since 
moving over. I could go on about how Salesforce has a great culture but I'll 
spare you the tangent. (Smile) However those of us working in the various 
storage and compute platform groups here still have to make the case we're a 
good team to work on in an open internal hiring market. There are some direct 
to position external recruitments but they're in the minority. Two observations:
- Even with an overall diverse workforce in our org storage is overwhelming 
staffed with males. I can count females on one hand and none are working on the 
HBase team. So what is it about Storage (or the people working on it) that do 
not make or counter an appeal? 
- If we want to increase team diversity - and we do, internally and out in the 
greater community - then we should be actively recruiting. How should that 
pitch go? 


> On May 16, 2016, at 12:49 AM, Daniel Vimont  wrote:
> 
> Stack & Sean -- Thanks for your positive responses to all this (and the
> intriguing Mother Jones link).
> 
> Will any of the HBaseCon2016 events be live-streamed? (I can't make it --
> an Osaka/SF round-trip was not in the budget this spring!)
> 
>> On Sun, May 15, 2016 at 2:05 PM, Stack  wrote:
>> 
>> On Sat, May 14, 2016 at 7:18 PM, Daniel Vimont 
>> wrote:
>> 
>>> The first step in working through a problem is acknowledging that there
>> IS
>>> a problem, so in that respect, this conversation is going well so far!!
>>> Here are some thoughts (some of which are barely half-baked, so pardon
>> the
>>> "raw dough").
>>> 
>>> SHORT-TERM BRAINSTORMING
>>> I know that HBaseCon2016 is very close, and all the speaking slots and
>>> topic sessions are locked into place, but I do have a few suggestions:
>>> 
>>> (1) At *several* points (not just once!) throughout the plenary sessions,
>>> make a statement that this conversation has come up on hbase-dev, and
>> it's
>>> an issue that you think is worthy of our attention and discussion. (I
>> think
>>> we're just at the beginnings of a very long conversation, but just
>>> announcing that the conversation is underway and inviting people into it
>>> seems a good way to start.)
>> 
>> We can do that.
>> 
>> 
>>> (2) I see that Intel is sending four representatives to make
>> presentations
>>> (two women and two men!). Intel seems to be at the forefront of
>>> diversification efforts in the broader IT community, having set its goal
>>> for "full representation" in its workforce by 2020. If these Intel
>>> presenters feel comfortable, it would be great to devote a few minutes in
>>> one of the plenary sessions to allow them to informally talk about how
>> the
>>> Intel z is going from their
>>> perspective, and whether that offers any lessons that might be applied in
>>> the Hadoop/HBase ecosystems.
>> 
>> Let me ask the Intel'ers who are coming to the conference.
>> 
>> 
>>> (3) For HBaseCon2017, devote complete sessions to the issue of
>>> diversification. (Perhaps invite Intel's Chief Diversity Officer,
>> Danielle
>>> Brown, to lead up some proactive discussions.)
>> Yes. I like this suggestion.
>> 
>> 
>> 
>>> LONGER-TERM STRATEGIZING
>>> Stepping back to the generic questions of how and why to "recruit" new
>>> contributors, perhaps this is where my "newbie" eyes can assist in
>> taking a
>>> fresh look at things, beginning with a big lead-in question:
>>> 
>>> The current "staff" of the HBase project: how were we "recruited", and
>> why
>>> are we here?
>>> 
>>> When I look at the contributors list, I see what I expect to see: Almost
>>> all of the contributors are employees of companies that are the biggest
>>> stakeholders in the HBase ecosystem, with the biggest number coming from
>>> those companies that are existentially bound to the HBase/Hadoop
>>> ecosystems, and a smaller number of contributors coming from companies
>> that
>>> make strategically vital use of HBase in their operations. My assumption
>> is
>>> that these companies are not magnanimous philanthropists, they're just
>>> (quite appropriately) looking after their own vital interests. Bottom
>> line:
>>> this majority of contributors are *paid* to be here; their employers
>>> "allow" them to spend a percentage (or maybe in some cases, all) of their
>>> working hours tending to the care and feeding of the open-source golden
>>> goose that keeps their proprietary enterprises going.
>>> 
>>> And then there are people like me (likely in much smaller numbers) who
>> are
>>> here to beef up their expertise in current technologies in a way that
>> also
>>> gives them a publicly-displayed "portfolio" of work with which to
>>> subsequently seek out paid engagements (of either the contract or
>>> employment variety). I'm sure I don't fully buy into the prevailing
>> wisdom
>>> that "code is the new resume" and "just put your stuff out there on
>> GitHub
>>> -- WE'LL find YOU!!"; 

[jira] [Created] (HBASE-15836) The -set parameter in "hbase backup" should be in the help text

[Ted Yu (JIRA)]

Ted Yu created HBASE-15836:
--

 Summary: The -set parameter in "hbase backup" should be in the 
help text
 Key: HBASE-15836
 URL: https://issues.apache.org/jira/browse/HBASE-15836
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu


[~cartershanklin] noticed the following:
{code}
$ hbase backup create
ERROR: wrong number of arguments
Usage: hbase backup create   [tables] [-s name] 
[-convert] [-silent] [-w workers][-b bandwith]
 type  "full" to create a full backup image;
   "incremental" to create an incremental backup image
  backup_root_path   The full root path to store the backup image,
the prefix can be hdfs, webhdfs or gpfs
 Options:
  tables  If no tables ("") are specified, all tables are backed up. 
Otherwise it is a
   comma separated list of tables.
 -s name Use the specified snapshot for full backup
 -convertFor an incremental backup, convert WAL files to HFiles
 -w  number of parallel workers.
 -b  bandwith per one worker (in MB sec)
{code}
A working backup command may look like the following (assuming the set named 
green exists):
{code}
sudo -u hbase hbase backup create full hdfs://ip:8020/tmp/backup -set green
{code}
Help text for backup command should be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Successful: HBase Generate Website

[Apache Jenkins Server]

Build status: Successful

If successful, the website and docs have been generated. If failed, skip to the 
bottom of this email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/230/artifact/website.patch.zip
 | funzip > e0aff109018918514c8f27a8a90159e695e8b542.patch
  git fetch
  git checkout -b asf-site-e0aff109018918514c8f27a8a90159e695e8b542 
origin/asf-site
  git am --whitespace=fix e0aff109018918514c8f27a8a90159e695e8b542.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-e0aff109018918514c8f27a8a90159e695e8b542 branch, and you can review 
the differences by running:

  git diff origin/asf-site

There are lots of spurious changes, such as timestamps and CSS styles in 
tables. To see a list of files that have been added, deleted, renamed, changed 
type, or are otherwise interesting, use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using this 
command:

  git push origin asf-site-e0aff109018918514c8f27a8a90159e695e8b542:asf-site

Changes take a couple of minutes to be propagated. You can then remove your 
asf-site-e0aff109018918514c8f27a8a90159e695e8b542 branch:

  git checkout asf-site && git branch -d 
asf-site-e0aff109018918514c8f27a8a90159e695e8b542



If failed, see https://builds.apache.org/job/hbase_generate_website/230/console

Re: ASF jira permissions

[Daniel Vimont]

Regarding the new tighter, spam-prevention permissions: As a basic
contributor, do I now NOT have the ability to assign a JIRA entry to
myself? (I just opened HBASE-15835, and don't seem to be able to assign it
to myself.)

THANKS!

On Mon, May 16, 2016 at 1:34 PM, Mikhail Antonov 
wrote:

> Oh, I see. Thanks for clarification. And yeah, I see I can now do
> things I don't remember being able to do before, like creating new
> versions. Thanks!
>
> Mikhail
>
> On 5/15/16, Andrew Purtell  wrote:
> > The issue is we didn't have you in the proper permissions group. You were
> > able to do those things because we had the 'jira-users' group (everybody
> > with a jira account) authorized for our contributors role. We had to do
> away
> > with that as a counter spam measure.
> >
> >
> >> On May 15, 2016, at 9:26 PM, Mikhail Antonov 
> >> wrote:
> >>
> >> Thanks Stack, my permissions are back. Probably some jira glitch, or
> >> might
> >> be artifact of anti-spam work..
> >>
> >> -Mikhail
> >>
> >>> On Sun, May 15, 2016 at 9:23 PM, Stack  wrote:
> >>>
> >>> Try now. You didn't seem to be in the committers group. I added you.
> >>> St.Ack
> >>>
> >>> On Sun, May 15, 2016 at 9:20 PM, Mikhail Antonov 
> >>> wrote:
> >>>
>  Seems like after recent issues with ASF jira I've lost some
> permissions
>  I
>  had before, for example, I can't edit fix versions, components and
>  labels
>  on jiras. Does anyone else see such a problem?
> 
>  -Mikhail
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Michael Antonov
> >
>
>
> --
> Thanks,
> Michael Antonov
>

[jira] [Created] (HBASE-15835) HBaseTestingUtility#startMiniCluster throws "HMasterAddress already in use" RuntimeException when a local instance of HBase is running

[Daniel Vimont (JIRA)]

Daniel Vimont created HBASE-15835:
-

 Summary: HBaseTestingUtility#startMiniCluster throws 
"HMasterAddress already in use" RuntimeException when a local instance of HBase 
is running
 Key: HBASE-15835
 URL: https://issues.apache.org/jira/browse/HBASE-15835
 Project: HBase
  Issue Type: Bug
  Components: API
Affects Versions: 2.0.0
Reporter: Daniel Vimont
 Fix For: 2.0.0


When a MiniCluster is being started with the 
{{HBaseTestUtility#startMiniCluster}} method (most typically in the context of 
JUnit testing), if a local HBase instance is already running (or for that 
matter, another thread with another MiniCluster is already running), the 
startup will fail with a RuntimeException saying "HMasterAddress already in 
use", referring explicitly to contention for the same default master info port 
(16010).

This problem most recently came up in conjunction with HBASE-14876 and its 
sub-JIRAs (development of new HBase-oriented Maven archetypes), but this is 
apparently a known issue to veteran developers, who tend to set up the 
@BeforeClass sections of their test modules with code similar to the following:
{code}
UTIL = HBaseTestingUtility.createLocalHTU();
// disable UI's on test cluster.
UTIL.getConfiguration().setInt("hbase.master.info.port", -1);
UTIL.getConfiguration().setInt("hbase.regionserver.info.port", -1);
UTIL.startMiniCluster();
{code}

A comprehensive solution modeled on this should be put directly into 
HBaseTestUtility's main constructor, using one of the following options:

OPTION 1 (always force random port assignment):
{code}
this.getConfiguration().setInt(HConstants.MASTER_INFO_PORT, -1);
this.getConfiguration().setInt(HConstants.REGIONSERVER_PORT, -1);
{code}

OPTION 2 (always force random port assignment if user has not explicitly 
defined alternate port):
{code}
Configuration conf = this.getConfiguration();
if (conf.getInt(HConstants.MASTER_INFO_PORT, 
HConstants.DEFAULT_MASTER_INFOPORT)
== HConstants.DEFAULT_MASTER_INFOPORT) {
  conf.setInt(HConstants.MASTER_INFO_PORT, -1);
}
if (conf.getInt(HConstants.REGIONSERVER_PORT, 
HConstants.DEFAULT_REGIONSERVER_PORT)
== HConstants.DEFAULT_REGIONSERVER_PORT) {
  conf.setInt(HConstants.REGIONSERVER_PORT, -1);
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Using the new 'help wanted' tool from comdev

[Daniel Vimont]

Stack & Sean -- Thanks for your positive responses to all this (and the
intriguing Mother Jones link).

Will any of the HBaseCon2016 events be live-streamed? (I can't make it --
an Osaka/SF round-trip was not in the budget this spring!)

On Sun, May 15, 2016 at 2:05 PM, Stack  wrote:

> On Sat, May 14, 2016 at 7:18 PM, Daniel Vimont 
> wrote:
>
> > The first step in working through a problem is acknowledging that there
> IS
> > a problem, so in that respect, this conversation is going well so far!!
> > Here are some thoughts (some of which are barely half-baked, so pardon
> the
> > "raw dough").
> >
> > SHORT-TERM BRAINSTORMING
> > I know that HBaseCon2016 is very close, and all the speaking slots and
> > topic sessions are locked into place, but I do have a few suggestions:
> >
> > (1) At *several* points (not just once!) throughout the plenary sessions,
> > make a statement that this conversation has come up on hbase-dev, and
> it's
> > an issue that you think is worthy of our attention and discussion. (I
> think
> > we're just at the beginnings of a very long conversation, but just
> > announcing that the conversation is underway and inviting people into it
> > seems a good way to start.)
> >
>
> We can do that.
>
>
> > (2) I see that Intel is sending four representatives to make
> presentations
> > (two women and two men!). Intel seems to be at the forefront of
> > diversification efforts in the broader IT community, having set its goal
> > for "full representation" in its workforce by 2020. If these Intel
> > presenters feel comfortable, it would be great to devote a few minutes in
> > one of the plenary sessions to allow them to informally talk about how
> the
> > Intel z is going from their
> > perspective, and whether that offers any lessons that might be applied in
> > the Hadoop/HBase ecosystems.
> >
>
> Let me ask the Intel'ers who are coming to the conference.
>
>
> > (3) For HBaseCon2017, devote complete sessions to the issue of
> > diversification. (Perhaps invite Intel's Chief Diversity Officer,
> Danielle
> > Brown, to lead up some proactive discussions.)
> >
> >
> Yes. I like this suggestion.
>
>
>
> > LONGER-TERM STRATEGIZING
> > Stepping back to the generic questions of how and why to "recruit" new
> > contributors, perhaps this is where my "newbie" eyes can assist in
> taking a
> > fresh look at things, beginning with a big lead-in question:
> >
> > The current "staff" of the HBase project: how were we "recruited", and
> why
> > are we here?
> >
> > When I look at the contributors list, I see what I expect to see: Almost
> > all of the contributors are employees of companies that are the biggest
> > stakeholders in the HBase ecosystem, with the biggest number coming from
> > those companies that are existentially bound to the HBase/Hadoop
> > ecosystems, and a smaller number of contributors coming from companies
> that
> > make strategically vital use of HBase in their operations. My assumption
> is
> > that these companies are not magnanimous philanthropists, they're just
> > (quite appropriately) looking after their own vital interests. Bottom
> line:
> > this majority of contributors are *paid* to be here; their employers
> > "allow" them to spend a percentage (or maybe in some cases, all) of their
> > working hours tending to the care and feeding of the open-source golden
> > goose that keeps their proprietary enterprises going.
> >
> > And then there are people like me (likely in much smaller numbers) who
> are
> > here to beef up their expertise in current technologies in a way that
> also
> > gives them a publicly-displayed "portfolio" of work with which to
> > subsequently seek out paid engagements (of either the contract or
> > employment variety). I'm sure I don't fully buy into the prevailing
> wisdom
> > that "code is the new resume" and "just put your stuff out there on
> GitHub
> > -- WE'LL find YOU!!"; nonetheless, for the last several months I've been
> > setting aside my doubts and building just such a "portfolio".
> >
> >
> The above is a fair assessment I'd say.
>
>
> > So, if my take on the current "staffing" situation is approximately
> > correct, it suggests to me that the quickest path to "full
> representation"
> > in the HBase project is initially through conversations with the Human
> > Resources chiefs and the CEOs of the corporations that provide the
> majority
> > of this project's "staff". What initiatives are underway in these firms
> (1)
> > to recruit a diverse workforce and (2) to assure that once members of
> > underrepresented minorities are brought in, that they find themselves in
> a
> > safe, supportive environment in which they want to stay? Finally, if good
> > initiatives are already underway, what might be done to hasten the
> "trickle
> > down" from these initiatives into a more diversely-staffed HBase project?
> >
> >
>
> There was a call-to-arms a while back that echoed around the valley (Here
> is one origin story [1] that references this i

[jira] [Created] (HBASE-15834) Correct Bloom filter documentation in section 96.4 of Reference Guide

[li xiang (JIRA)]

li xiang created HBASE-15834:


 Summary: Correct Bloom filter documentation in section 96.4 of 
Reference Guide
 Key: HBASE-15834
 URL: https://issues.apache.org/jira/browse/HBASE-15834
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: li xiang
Priority: Minor


In section 96.4, the second paragraph from the bottom
{code}
Since HBase 0.96, row-based Bloom filters are enabled by default. (HBASE-)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)