[jira] [Created] (HBASE-14901) There is duplicated code to create/manage encryption keys

[Nate Edel (JIRA)]

Nate Edel created HBASE-14901:
-

 Summary: There is duplicated code to create/manage encryption keys
 Key: HBASE-14901
 URL: https://issues.apache.org/jira/browse/HBASE-14901
 Project: HBase
  Issue Type: Bug
  Components: encryption
Affects Versions: 2.0.0
Reporter: Nate Edel
Assignee: Nate Edel
Priority: Minor
 Fix For: 2.0.0


There is duplicated code from MobUtils.createEncryptionContext in HStore, and 
there is a subset of that code in HFileReaderImpl.

Refactored key selection 
Moved both to EncryptionUtil.java
Can't figure out how to write a unit test for this, but there's no new code 
just refactoring.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening

[stack (JIRA)]

stack created HBASE-14902:
-

 Summary: Revert some of the stringency recently introduced by 
checkstyle tightening
 Key: HBASE-14902
 URL: https://issues.apache.org/jira/browse/HBASE-14902
 Project: HBase
  Issue Type: Sub-task
Reporter: stack


I think we should undo some of the plugins that were recently added to 
checkstyle. They are too much.

JavadocTagContinuationIndentationCheck is about adding indent if javadoc is two 
lines or more (javadoc tool doesn't care)

NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: 
e.g. @throws IOException needs to have text added.

NeedBracesCheck has us undoing cases where an if fits all on one line (don't 
want to start style wars but if short and fits on one line, I think its more 
readable... but I could relent on this one ).

The first two at least should go.

You ok w/ that [~appy]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14903) Table Or Region?

[JIRA]

胡托 created HBASE-14903:
--

 Summary: Table Or Region?
 Key: HBASE-14903
 URL: https://issues.apache.org/jira/browse/HBASE-14903
 Project: HBase
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.0.0
Reporter: 胡托
Priority: Blocker


 I've been reading on Latest Reference Guide and try to translated into 
Chinese!
 I think this sentence "When a table is in the process of splitting," will 
be "When a Region is in the process of splitting," on chapter 【62.2. 
hbase:meta】。
 By the way,is this document the 
latest?【http://hbase.apache.org/book.html#arch.overview】I will translate it！



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Official Docker image

[Sean Busbey]

-user@
+dev@

Apache Geode (incubating) recently went through a round-a-bout with what
can / can not be pushed to Docker Hub due to it's status as a public
consumption point for software. The short version is that we should make
sure whatever goes there (in our name) has been subject to a PMC release
vote.

On Tue, Dec 1, 2015 at 1:00 AM, Stack  wrote:

> On Sat, Nov 28, 2015 at 7:05 PM, Otis Gospodnetić <
> otis.gospodne...@gmail.com> wrote:
>
> > Hi,
> >
> > I was going through Cosmin Lehene's
> > http://www.slideshare.net/clehene/elastic-hbase-on-mesos and noticed
> there
> > was no official HBase Docker image:
> >
> >
> https://hub.docker.com/search/?q=hbase=1=0=0=0=0
> >
> > Is there a particular reason for that or it just slipped everyone's mind
> or
> > is just not important?
> >
> >
> It'd be a nice to have for sure. Our Dima has images for doing his
> integrated test runs but they are bound up with internal infrastructure. He
> is working on getting them out here to apache land. You have one lying
> about the place Otis?
> St.Ack
>
>
>
> > Thanks,
> > Otis
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection
> > Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >
>



-- 
Sean

[jira] [Created] (HBASE-14899) Create custom Streaming ReplicationEndpoint

[Cody Marcel (JIRA)]

Cody Marcel created HBASE-14899:
---

 Summary: Create custom Streaming ReplicationEndpoint
 Key: HBASE-14899
 URL: https://issues.apache.org/jira/browse/HBASE-14899
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Cody Marcel
Assignee: Cody Marcel


Create a custom Replication Endpoint that streams WALEntry directly through 
client API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14906) Improvements on FlushLargeStoresPolicy

[Yu Li (JIRA)]

Yu Li created HBASE-14906:
-

 Summary: Improvements on FlushLargeStoresPolicy
 Key: HBASE-14906
 URL: https://issues.apache.org/jira/browse/HBASE-14906
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Yu Li
Assignee: Yu Li


When checking FlushLargeStoragePolicy, found below possible improving points:

1. Currently in selectStoresToFlush, we will do the selection no matter how 
many actual families, which is not necessary for one single family

2. Default value for hbase.hregion.percolumnfamilyflush.size.lower.bound could 
not fit in all cases, and requires user to know details of the implementation 
to properly set it. We propose to use 
"hbase.hregion.memstore.flush.size/column_family_number" instead:
{noformat}
  
hbase.hregion.percolumnfamilyflush.size.lower.bound
16777216

If FlushLargeStoresPolicy is used and there are multiple column families,
then every time that we hit the total memstore limit, we find out all the
column families whose memstores exceed a "lower bound" and only flush them
while retaining the others in memory. The "lower bound" will be
"hbase.hregion.memstore.flush.size / column_family_number" by default
unless value of this property is larger than that. If none of the families
have their memstore size more than lower bound, all the memstores will be
flushed (just as usual).

  
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-14904:
-

 Summary: Mark Base[En|De]coder LimitedPrivate and fix binary 
compat issue
 Key: HBASE-14904
 URL: https://issues.apache.org/jira/browse/HBASE-14904
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4


PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary 
compatibility in Phoenix compiled with earlier versions of HBase and run agains 
later versions. 

This is one of the areas that the boundary is not clear, but it won't hurt us 
to fix it. 

The exception trace is: 
{code}
Exception in thread "main" java.lang.NoSuchFieldError: in
at 
org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
at 
org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
at 
org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
at 
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
at 
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
at 
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
at 
org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
at 
org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
at 
org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
at 
org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
at 
org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
at 
org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
at 
org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
{code}

Although {{BaseDecoder.in}} is still there, it got changed to be a class rather 
than an interface. BaseDecoder is marked Private, thus the binary compat check 
is not run at all. Not sure whether it would have caught this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14905) VerifyReplication does not honour versions option

[Vishal (JIRA)]

Vishal created HBASE-14905:
--

 Summary: VerifyReplication does not honour versions option
 Key: HBASE-14905
 URL: https://issues.apache.org/jira/browse/HBASE-14905
 Project: HBase
  Issue Type: Bug
  Components: tooling
Reporter: Vishal


source:

hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
ROW  COLUMN+CELL
   
 r1  column=f1:, timestamp=1449030102091, 
value=value1112 
 
 r1  column=f1:, timestamp=1449029774173, 
value=value1001 
 
 r1  column=f1:, timestamp=1449029709974, 
value=value1002 
 

target:
hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
ROW  COLUMN+CELL
   
 r1  column=f1:, timestamp=1449030102091, 
value=value1112 
 
 r1  column=f1:, timestamp=1449030090758, 
value=value1112 
 
 r1  column=f1:, timestamp=1449029984282, 
value=value 
 
 r1  column=f1:, timestamp=1449029774173, 
value=value1001 
 
 r1  column=f1:, timestamp=1449029709974, 
value=value1002   

/bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication 
--versions=100 1 t1

org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
GOODROWS=1

Does not show any mismatch. Ideally it should show. This is because in 
VerifyReplication Class maxVersion is not correctly set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14900) Make global config option for ReplicationEndpoint

[Cody Marcel (JIRA)]

Cody Marcel created HBASE-14900:
---

 Summary: Make global config option for ReplicationEndpoint
 Key: HBASE-14900
 URL: https://issues.apache.org/jira/browse/HBASE-14900
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Cody Marcel
Assignee: Cody Marcel
Priority: Minor


Currently ReplicationEndpoint implementations can only be configured through 
the HBase shell. We should be able to to use a property in the hbase-site.xml 
to globally set an alternate default. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Successful: HBase Generate Website

[Apache Jenkins Server]

Build status: Successful

If successful, the website and docs have been generated. If failed, skip to the 
bottom of this email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site:

  wget -O- 
https://builds.apache.org/job/hbase_generate_website/50/artifact/website.patch.zip
 | funzip > 7979ac46cce36f21033f8ed03c8d0dd5fddde005.patch
  git fetch
  git checkout -b asf-site-7979ac46cce36f21033f8ed03c8d0dd5fddde005 
origin/asf-site
  git am 7979ac46cce36f21033f8ed03c8d0dd5fddde005.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-7979ac46cce36f21033f8ed03c8d0dd5fddde005 branch, and you can review 
the differences by running:

  git diff origin/asf-site

When you are satisfied, publish your changes to origin/asf-site using this 
command:

  git push origin asf-site-7979ac46cce36f21033f8ed03c8d0dd5fddde005:asf-site

Changes take a couple of minutes to be propagated. You can then remove your 
asf-site-7979ac46cce36f21033f8ed03c8d0dd5fddde005 branch:

  git checkout master && git branch -d 
asf-site-7979ac46cce36f21033f8ed03c8d0dd5fddde005



If failed, see https://builds.apache.org/job/hbase_generate_website/50/console

[jira] [Reopened] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-14223:
---

Reopening. Can you take a look at the fails over in 1.2 [~enis]? Since this 
went in, TestRegionRebalancing on 1.7 builds is failing every build:

https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/414/jdk=latest1.7,label=Hadoop/testReport/TEST-org.apache.hadoop.hbase.TestRegionRebalancing/xml/_init_/history/

It passed when it went in originally on 1.3 but then failed on subsequent build:

https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/406/

Punt back to me and reresolve this issue if you don't think related. Thanks.



> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch, 
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch, 
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch, 
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed. 
> The last WAL file just sits there in the RS WAL directory. If RS stops 
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for 
> meta is not cleaned. It is also not split (which is correct) since master 
> determines that the RS no longer hosts meta at the time of RS abort. 
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}} 
> directories left uncleaned: 
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls 
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x   - hbase hadoop  0 2015-06-05 01:14 
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x   - hbase hadoop  0 2015-06-05 07:54 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x   - hbase hadoop  0 2015-06-05 09:28 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x   - hbase hadoop  0 2015-06-05 10:01 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta: 
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r--   3 hbase hadoop 201608 2015-06-05 03:15 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r--   3 hbase hadoop  44420 2015-06-05 04:36 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time: 
> {code}
> 2015-06-05 03:14:28,692 INFO  [PostOpenDeployTasks:1588230740] 
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper 
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO  
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed 
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created: 
> {code}
> 2015-06-05 03:15:11,707 INFO  
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog: 
> Rolled WAL 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
>  with entries=385, filesize=196.88 KB; new WAL 
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files: 
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075 
> INFO  [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0] 
> master.SplitLogManager: started splitting 2 logs in 
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
>  for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300 
> INFO  [main-EventThread] wal.WALSplitter: Archived processed log 
>