[jira] [Commented] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257085#comment-13257085
 ] 

Enis Soztutar commented on HBASE-5349:
--

I have been thinking about this, and I think we can have a shot at a simple 
implementation. Let me summarize what I have in mind before starting the 
implementation: 
Goals: 
 - Provide min - max heap percentages for block cache (memstore kind of has 
it). I think we should keep max-min sanity bounds, and if they are equal, 
disable auto-tuning. 
 - enable optimizing the available memory for adaptive workloads (mostly writes 
during the day, a lot of reads once MR job starts, etc). For example, when a 
large write job is started after ~10 minutes, region servers should tune for 
write workload. 
Non-goals: 
 - find the optimum mem-utilization algorithm
 - introduce a bunch of other parameters, to get rid of the current ones
 - make it very experimental so that nobody enables it in production. 

Ideally, to optimize the usage of the available memory, we should predict the 
future workload (possibly from past workload), and devise a model capturing all 
the costs associated with block cache hits / misses, flushes, compactions, etc. 
But this model will be very complex to do it properly.

I have checked Hypertable's implementation, and it seems that they check 
whether the load is read/write heavy by some hard coded values for the 
counters, and increment/decrement the mem limits, much like what Zhihong 
proposes above. I also want to start with something similar. 

Implementation layer: 
 - Currently global memstore limit is a soft limit, we may have to make it a 
hard limit (blocking writes)
 - we should enable incrementing / decrementing and setting global memstore and 
block cache maximum limits. We do not have live configuration changes, but 
regardless of auto-tuning, we should be able to manually set those online. 
 - Periodically we should check past workload (like past 10 min), and depending 
on whether it is write heavy or read heavy (from metrics), adjust the mem 
limits in small intervals. 

What do you guys think? Still worth pursuing?

 Automagically tweak global memstore and block cache sizes based on workload
 ---

 Key: HBASE-5349
 URL: https://issues.apache.org/jira/browse/HBASE-5349
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
 Fix For: 0.96.0


 Hypertable does a neat thing where it changes the size given to the CellCache 
 (our MemStores) and Block Cache based on the workload. If you need an image, 
 scroll down at the bottom of this link: 
 http://www.hypertable.com/documentation/architecture/
 That'd be one less thing to configure.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5788) Move Dynamic Metrics storage off of HRegion.

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256110#comment-13256110
 ] 

Enis Soztutar commented on HBASE-5788:
--

bq. The next step will be to simplify the api exposed so that using it will be 
easier.
@Elliot, do you have something in mind for this? While working on HBASE-5601, I 
also noticed that SchemaMetrics needs some simplification as well.


 Move Dynamic Metrics storage off of HRegion.
 

 Key: HBASE-5788
 URL: https://issues.apache.org/jira/browse/HBASE-5788
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-5788-0.patch, HBASE-5788-1.patch, 
 HBASE-5788-2.patch, HBASE-5788-3.patch, HBASE-5788-4.patch


 HRegion right now has the responsibility of storing static counts and latency 
 numbers for use by the metrics package.  Since these maps are incremented and 
 set from lots of places it makes adding functionality hard.
  
 So move the metrics functionality into SchemaMetrics making it more than just 
 a class for naming.  The next step will be to simplify the api exposed so 
 that using it will be easier.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5788) Move Dynamic Metrics storage off of HRegion.

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256121#comment-13256121
 ] 

Enis Soztutar commented on HBASE-5788:
--

bq. I moved metrics about get/put/delete/icv into a helper class that mutates 
data in RegionMetricsStorage. I was hoping to go that way with pretty simple 
helper classes that expose update methods. Then SchemaMetrics can be mutated in 
the same way and metric naming can be handled by static helpers only.
Ok, i see. Helper methods should help. But I think we should get rid of the 
static state there. 

I also left a comment on HBASE-3614. 

 Move Dynamic Metrics storage off of HRegion.
 

 Key: HBASE-5788
 URL: https://issues.apache.org/jira/browse/HBASE-5788
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-5788-0.patch, HBASE-5788-1.patch, 
 HBASE-5788-2.patch, HBASE-5788-3.patch, HBASE-5788-4.patch


 HRegion right now has the responsibility of storing static counts and latency 
 numbers for use by the metrics package.  Since these maps are incremented and 
 set from lots of places it makes adding functionality hard.
  
 So move the metrics functionality into SchemaMetrics making it more than just 
 a class for naming.  The next step will be to simplify the api exposed so 
 that using it will be easier.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3614) Expose per-region request rate metrics

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256122#comment-13256122
 ] 

Enis Soztutar commented on HBASE-3614:
--

This is fine for JMX, but these per-region metrics will be exposed via the 
Hadoop metrics system as well right? If so, the large number of regions and the 
fact that regions are dynamic would make it useless for ganglia (or similar). 
So, I think we should either not expose them through the hadoop metrics, but do 
itJMX only, or make per-region metrics configurable IMO.

 Expose per-region request rate metrics
 --

 Key: HBASE-3614
 URL: https://issues.apache.org/jira/browse/HBASE-3614
 Project: HBase
  Issue Type: Improvement
  Components: metrics, regionserver
Reporter: Gary Helmling
Assignee: Elliott Clark
Priority: Minor
 Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, 
 HBASE-3614-2.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png


 We currently export metrics on request rates for each region server, and this 
 can help with identifying uneven load at a high level. But once you see a 
 given server under high load, you're forced to extrapolate based on your 
 application patterns and the data it's serving what the likely culprit is.  
 This can and should be much easier if we just exported request rate metrics 
 per-region on each server.
 Dynamically updating the metrics keys based on assigned regions may pose some 
 minor challenges, but this seems a very valuable diagnostic tool to have 
 available.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5760) Unit tests should write only under /target

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251074#comment-13251074
 ] 

Enis Soztutar commented on HBASE-5760:
--

The downstream patch in HBASE-5747 seems not much correlated with this patch. 
The biggest change in this is to fix up the MiniMRCluster to use the proper 
mapred.local.dir, which should also help HBASE-5747. Other tests are not there 
at 0.89-fb. 

 Unit tests should write only under /target
 --

 Key: HBASE-5760
 URL: https://issues.apache.org/jira/browse/HBASE-5760
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.96.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Minor
 Attachments: HBASE-5760_v1.patch


 Some of the unit test runs result in files under $hbase_home/test, 
 $hbase_home/build, or $hbase_home/. We should ensure that all tests use 
 target as their data location.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4336) Convert source tree into maven modules

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250023#comment-13250023
 ] 

Enis Soztutar commented on HBASE-4336:
--

bq. I think an hbase-common, hbase-security and hbase-core packages would be a 
reasonable
+1 to that. At least getting the module structure in place will help us a lot. 
After that we can gradually attack client-server separation. 

Since 0.96 is the singularity, should we make client side module a blocker? It 
seems we might have to break BC for this as well. 

 Convert source tree into maven modules
 --

 Key: HBASE-4336
 URL: https://issues.apache.org/jira/browse/HBASE-4336
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.96.0


 When we originally converted the build to maven we had a single core module 
 defined, but later reverted this to a module-less build for the sake of 
 simplicity.
 It now looks like it's time to re-address this, as we have an actual need for 
 modules to:
 * provide a trimmed down client library that applications can make use of
 * more cleanly support building against different versions of Hadoop, in 
 place of some of the reflection machinations currently required
 * incorporate the secure RPC engine that depends on some secure Hadoop classes
 I propose we start simply by refactoring into two initial modules:
 * core - common classes and utilities, and client-side code and interfaces
 * server - master and region server implementations and supporting code
 This would also lay the groundwork for incorporating the HBase security 
 features that have been developed.  Once the module structure is in place, 
 security-related features could then be incorporated into a third module -- 
 security -- after normal review and approval.  The security module could 
 then depend on secure Hadoop, without modifying the dependencies of the rest 
 of the HBase code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4336) Convert source tree into maven modules

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250066#comment-13250066
 ] 

Enis Soztutar commented on HBASE-4336:
--

bq. BC
BC = backwards compatibility. Sorry for the confusion. I assume that for 
separating the client side, we have to make some changes to core client APIs 
(like splitting the KeyValue, Bytes, ZK classes, etc). 

 Convert source tree into maven modules
 --

 Key: HBASE-4336
 URL: https://issues.apache.org/jira/browse/HBASE-4336
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.96.0


 When we originally converted the build to maven we had a single core module 
 defined, but later reverted this to a module-less build for the sake of 
 simplicity.
 It now looks like it's time to re-address this, as we have an actual need for 
 modules to:
 * provide a trimmed down client library that applications can make use of
 * more cleanly support building against different versions of Hadoop, in 
 place of some of the reflection machinations currently required
 * incorporate the secure RPC engine that depends on some secure Hadoop classes
 I propose we start simply by refactoring into two initial modules:
 * core - common classes and utilities, and client-side code and interfaces
 * server - master and region server implementations and supporting code
 This would also lay the groundwork for incorporating the HBase security 
 features that have been developed.  Once the module structure is in place, 
 security-related features could then be incorporated into a third module -- 
 security -- after normal review and approval.  The security module could 
 then depend on secure Hadoop, without modifying the dependencies of the rest 
 of the HBase code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5754) data lost with gora continuous ingest test (goraci)

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250316#comment-13250316
 ] 

Enis Soztutar commented on HBASE-5754:
--

@Stack, if you need any help with Gora, I can help with that. 

 data lost with gora continuous ingest test (goraci)
 ---

 Key: HBASE-5754
 URL: https://issues.apache.org/jira/browse/HBASE-5754
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1
 Environment: 10 node test cluster
Reporter: Eric Newton
Assignee: stack

 Keith Turner re-wrote the accumulo continuous ingest test using gora, which 
 has both hbase and accumulo back-ends.
 I put a billion entries into HBase, and ran the Verify map/reduce job.  The 
 verification failed because about 21K entries were missing.  The goraci 
 [README|https://github.com/keith-turner/goraci] explains the test, and how it 
 detects missing data.
 I re-ran the test with 100 million entries, and it verified successfully.  
 Both of the times I tested using a billion entries, the verification failed.
 If I run the verification step twice, the results are consistent, so the 
 problem is
 probably not on the verify step.
 Here's the versions of the various packages:
 ||package||version||
 |hadoop|0.20.205.0|
 |hbase|0.92.1|
 |gora|http://svn.apache.org/repos/asf/gora/trunk r1311277|
 |goraci|https://github.com/ericnewton/goraci  tagged 2012-04-08|
 The change I made to goraci was to configure it for hbase and to allow it to 
 build properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245672#comment-13245672
 ] 

Enis Soztutar commented on HBASE-5487:
--

Keith thanks a lot for the refs, we badly need something like FATE. 
Mubarak, or anybody else plans to work on this?

 Generic framework for Master-coordinated tasks
 --

 Key: HBASE-5487
 URL: https://issues.apache.org/jira/browse/HBASE-5487
 Project: HBase
  Issue Type: New Feature
  Components: master, regionserver, zookeeper
Affects Versions: 0.94.0
Reporter: Mubarak Seyed
  Labels: noob

 Need a framework to execute master-coordinated tasks in a fault-tolerant 
 manner. 
 Master-coordinated tasks such as online-scheme change and delete-range 
 (deleting region(s) based on start/end key) can make use of this framework.
 The advantages of framework are
 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for 
 master-coordinated tasks
 2. Ability to abstract the common functions across Master - ZK and RS - ZK
 3. Easy to plugin new master-coordinated tasks without adding code to core 
 components

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5701) Put RegionServerDynamicStatistics under RegionServer in MBean hierarchy rather than have it as a peer.

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244864#comment-13244864
 ] 

Enis Soztutar commented on HBASE-5701:
--

I've just seen this after leaving my comments on HBASE-5601.

+1 on the patch (non-binding)

 Put RegionServerDynamicStatistics under RegionServer in MBean hierarchy 
 rather than have it as a peer.
 --

 Key: HBASE-5701
 URL: https://issues.apache.org/jira/browse/HBASE-5701
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Attachments: 5701.txt, 5701.txt, screenshot-1.jpg, screenshot-2.jpg




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241863#comment-13241863
 ] 

Enis Soztutar commented on HBASE-5619:
--

I think for compiling protoc installation should not be a dependency. Why don't 
we add the generated java files to the source tree, so that only if you change 
the protoc definitions, you would have to recompile the .proto files. As far as 
I know, this is how it is done for thrift dependencies in other projects. 

 Create PB protocols for HRegionInterface
 

 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
 hbase-5619_v4.patch


 Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
 client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241916#comment-13241916
 ] 

Enis Soztutar commented on HBASE-5619:
--

I was suggesting adding the generated files under src/, and adding the runtime 
protobuf dependency in maven. But if it won't work, we can go with requiring 
protoc. Dynamic installation will be a little bit overkill. 

 Create PB protocols for HRegionInterface
 

 Key: HBASE-5619
 URL: https://issues.apache.org/jira/browse/HBASE-5619
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
 hbase-5619_v4.patch


 Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
 client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238938#comment-13238938
 ] 

Enis Soztutar commented on HBASE-5623:
--

Thanks Lars for pushing this. Just as a note, I just tested the final version 
of the patch on a 4 node cluster with ycsb -threads 100. No problems.  

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Fix For: 0.94.0

 Attachments: 5623-suggestion.txt, 5623-v7.txt, 5623-v8.txt, 5623.txt, 
 5623v2.txt, HBASE-5623_v0.patch, HBASE-5623_v4.patch, HBASE-5623_v5.patch, 
 HBASE-5623_v6-alt.patch, HBASE-5623_v6-alt.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
 {code}
 It seems the root cause of the issue is that we open a new log writer and 
 close the old one at HLog#rollWriter() holding the updateLock, but the other 
 threads doing syncer() calls
 {code} 
 logSyncerThread.hlogFlush(this.writer);
 {code}
 without holding the updateLock. LogSyncer only synchronizes against 
 concurrent appends and flush(), but not on the passed writer, which can be 
 closed already by rollWriter(). In this case, since 
 SequenceFile#Writer.close() sets it's out field as null, we get the NPE. 

--
This message is automatically generated by JIRA.
If you 

[jira] [Commented] (HBASE-5636) TestTableMapReduce doesn't work properly.

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239125#comment-13239125
 ] 

Enis Soztutar commented on HBASE-5636:
--

Very good find. It seems that HBASE-4503 might have caused it. Do you plan to 
work on this? 

 TestTableMapReduce doesn't work properly.
 -

 Key: HBASE-5636
 URL: https://issues.apache.org/jira/browse/HBASE-5636
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.92.1
Reporter: Takuya Ueshin

 No map function is called because there are no test data put before test 
 starts.
 The following three tests are in the same situation:
 - org.apache.hadoop.hbase.mapred.TestTableMapReduce
 - org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
 - org.apache.hadoop.hbase.mapreduce.TestMulitthreadedTableMapper

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236889#comment-13236889
 ] 

Enis Soztutar commented on HBASE-5623:
--

@Stack, thanks for the unit test. I'll test the the unit test with and without 
the patch today. Normally, it is very easy to replicate this bug. I just set up 
a 4 node cluster with Hadoop-1.0.1, and run ycsb with -load -threads 30.  

@Lars
Yes, it is the same with HBASE-5336. I had searched for the exception but did 
not find that issue. Anyway I'll close that one as duplicate. 
bq. If we want maximum concurrency here, we need to do the same we do for 
writer.sync(). I.e. try without lock held, if failing get the lock and retry, 
along with SequenceFileLogWriter.getLength() catching Exception and retrow as 
IOException.
Yes we can do that, but we cannot do this double-checking for the 
logSyncerThread.hlogFlush(this.writer) since it is not idempotent, so in any 
case we should be holding the updateLock, no? From my understanding you are 
suggesting that we sync on the hlogFlush, and do the retry mechanism in 
writer.sync and getLength(), catching all the exceptions in the first try, 
right? 

 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Attachments: 5623v2.txt, HBASE-5623_v0.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
  

[jira] [Commented] (HBASE-5623) Race condition when rolling the HLog and hlogFlush

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237234#comment-13237234
 ] 

Enis Soztutar commented on HBASE-5623:
--

bq. Ran the about 300 times now, no failure.
Lars, did you run the Stack's patch as it is? It does not fail also in trunk, 
but the problem is that the threads exit with NPE, and it gets written to the 
log, but the test passes. 


 Race condition when rolling the HLog and hlogFlush
 --

 Key: HBASE-5623
 URL: https://issues.apache.org/jira/browse/HBASE-5623
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Fix For: 0.94.0

 Attachments: 5623.txt, 5623v2.txt, HBASE-5623_v0.patch, 
 HBASE-5623_v4.patch


 When doing a ycsb test with a large number of handlers 
 (regionserver.handler.count=60), I get the following exceptions:
 {code}
 Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.getLength(SequenceFile.java:1099)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.getLength(SequenceFileLogWriter.java:314)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1291)
   at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1388)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:920)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:152)
   at $Proxy1.multi(Unknown Source)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1691)
   at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1689)
   at 
 org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:214)
 {code}
 and 
 {code}
   java.lang.NullPointerException
   at 
 org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1026)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1068)
   at 
 org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1035)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.append(SequenceFileLogWriter.java:279)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.hlogFlush(HLog.java:1237)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1271)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1391)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:2192)
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1985)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3400)
   at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:366)
   at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1351)
 {code}
 It seems the root cause of the issue is that we open a new log writer and 
 close the old one at HLog#rollWriter() holding the updateLock, but the other 
 threads doing syncer() calls
 {code} 
 logSyncerThread.hlogFlush(this.writer);
 {code}
 without holding the updateLock. LogSyncer only synchronizes against 
 concurrent appends and flush(), but not on the passed writer, which can be 
 closed already by rollWriter(). In this case, since 
 SequenceFile#Writer.close() sets it's out field as null, we get the NPE. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please 

[jira] [Commented] (HBASE-5522) hbase 0.92 test artifacts are missing from Maven central

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225596#comment-13225596
 ] 

Enis Soztutar commented on HBASE-5522:
--

@Roman, 
Are you able to fetch the test artifacts now? 

 hbase 0.92 test artifacts are missing from Maven central
 

 Key: HBASE-5522
 URL: https://issues.apache.org/jira/browse/HBASE-5522
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Assignee: stack
 Fix For: 0.92.1, 0.94.0

 Attachments: 5522.txt


 Could someone with enough karma, please, publish the test artifacts for 
 0.92.0?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220451#comment-13220451
 ] 

Enis Soztutar commented on HBASE-5371:
--

@Andrew, did you have a chance to look into this? Would appreciate your input. 

 Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) 
 API
 

 Key: HBASE-5371
 URL: https://issues.apache.org/jira/browse/HBASE-5371
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.92.1, 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.94.0

 Attachments: HBASE-5371-addendum_v1.patch, HBASE-5371_v2.patch, 
 HBASE-5371_v3-noprefix.patch, HBASE-5371_v3.patch


 We need to introduce something like 
 AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so 
 that clients can check access rights before carrying out the operations. We 
 need this kind of operation for HCATALOG-245, which introduces authorization 
 providers for hbase over hcat. We cannot use getUserPermissions() since it 
 requires ADMIN permissions on the global/table level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214164#comment-13214164
 ] 

Enis Soztutar commented on HBASE-5371:
--

Thanks Ted for taking a look. If you push this to 92, we have to also push 
HBASE-5358, since it is a blocker for this. 

 Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) 
 API
 

 Key: HBASE-5371
 URL: https://issues.apache.org/jira/browse/HBASE-5371
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.92.1, 0.94.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.94.0

 Attachments: HBASE-5371-addendum_v1.patch, HBASE-5371_v2.patch, 
 HBASE-5371_v3-noprefix.patch, HBASE-5371_v3.patch


 We need to introduce something like 
 AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so 
 that clients can check access rights before carrying out the operations. We 
 need this kind of operation for HCATALOG-245, which introduces authorization 
 providers for hbase over hcat. We cannot use getUserPermissions() since it 
 requires ADMIN permissions on the global/table level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4403) Adopt interface stability/audience classifications from Hadoop

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13210636#comment-13210636
 ] 

Enis Soztutar commented on HBASE-4403:
--

A lot of the downstream projects also depend on MiniHBaseCluster and 
MiniZookeeperCluster for testing. Should we also mark these as public, 
evolving, and create a minicluster artifact, a la HADOOP-8009. This can be done 
as part of HBASE-4336. 

 Adopt interface stability/audience classifications from Hadoop
 --

 Key: HBASE-4403
 URL: https://issues.apache.org/jira/browse/HBASE-4403
 Project: HBase
  Issue Type: Task
Affects Versions: 0.90.5, 0.92.0
Reporter: Todd Lipcon
Assignee: Jimmy Xiang
 Attachments: hbase-4403-interface.txt, 
 hbase-4403-nowhere-near-done.txt


 As HBase gets more widely used, we need to be more explicit about which APIs 
 are stable and not expected to break between versions, which APIs are still 
 evolving, etc. We also have many public classes that are really internal to 
 the RS or Master and not meant to be used by users. Hadoop has adopted a 
 classification scheme for audience (public, private, or limited-private) as 
 well as stability (stable, evolving, unstable). I think we should copy these 
 annotations to HBase and start to classify our public classes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13208717#comment-13208717
 ] 

Enis Soztutar commented on HBASE-5325:
--

bq. Uploading a patch for 0.92 branch in case it is worthwhile to port this 
back into 0.92 as it is for usability improvement.
+1. Would love to see this in 0.92, since this will ease up monitoring 0.92 
deployments a lot.

 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Improvement
Reporter: Hitesh Shah
Assignee: Hitesh Shah
Priority: Minor
 Fix For: 0.94.0

 Attachments: HBASE-5325.1.patch, HBASE-5325.2.patch, 
 HBASE-5325.3.branch-0.92.patch, HBASE-5325.3.patch, HBASE-5325.wip.patch


 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13209076#comment-13209076
 ] 

Enis Soztutar commented on HBASE-5371:
--

bq. It can't because it changes the protocol version of 
AccessControllerProtocol.
Right, since we bumped the protocol version, it will be incompatible. But 
without this, we cannot continue with the patch in HCATALOG-245, which will 
hopefully ported to Hive as well. What about re-changing the version to 1, 
since we just added a new method, but not changed anything on the wire, it 
should be compatible. The only catch is that if you invoke the new API from a 
new client, but the server is using the old version, you would get a 
NoSuchMethod or smt. Is that asking for too much, wdyt?

 Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) 
 API
 

 Key: HBASE-5371
 URL: https://issues.apache.org/jira/browse/HBASE-5371
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.94.0

 Attachments: HBASE-5371_v2.patch, HBASE-5371_v3-noprefix.patch, 
 HBASE-5371_v3.patch


 We need to introduce something like 
 AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so 
 that clients can check access rights before carrying out the operations. We 
 need this kind of operation for HCATALOG-245, which introduces authorization 
 providers for hbase over hcat. We cannot use getUserPermissions() since it 
 requires ADMIN permissions on the global/table level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207983#comment-13207983
 ] 

Enis Soztutar commented on HBASE-5371:
--

I ran the full test suite yesterday with 2 failures which seem unrelated. 
However the security/src/test/ classes were not picked up, since they did not 
define @Category's. Shall we open another ticket to annotate them as large 
tests? Currently, I am running the tests with:
{code} mvn test -P security,localTests {code}
but i think it will take some time to finish. 


 Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) 
 API
 

 Key: HBASE-5371
 URL: https://issues.apache.org/jira/browse/HBASE-5371
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5371_v2.patch, HBASE-5371_v3.patch


 We need to introduce something like 
 AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so 
 that clients can check access rights before carrying out the operations. We 
 need this kind of operation for HCATALOG-245, which introduces authorization 
 providers for hbase over hcat. We cannot use getUserPermissions() since it 
 requires ADMIN permissions on the global/table level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13208098#comment-13208098
 ] 

Enis Soztutar commented on HBASE-5371:
--

test results:
{code}
...
Running org.apache.hadoop.hbase.security.access.TestAccessControlFilter
2012-02-14 14:14:34.390 java[67005:1903] Unable to load realm info from 
SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.002 sec
Running org.apache.hadoop.hbase.security.access.TestAccessController
2012-02-14 14:15:06.963 java[67014:1903] Unable to load realm info from 
SCDynamicStore
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.355 sec
Running org.apache.hadoop.hbase.security.access.TestTablePermissions
2012-02-14 14:15:58.028 java[67023:1903] Unable to load realm info from 
SCDynamicStore
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.739 sec
Running org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher
2012-02-14 14:16:35.517 java[67032:1903] Unable to load realm info from 
SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.579 sec
Running org.apache.hadoop.hbase.security.TestUser
2012-02-14 14:17:00.817 java[67041:1903] Unable to load realm info from 
SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.445 sec
Running org.apache.hadoop.hbase.security.token.TestTokenAuthentication
2012-02-14 14:17:01.835 java[67043:1903] Unable to load realm info from 
SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.099 sec
Running org.apache.hadoop.hbase.security.token.TestZKSecretWatcher
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.46 sec

...

Results :

Failed tests:   
testCyclicReplication(org.apache.hadoop.hbase.replication.TestMasterReplication):
 Waited too much time for put replication
  
testSimplePutDelete(org.apache.hadoop.hbase.replication.TestMasterReplication): 
Waited too much time for put replication
  
testMultiSlaveReplication(org.apache.hadoop.hbase.replication.TestMultiSlaveReplication):
 Waited too much time for put replication
  testClientSessionExpired(org.apache.hadoop.hbase.TestZooKeeper)
  testLeaderSelection(org.apache.hadoop.hbase.zookeeper.TestZKLeaderManager): 
New leader should exist

Tests in error: 
  
testTralingGarbageCorruptionFileSkipErrorsPasses(org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit):
 All datanodes 127.0.0.1:58462 are bad. Aborting...

Tests run: 1429, Failures: 5, Errors: 1, Skipped: 10
{code}

All the failing tests seem to be not related to this patch. It seems they are 
timing out due to running slow.  

 Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) 
 API
 

 Key: HBASE-5371
 URL: https://issues.apache.org/jira/browse/HBASE-5371
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5371_v2.patch, HBASE-5371_v3.patch


 We need to introduce something like 
 AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so 
 that clients can check access rights before carrying out the operations. We 
 need this kind of operation for HCATALOG-245, which introduces authorization 
 providers for hbase over hcat. We cannot use getUserPermissions() since it 
 requires ADMIN permissions on the global/table level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) Push the security 0.92 profile to maven repo

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207170#comment-13207170
 ] 

Enis Soztutar commented on HBASE-5341:
--

@Jonathan, good idea. I have changed the title of this issue, linked 
HBASE-4336, and left a comment there. I guess the security module can be built 
as a part of HBASE-4336 or a subticket there.

 Push the security 0.92 profile to maven repo
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.92.1


 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 
 Edit: after some discussion, and the plans for modularizing the build to 
 include a security module, we changed the issue description to push the 
 security jars in 0.92.1 to maven repo. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207473#comment-13207473
 ] 

Enis Soztutar commented on HBASE-5371:
--

Sure, will do that and attach the results here. 

 Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) 
 API
 

 Key: HBASE-5371
 URL: https://issues.apache.org/jira/browse/HBASE-5371
 Project: HBase
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5371_v2.patch


 We need to introduce something like 
 AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so 
 that clients can check access rights before carrying out the operations. We 
 need this kind of operation for HCATALOG-245, which introduces authorization 
 providers for hbase over hcat. We cannot use getUserPermissions() since it 
 requires ADMIN permissions on the global/table level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13204929#comment-13204929
 ] 

Enis Soztutar commented on HBASE-5325:
--

My 2 cents: 
bq. @Enis metrics2 seems a bit out there for us (hadoop 0.23?). We want to run 
on 0.23 and 1.0 and 2.0, etc., so it'd be a while before we could lean on it. 
metrics2 has facility that would help? (I've not studied it).
It seems that Hadoop-1.0 includes some of the metrics2 changes  but not all, so 
I agree that without going full 0.23, it is not possible to use metrics2. 

bq. I'd be interested in what you think. We need to figure being able to config 
a running cluster; i.e. change Configuration values and have hbase notice. 
Having this go via jmx would likely be like taking the 'killarney road to 
dingle' as my grandma used to say (its shorter if you take the tralee road) so 
maybe jmx is read-only rather than 'management'.
With an offline discussion with Hitesh, we completely agree on the roadmap for 
managing the cluster configuration in one place, but I think JMX is not 
suitable for this. We can keep the current approach of read-only JMX beans, and 
deal with the issue of configuration management in another ticket. I'll try to 
read up on how other distributed software projects approach this problem. 

I totally agree that we might want to aggregate some of the metrics in master, 
and possibly redesign the HServerLoad using metrics, but we can deal with that 
in yet another issue. Since all of the metrics are collected via MasterMetrics, 
and exposed via MasterStatistics, I think we will not need any API change for 
that, no? 

I think the current patch, as it is, is enough for defining a base for future 
work. It only exposes data that is available in the webapi, and with it we 
would have 3 API endpoints/JMX beans that we can query relevant info. 
HBaseInfo for common information 
HBaseMasterInfo for master level status information
MasterStatistics for metrics 

 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Improvement
Reporter: Hitesh Shah
Assignee: Hitesh Shah
Priority: Minor
 Fix For: 0.94.0

 Attachments: HBASE-5325.1.patch, HBASE-5325.wip.patch


 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5372) Table mutation operations should check table level rights, not global rights

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13205035#comment-13205035
 ] 

Enis Soztutar commented on HBASE-5372:
--

Replicating Andrew's comments on the parent issue:
bq. This certainly makes sense for grant/revoke.
bq. However, creating, dropping, or modifying a table has global ramifications 
on the cluster. When making changes here it should remain possible to require 
global CREATE/ADMIN rights for those actions by policy. This way a site admin 
can prevent users from taking actions that perturb region assignment if desired.

Agreed on the possible affects on the global state of the cluster for some 
table level operations. If we do change the enforcement to the table level 
though, cluster admins can still opt to not give admin rights on the table 
level, thus they can prevent possibly disruptive operations. So this change 
will only give more flexibility, but will also allow the current setup, where 
ADMIN permissions are set only globally. 

However, for this maybe we have to revisit table ownership rights. Currently, 
the table owner has every right on the table, and this is not managed through 
the normal grant/revoke operations, but on the table metadata. We may want to 
remove table ownership, but introduce default table creation rights, which 
means that when a user creates a table, she automatically get those rights 
allocated. But another user can grant extra rights, or revoke them. 

 Table mutation operations should check table level rights, not global rights 
 -

 Key: HBASE-5372
 URL: https://issues.apache.org/jira/browse/HBASE-5372
 Project: HBase
  Issue Type: Sub-task
  Components: security
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 getUserPermissions(tableName)/grant/revoke and drop/modify table operations 
 should not check for global CREATE/ADMIN rights, but table CREATE/ADMIN 
 rights. The reasoning is that if a user is able to admin or read from a 
 table, she should be able to read the table's permissions. We can choose 
 whether we want only READ or ADMIN permissions for getUserPermission(). Since 
 we check for global permissions first for table permissions, configuring 
 table access using global permissions will continue to work. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202885#comment-13202885
 ] 

Enis Soztutar commented on HBASE-5341:
--

I agree with Jonathan, that if the compiled binary works with 0.20.xxx, which 
as far as I know is the case, then compilation should not be a big concern. And 
I agree completely if we are going to modularize the build, then hbase-security 
module makes sense. Do we also plan to backport modules to 0.92 branch. If not, 
then we should either go with the plan in this jira, or fix HBASE-5288, and 
push security artifacts to maven repo. 

 HBase build artifact should include security code by defult 
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5343) Access control API in HBaseAdmin.java

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202903#comment-13202903
 ] 

Enis Soztutar commented on HBASE-5343:
--

bq. Adding coprocessor specific methods to HBaseAdmin completely undermines the 
purpose of coprocessors as optionally enabled extensions, and fails to scale as 
features are added. Having HBaseAdmin be a jumble of methods related to 
specific coprocessors is not very user friendly either.
I agree that we cannot throw every interface to HBaseAdmin that is implemented 
as coprocessors. However, my thinking is that from the perspective of the end 
user, co-processors are very-advanced API, and the client should be abstracted 
away from the fact that authorization features are implemented as 
co-processors. The initial reasoning behind this issue also includes the fact 
that, authorization and authentication are core features for a database. 

What about introducing HBaseSecurityAdmin under ./security, which extends 
HBaseAdmin. We can also mark this interface as the stable public interface for 
security related APIs. 

 Access control API in HBaseAdmin.java  
 ---

 Key: HBASE-5343
 URL: https://issues.apache.org/jira/browse/HBASE-5343
 Project: HBase
  Issue Type: Improvement
  Components: client, coprocessors, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 To use the access control mechanism added in HBASE-3025, users should either 
 use the shell interface, or use the coprocessor API directly, which is not 
 very user friendly. We can add grant/revoke/user_permission commands similar 
 to the shell interface to HBaseAdmin assuming HBASE-5341 is in. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5342) Grant/Revoke global permissions

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202905#comment-13202905
 ] 

Enis Soztutar commented on HBASE-5342:
--

I'll work on this once I find some time. 

 Grant/Revoke global permissions
 ---

 Key: HBASE-5342
 URL: https://issues.apache.org/jira/browse/HBASE-5342
 Project: HBase
  Issue Type: Improvement
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 HBASE-3025 introduced simple ACLs based on coprocessors. It defines 
 global/table/cf/cq level permissions. However, there is no way to 
 grant/revoke global level permissions, other than the hbase.superuser conf 
 setting. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202920#comment-13202920
 ] 

Enis Soztutar commented on HBASE-5341:
--

thanks Stack, i feel for your pain :) 

This means that, we should do every 0.92 release with two artifacts. Can you 
resolve this issue as duplicate, when you push the security maven profile? 

 HBase build artifact should include security code by defult 
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201720#comment-13201720
 ] 

Enis Soztutar commented on HBASE-5341:
--

I can provide a patch, if we agree on this. 

 HBase build artifact should include security code by defult 
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201818#comment-13201818
 ] 

Enis Soztutar commented on HBASE-5341:
--

Also, there is no secure artifact in the maven repo, so depending on when 
0.92.1 is cut out, we might want to push 0.92.0-security as well. 

 HBase build artifact should include security code by defult 
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201881#comment-13201881
 ] 

Enis Soztutar commented on HBASE-5341:
--

The only artifact build will be plain 0.92.1 or 0.94 (no -security appended). 
But this will include security related code. It's like Hadoop-1.0.0 which 
includes security related codes in one artifact. 

 HBase build artifact should include security code by defult 
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201898#comment-13201898
 ] 

Enis Soztutar commented on HBASE-5341:
--

Sorry, I did not understand what you are referring to with the certification 
process. Do you mean voting for the RC, signing the release, etc? 

 HBase build artifact should include security code by defult 
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201917#comment-13201917
 ] 

Enis Soztutar commented on HBASE-5341:
--

I don't see a reason for changing the release process. The vote for 0.92.0 
release included both the plain and secure artifacts, see 
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/25671

 HBase build artifact should include security code by defult 
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202000#comment-13202000
 ] 

Enis Soztutar commented on HBASE-5341:
--

Agreed. Conceptually, security related features are not very different that 
other features. They can be included in the code base, disabled by default, and 
marked as experimental if not tested well. 

 HBase build artifact should include security code by defult 
 

 Key: HBASE-5341
 URL: https://issues.apache.org/jira/browse/HBASE-5341
 Project: HBase
  Issue Type: Improvement
  Components: build, security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar

 Hbase 0.92.0 was released with two artifacts, plain and security. The 
 security code is built with -Psecurity. There are two tarballs, but only the 
 plain jar in maven repo at repository.a.o. 
 I see no reason to do a separate artifact for the security related code, 
 since 0.92 already depends on secure Hadoop 1.0.0, and all of the security 
 related code is not loaded by default. In this issue, I propose, we merge the 
 code under /security to src/ and remove the maven profile. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200015#comment-13200015
 ] 

Enis Soztutar commented on HBASE-5325:
--

@Stack, 
I think, what you are suggesting for registering multiple MBeans, and 
co-processor Mbeans can be best handled after we migrated to the metrics2 
system (HBASE-4050), no?. I agree that master/rs mbeans incorporate more 
functionality that just metrics, but we can start with the metrics aggregation 
design and go from there.

 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Priority: Minor
 Attachments: HBASE-5325.wip.patch


 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5309) Hbase web-app /jmx throws an exception

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197543#comment-13197543
 ] 

Enis Soztutar commented on HBASE-5309:
--

The issue arises due to the runtime version mismatch between jackson 1.0.1 
coming from hadoop-1.0 and jackson-1.5.5 from hbase. I think the best way to 
solve this is to update the Jackson version in hadoop-1.0.1, and update hadoop 
in hbase once it is released. See HADOOP-7470

 Hbase web-app /jmx throws an exception
 --

 Key: HBASE-5309
 URL: https://issues.apache.org/jira/browse/HBASE-5309
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Assignee: Enis Soztutar

 hbasemaster:60010/jmx throws an NoSuchMethodError exception

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4920) We need a mascot, a totem

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191317#comment-13191317
 ] 

Enis Soztutar commented on HBASE-4920:
--

Orca +1, Hadoop - Elephant, HBase - Orca makes sense in my view. I liked 
design option 2's as well, can your friend put together the logo with the HBase 
text, to see how they look together. 

 We need a mascot, a totem
 -

 Key: HBASE-4920
 URL: https://issues.apache.org/jira/browse/HBASE-4920
 Project: HBase
  Issue Type: Task
Reporter: stack
 Attachments: HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot 
 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, photo 
 (2).JPG


 We need a totem for our t-shirt that is yet to be printed.  O'Reilly owns the 
 Clyesdale.  We need something else.
 We could have a fluffy little duck that quacks 'hbase!' when you squeeze it 
 and we could order boxes of them from some off-shore sweatshop that 
 subcontracts to a contractor who employs child labor only.
 Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from 
 Salesforce showed me, that was a bit too spiritual for me to be seen quoting 
 here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in 
 translation, bigdata).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira