[jira] [Created] (HBASE-18000) Make sure we always return the scanner id with ScanResponse

2017-05-04 Thread Lars Hofhansl (JIRA)
Lars Hofhansl created HBASE-18000:
-

 Summary: Make sure we always return the scanner id with 
ScanResponse
 Key: HBASE-18000
 URL: https://issues.apache.org/jira/browse/HBASE-18000
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl


Some external tooling (like OpenTSDB) relies on the scanner id to tie 
asynchronous responses back to their requests.

(see comments on HBASE-17489)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17999) Pyspark HBase Connector

2017-05-04 Thread Waqar Muhammad (JIRA)
Waqar Muhammad created HBASE-17999:
--

 Summary: Pyspark HBase Connector
 Key: HBASE-17999
 URL: https://issues.apache.org/jira/browse/HBASE-17999
 Project: HBase
  Issue Type: Brainstorming
  Components: API
Affects Versions: 1.2.4
 Environment: Centos7, Python
Reporter: Waqar Muhammad


Is there a way/connector to connect hbase from pyspark and perform queries? Is 
there any official documentation for that? Would be awsome if someone could 
point me in the right direction
Thanks in advance



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (HBASE-17899) VerifyReplication not exiting for invalid arguments even though we have check for it

2017-05-04 Thread Maddineni Sukumar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maddineni Sukumar resolved HBASE-17899.
---
Resolution: Fixed

> VerifyReplication not exiting for invalid arguments even though we have check 
> for it
> 
>
> Key: HBASE-17899
> URL: https://issues.apache.org/jira/browse/HBASE-17899
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.3.1
>Reporter: Maddineni Sukumar
>Assignee: Maddineni Sukumar
>Priority: Minor
>
> In doCommandLine() method when ever we see invalid argument, instead of 
> exiting we are continuing VerifyReplication job due to "return false;" 
> missing in below if condition.
> if (cmd.startsWith("--")) {
>   printUsage("Invalid argument '" + cmd + "'");
> }



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (HBASE-17900) VerifyReplication - input param variables declared as static causing issues to run VerifyReplication multiple times in single JVM(mainly for unit tests)

2017-05-04 Thread Maddineni Sukumar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maddineni Sukumar resolved HBASE-17900.
---
   Resolution: Resolved
Fix Version/s: 2.0.0

> VerifyReplication  - input param variables declared as static causing issues 
> to run VerifyReplication multiple times in single JVM(mainly for unit tests)
> -
>
> Key: HBASE-17900
> URL: https://issues.apache.org/jira/browse/HBASE-17900
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.3.1
>Reporter: Maddineni Sukumar
>Assignee: Maddineni Sukumar
>Priority: Minor
> Fix For: 2.0.0
>
>
> VerifyReplication  - input param variables declared as static causing issues 
> to run VerifyReplication multiple times in single JVM(mainly for unit tests)
> Input params related variables are declared as static like below .
> static long startTime = 0;
> static long endTime = Long.MAX_VALUE;
> So if I want to run VerifyReplication of table A and table B one after 
> another in single JVM then while running for Table B it is running with both 
> options provided during Table A and Table B  because of static input 
> variables. 
> We need to make these variables as class level variables or use some cleanup 
> before each run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17998) Improve HBase RPC write throttling size estimation

2017-05-04 Thread Karan Mehta (JIRA)
Karan Mehta created HBASE-17998:
---

 Summary: Improve HBase RPC write throttling size estimation
 Key: HBASE-17998
 URL: https://issues.apache.org/jira/browse/HBASE-17998
 Project: HBase
  Issue Type: Improvement
Reporter: Karan Mehta
Assignee: Karan Mehta


Currently when RPC throttling, the size of each put is estimated using a 
hardcoded value 100 bytes. This can be improved by using the protobuf size as 
an estimate, without having to deserialize or do a big refactoring.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


Re: CFP for Dataworks Summit Sydney

2017-05-04 Thread Devaraj Das
I meant HBase. Oops :)

From: Devaraj Das
Sent: Thursday, May 04, 2017 12:51 PM
To: u...@hbase.apache.org; dev@hbase.apache.org
Subject: CFP for Dataworks Summit Sydney

The Australia/Pacific version of Dataworks Summit is in Sydney this year, 
September 20-21.   This is a great place to talk about work you are doing in 
Apache Knox or how you are using Knox.  Information on submitting an abstract 
is at https://dataworkssummit.com/sydney-2017/abstracts/submit-abstract/

Tracks:
Apache Hadoop
Apache Spark and Data Science
Cloud and Applications
Data Processing and Warehousing
Enterprise Adoption
IoT and Streaming
Operations, Governance and Security

Deadline: Friday, May 26th, 2017.

Devaraj



CFP for Dataworks Summit Sydney

2017-05-04 Thread Devaraj Das
The Australia/Pacific version of Dataworks Summit is in Sydney this year, 
September 20-21.   This is a great place to talk about work you are doing in 
Apache Knox or how you are using Knox.  Information on submitting an abstract 
is at https://dataworkssummit.com/sydney-2017/abstracts/submit-abstract/

Tracks:
Apache Hadoop
Apache Spark and Data Science
Cloud and Applications
Data Processing and Warehousing
Enterprise Adoption
IoT and Streaming
Operations, Governance and Security

Deadline: Friday, May 26th, 2017.

Devaraj



[jira] [Created] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt

2017-05-04 Thread Ted Yu (JIRA)
Ted Yu created HBASE-17997:
--

 Summary: jruby-complete-1.6.8.jar is in cached_classpath.txt
 Key: HBASE-17997
 URL: https://issues.apache.org/jira/browse/HBASE-17997
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu


HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.

However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt

This means that user would see exception similar to the following when starting 
hbase in  standalone mode with s3a as rootdir :
{code}
2017-05-04 16:41:32,854 WARN  
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659] 
internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu, 04 
May 2017 16:27:09 GMT
java.lang.IllegalStateException: Joda-time 2.2 or later version is required, 
but found version: null
  at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
  at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
  at 
com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
  at 
com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
  at 
com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
  at 
com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
  at 
com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
  at 
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
  at 
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
  at 
com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
  at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
  at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
  at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
  at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
  at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
  at 
org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
  at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
  at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
  at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
  at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
  at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
  at 
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
  at 
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
  at 
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
  at 
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
  at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
  at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
  at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
  at 
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
  at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
  at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
  at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
  at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17996) HBase master fails to start sometimes on RHEL7

2017-05-04 Thread David Knupp (JIRA)
David Knupp created HBASE-17996:
---

 Summary: HBase master fails to start sometimes on RHEL7
 Key: HBASE-17996
 URL: https://issues.apache.org/jira/browse/HBASE-17996
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: David Knupp


Impala includes HBase in its local test environment, and we have found that 
intermittently, the HBase master node fails to start when we are testing on 
RHEL7.

In these failures, what we typically see in the logs is this:
{noformat}
17/04/29 21:33:47 INFO zookeeper.ClientCnxn: Session establishment complete on 
server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15bbd21b797000a, 
negotiated timeout = 9
17/04/29 21:33:47 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is 
null
17/04/29 21:33:48 INFO master.ActiveMasterManager: Deleting ZNode for 
/hbase/backup-masters/localhost,16000,1493526758211 from backup master directory
{noformat}

On a successful startup, the log looks like this:
{noformat}
17/04/16 21:32:29 INFO zookeeper.ClientCnxn: Session establishment complete on 
server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15b7a2ed6860005, 
negotiated timeout = 9
17/04/16 21:32:29 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is 
null
17/04/16 21:32:30 INFO util.FSUtils: Created version file at 
hdfs://localhost:20500/hbase with version=8
17/04/16 21:32:31 INFO master.MasterFileSystem: BOOTSTRAP: creating hbase:meta 
region
{noformat}

So the event that seems to be lacking is {{master.MasterFileSystem: BOOTSTRAP: 
creating hbase:meta region}}.

The full logs will be attached.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17995) improve log messages during snapshot related tests

2017-05-04 Thread Sean Busbey (JIRA)
Sean Busbey created HBASE-17995:
---

 Summary: improve log messages during snapshot related tests
 Key: HBASE-17995
 URL: https://issues.apache.org/jira/browse/HBASE-17995
 Project: HBase
  Issue Type: Improvement
  Components: integration tests, mapreduce, snapshots, test
Affects Versions: 2.0.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Trivial
 Fix For: 2.0.0


while verifying the changes for HBASE-17964 I had to chase down a failure 
related to having the wrong hbase configs. Adding some additional logging 
detail let me see what was happening.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17994) Add async client test to Performance Evaluation tool

2017-05-04 Thread Guanghao Zhang (JIRA)
Guanghao Zhang created HBASE-17994:
--

 Summary: Add async client test to Performance Evaluation tool
 Key: HBASE-17994
 URL: https://issues.apache.org/jira/browse/HBASE-17994
 Project: HBase
  Issue Type: New Feature
Affects Versions: 2.0.0
Reporter: Guanghao Zhang






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


Re: [ANNOUNCE] - MemStore default implementation change

2017-05-04 Thread Sean Busbey
Hi Anastasia!

This is in master, right?


On May 4, 2017 5:09 AM, "Anastasia Braginsky" 
wrote:

Hi All


Following the performance benefits demonstrated by the new in-memory
compaction algorithm in HBase, please note the anticipated change to the
MemStore default implementation.
The default in-memory compaction level now become BASIC (the previous
default translates to NONE). The post on Apache HBase blog sketches the
algorithm, and fleshes out the configuration details.
This change is tracked under HBASE-17343.

Regards,Anastasia


[ANNOUNCE] - MemStore default implementation change

2017-05-04 Thread Anastasia Braginsky
Hi All


Following the performance benefits demonstrated by the new in-memory compaction 
algorithm in HBase, please note the anticipated change to the MemStore default 
implementation.
The default in-memory compaction level now become BASIC (the previous default 
translates to NONE). The post on Apache HBase blog sketches the algorithm, and 
fleshes out the configuration details. 
This change is tracked under HBASE-17343. 

Regards,Anastasia


[jira] [Created] (HBASE-17993) delete a redundant log

2017-05-04 Thread Jingyun Tian (JIRA)
Jingyun Tian created HBASE-17993:


 Summary: delete a redundant log 
 Key: HBASE-17993
 URL: https://issues.apache.org/jira/browse/HBASE-17993
 Project: HBase
  Issue Type: Improvement
  Components: rpc
Affects Versions: 1.0.0
Reporter: Jingyun Tian
Priority: Trivial


There is a log which is to track what current call is. It is used to debugging, 
we'd better delete it in released version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17992) The snapShot TimeoutException causes the cleanerChore thread to fail to complete the archive correctly

2017-05-04 Thread Bo Cui (JIRA)
Bo Cui created HBASE-17992:
--

 Summary: The snapShot TimeoutException causes the cleanerChore 
thread to fail to complete the archive correctly
 Key: HBASE-17992
 URL: https://issues.apache.org/jira/browse/HBASE-17992
 Project: HBase
  Issue Type: Bug
  Components: snapshots
Affects Versions: 1.3.0, 0.98.10
Reporter: Bo Cui


The problem is that when the snapshot occurs TimeoutException  or other 
Exceptions, there is no correct delete /hbase/.hbase-snapshot/tmp, which causes 
the cleanerChore to fail to complete the archive correctly.
Modifying the configuration parameter (hbase.snapshot.master.timeout.millis = 
60) only reduces the probability of the problem occurring.

So the solution to the problem is: multi-Threaded exceptions or 
TimeoutExceptions, the Main-thread must wait until all the tasks are finished 
or canceled, the Main-thread can be cleared 
/hbase/.hbase-snapshot/tmp/snapshotName.Otherwise the task is likely to write 
/hbase/.hbase-snapshot/tmp/snapshotName/region - mainfest

The problem exists in disabledTableSnapshot and enabledTableSnapshot, because 
I'm currently using the disabledTableSnapshot, so I provide the patch of 
disabledTableSnapshot



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17991) Add more details about compation queue on /dump

2017-05-04 Thread Guangxu Cheng (JIRA)
Guangxu Cheng created HBASE-17991:
-

 Summary: Add more details about compation queue on /dump
 Key: HBASE-17991
 URL: https://issues.apache.org/jira/browse/HBASE-17991
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
Priority: Minor
 Fix For: 2.0.0, 1.4.0


RS dump information as follows
{code}
RS Queue:
===
Compaction/Split Queue summary: compaction_queue=(20:0), split_queue=0, 
merge_queue=0
Compaction/Split Queue dump:
  LargeCompation Queue:
Request = 
regionName=usertable,user4180497275766179957,1491904095205.1d4085e4438752f3611f66e7b043fe44.,
 storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1, 
time=21697920409804647
Request = 
regionName=usertable,user4568009753557153251,1491904099262.95bf004e3c9b35a58c60ca5d5b11d190.,
 storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1, 
time=21697920413223800
  SmallCompation Queue:
Store = b, pri = 108
Store = b, pri = 108
Store = b, pri = 108
Store = b, pri = 108
Store = b, pri = 108
Store = b, pri = 109
{code}
Compaction queue information will be displayed on page /dump.
If compation has selected the file, it will print the details information of 
the compation, otherwise only print storename and priority(Store = b, pri = 
108) which is useless for us.
So, we should also print more detailed information, such as regionName, 
starttime etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)