[jira] [Created] (HBASE-18000) Make sure we always return the scanner id with ScanResponse
Lars Hofhansl created HBASE-18000: - Summary: Make sure we always return the scanner id with ScanResponse Key: HBASE-18000 URL: https://issues.apache.org/jira/browse/HBASE-18000 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Some external tooling (like OpenTSDB) relies on the scanner id to tie asynchronous responses back to their requests. (see comments on HBASE-17489) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17999) Pyspark HBase Connector
Waqar Muhammad created HBASE-17999: -- Summary: Pyspark HBase Connector Key: HBASE-17999 URL: https://issues.apache.org/jira/browse/HBASE-17999 Project: HBase Issue Type: Brainstorming Components: API Affects Versions: 1.2.4 Environment: Centos7, Python Reporter: Waqar Muhammad Is there a way/connector to connect hbase from pyspark and perform queries? Is there any official documentation for that? Would be awsome if someone could point me in the right direction Thanks in advance -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17899) VerifyReplication not exiting for invalid arguments even though we have check for it
[ https://issues.apache.org/jira/browse/HBASE-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maddineni Sukumar resolved HBASE-17899. --- Resolution: Fixed > VerifyReplication not exiting for invalid arguments even though we have check > for it > > > Key: HBASE-17899 > URL: https://issues.apache.org/jira/browse/HBASE-17899 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.3.1 >Reporter: Maddineni Sukumar >Assignee: Maddineni Sukumar >Priority: Minor > > In doCommandLine() method when ever we see invalid argument, instead of > exiting we are continuing VerifyReplication job due to "return false;" > missing in below if condition. > if (cmd.startsWith("--")) { > printUsage("Invalid argument '" + cmd + "'"); > } -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17900) VerifyReplication - input param variables declared as static causing issues to run VerifyReplication multiple times in single JVM(mainly for unit tests)
[ https://issues.apache.org/jira/browse/HBASE-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maddineni Sukumar resolved HBASE-17900. --- Resolution: Resolved Fix Version/s: 2.0.0 > VerifyReplication - input param variables declared as static causing issues > to run VerifyReplication multiple times in single JVM(mainly for unit tests) > - > > Key: HBASE-17900 > URL: https://issues.apache.org/jira/browse/HBASE-17900 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.3.1 >Reporter: Maddineni Sukumar >Assignee: Maddineni Sukumar >Priority: Minor > Fix For: 2.0.0 > > > VerifyReplication - input param variables declared as static causing issues > to run VerifyReplication multiple times in single JVM(mainly for unit tests) > Input params related variables are declared as static like below . > static long startTime = 0; > static long endTime = Long.MAX_VALUE; > So if I want to run VerifyReplication of table A and table B one after > another in single JVM then while running for Table B it is running with both > options provided during Table A and Table B because of static input > variables. > We need to make these variables as class level variables or use some cleanup > before each run. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17998) Improve HBase RPC write throttling size estimation
Karan Mehta created HBASE-17998: --- Summary: Improve HBase RPC write throttling size estimation Key: HBASE-17998 URL: https://issues.apache.org/jira/browse/HBASE-17998 Project: HBase Issue Type: Improvement Reporter: Karan Mehta Assignee: Karan Mehta Currently when RPC throttling, the size of each put is estimated using a hardcoded value 100 bytes. This can be improved by using the protobuf size as an estimate, without having to deserialize or do a big refactoring. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: CFP for Dataworks Summit Sydney
I meant HBase. Oops :) From: Devaraj Das Sent: Thursday, May 04, 2017 12:51 PM To: u...@hbase.apache.org; dev@hbase.apache.org Subject: CFP for Dataworks Summit Sydney The Australia/Pacific version of Dataworks Summit is in Sydney this year, September 20-21. This is a great place to talk about work you are doing in Apache Knox or how you are using Knox. Information on submitting an abstract is at https://dataworkssummit.com/sydney-2017/abstracts/submit-abstract/ Tracks: Apache Hadoop Apache Spark and Data Science Cloud and Applications Data Processing and Warehousing Enterprise Adoption IoT and Streaming Operations, Governance and Security Deadline: Friday, May 26th, 2017. Devaraj
CFP for Dataworks Summit Sydney
The Australia/Pacific version of Dataworks Summit is in Sydney this year, September 20-21. This is a great place to talk about work you are doing in Apache Knox or how you are using Knox. Information on submitting an abstract is at https://dataworkssummit.com/sydney-2017/abstracts/submit-abstract/ Tracks: Apache Hadoop Apache Spark and Data Science Cloud and Applications Data Processing and Warehousing Enterprise Adoption IoT and Streaming Operations, Governance and Security Deadline: Friday, May 26th, 2017. Devaraj
[jira] [Created] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt
Ted Yu created HBASE-17997: -- Summary: jruby-complete-1.6.8.jar is in cached_classpath.txt Key: HBASE-17997 URL: https://issues.apache.org/jira/browse/HBASE-17997 Project: HBase Issue Type: Bug Reporter: Ted Yu HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory. However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt This means that user would see exception similar to the following when starting hbase in standalone mode with s3a as rootdir : {code} 2017-05-04 16:41:32,854 WARN [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659] internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu, 04 May 2017 16:27:09 GMT java.lang.IllegalStateException: Joda-time 2.2 or later version is required, but found version: null at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149) at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195) at com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78) at com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115) at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52) at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30) at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785) at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191) at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148) at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281) at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364) at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92) at org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363) at org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217) at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17996) HBase master fails to start sometimes on RHEL7
David Knupp created HBASE-17996: --- Summary: HBase master fails to start sometimes on RHEL7 Key: HBASE-17996 URL: https://issues.apache.org/jira/browse/HBASE-17996 Project: HBase Issue Type: Bug Components: master Reporter: David Knupp Impala includes HBase in its local test environment, and we have found that intermittently, the HBase master node fails to start when we are testing on RHEL7. In these failures, what we typically see in the logs is this: {noformat} 17/04/29 21:33:47 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15bbd21b797000a, negotiated timeout = 9 17/04/29 21:33:47 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null 17/04/29 21:33:48 INFO master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/localhost,16000,1493526758211 from backup master directory {noformat} On a successful startup, the log looks like this: {noformat} 17/04/16 21:32:29 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15b7a2ed6860005, negotiated timeout = 9 17/04/16 21:32:29 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null 17/04/16 21:32:30 INFO util.FSUtils: Created version file at hdfs://localhost:20500/hbase with version=8 17/04/16 21:32:31 INFO master.MasterFileSystem: BOOTSTRAP: creating hbase:meta region {noformat} So the event that seems to be lacking is {{master.MasterFileSystem: BOOTSTRAP: creating hbase:meta region}}. The full logs will be attached. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17995) improve log messages during snapshot related tests
Sean Busbey created HBASE-17995: --- Summary: improve log messages during snapshot related tests Key: HBASE-17995 URL: https://issues.apache.org/jira/browse/HBASE-17995 Project: HBase Issue Type: Improvement Components: integration tests, mapreduce, snapshots, test Affects Versions: 2.0.0 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Trivial Fix For: 2.0.0 while verifying the changes for HBASE-17964 I had to chase down a failure related to having the wrong hbase configs. Adding some additional logging detail let me see what was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17994) Add async client test to Performance Evaluation tool
Guanghao Zhang created HBASE-17994: -- Summary: Add async client test to Performance Evaluation tool Key: HBASE-17994 URL: https://issues.apache.org/jira/browse/HBASE-17994 Project: HBase Issue Type: New Feature Affects Versions: 2.0.0 Reporter: Guanghao Zhang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: [ANNOUNCE] - MemStore default implementation change
Hi Anastasia! This is in master, right? On May 4, 2017 5:09 AM, "Anastasia Braginsky" wrote: Hi All Following the performance benefits demonstrated by the new in-memory compaction algorithm in HBase, please note the anticipated change to the MemStore default implementation. The default in-memory compaction level now become BASIC (the previous default translates to NONE). The post on Apache HBase blog sketches the algorithm, and fleshes out the configuration details. This change is tracked under HBASE-17343. Regards,Anastasia
[ANNOUNCE] - MemStore default implementation change
Hi All Following the performance benefits demonstrated by the new in-memory compaction algorithm in HBase, please note the anticipated change to the MemStore default implementation. The default in-memory compaction level now become BASIC (the previous default translates to NONE). The post on Apache HBase blog sketches the algorithm, and fleshes out the configuration details. This change is tracked under HBASE-17343. Regards,Anastasia
[jira] [Created] (HBASE-17993) delete a redundant log
Jingyun Tian created HBASE-17993: Summary: delete a redundant log Key: HBASE-17993 URL: https://issues.apache.org/jira/browse/HBASE-17993 Project: HBase Issue Type: Improvement Components: rpc Affects Versions: 1.0.0 Reporter: Jingyun Tian Priority: Trivial There is a log which is to track what current call is. It is used to debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17992) The snapShot TimeoutException causes the cleanerChore thread to fail to complete the archive correctly
Bo Cui created HBASE-17992: -- Summary: The snapShot TimeoutException causes the cleanerChore thread to fail to complete the archive correctly Key: HBASE-17992 URL: https://issues.apache.org/jira/browse/HBASE-17992 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 1.3.0, 0.98.10 Reporter: Bo Cui The problem is that when the snapshot occurs TimeoutException or other Exceptions, there is no correct delete /hbase/.hbase-snapshot/tmp, which causes the cleanerChore to fail to complete the archive correctly. Modifying the configuration parameter (hbase.snapshot.master.timeout.millis = 60) only reduces the probability of the problem occurring. So the solution to the problem is: multi-Threaded exceptions or TimeoutExceptions, the Main-thread must wait until all the tasks are finished or canceled, the Main-thread can be cleared /hbase/.hbase-snapshot/tmp/snapshotName.Otherwise the task is likely to write /hbase/.hbase-snapshot/tmp/snapshotName/region - mainfest The problem exists in disabledTableSnapshot and enabledTableSnapshot, because I'm currently using the disabledTableSnapshot, so I provide the patch of disabledTableSnapshot -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17991) Add more details about compation queue on /dump
Guangxu Cheng created HBASE-17991: - Summary: Add more details about compation queue on /dump Key: HBASE-17991 URL: https://issues.apache.org/jira/browse/HBASE-17991 Project: HBase Issue Type: Bug Affects Versions: 2.0.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Priority: Minor Fix For: 2.0.0, 1.4.0 RS dump information as follows {code} RS Queue: === Compaction/Split Queue summary: compaction_queue=(20:0), split_queue=0, merge_queue=0 Compaction/Split Queue dump: LargeCompation Queue: Request = regionName=usertable,user4180497275766179957,1491904095205.1d4085e4438752f3611f66e7b043fe44., storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1, time=21697920409804647 Request = regionName=usertable,user4568009753557153251,1491904099262.95bf004e3c9b35a58c60ca5d5b11d190., storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1, time=21697920413223800 SmallCompation Queue: Store = b, pri = 108 Store = b, pri = 108 Store = b, pri = 108 Store = b, pri = 108 Store = b, pri = 108 Store = b, pri = 109 {code} Compaction queue information will be displayed on page /dump. If compation has selected the file, it will print the details information of the compation, otherwise only print storename and priority(Store = b, pri = 108) which is useless for us. So, we should also print more detailed information, such as regionName, starttime etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)