[jira] [Updated] (HBASE-10520) Add handling for swallowed InterruptedException thrown by Thread.sleep in MiniZooKeeperCluster
[ https://issues.apache.org/jira/browse/HBASE-10520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feng Honghua updated HBASE-10520: - Attachment: HBASE-10520-trunk_v3.patch v3 attached which changes to rethrow InterruptedIOException rather than recording and restoring interrupt status in v2, and looks like it is cleaner and still reasonable: # waitForServerUp(private) is called once by startup(), and startup() itself can throw IOException and InterruptedException. # waitForServerDown(private) is called by shutdown() which itself throws IOException, by killCurrentActiveZooKeeperServer which throws IOException and InterruptedException, by killOneBackupZooKeeperServer which throws IOException and InterruptedException # to align the interface, InterruptedIOException rather than InterruptedException is thrown, since shutdown() throws only IOException. Ping [~nkeywal] and thanks :-) > Add handling for swallowed InterruptedException thrown by Thread.sleep in > MiniZooKeeperCluster > -- > > Key: HBASE-10520 > URL: https://issues.apache.org/jira/browse/HBASE-10520 > Project: HBase > Issue Type: Sub-task > Components: Zookeeper >Reporter: Feng Honghua >Assignee: Feng Honghua >Priority: Minor > Attachments: HBASE-10520-trunk_v1.patch, HBASE-10520-trunk_v2.patch, > HBASE-10520-trunk_v3.patch > > > A sub-task of HBASE-10497 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10543) Two rare test failures with TestLogsCleaner and TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905034#comment-13905034 ] Hudson commented on HBASE-10543: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/]) HBASE-10543 Two rare test failures with TestLogsCleaner and TestSplitLogWorker (jxiang: rev 1569496) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java > Two rare test failures with TestLogsCleaner and TestSplitLogWorker > -- > > Key: HBASE-10543 > URL: https://issues.apache.org/jira/browse/HBASE-10543 > Project: HBase > Issue Type: Test >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.1, 0.99.0 > > Attachments: hbase-10543.patch > > > TestSplitLogWorker#testPreemptTask timed out in waiting for a task prempted. > TestLogsCleaner#testLogCleaning failed to check the files remaining. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905031#comment-13905031 ] Hudson commented on HBASE-10501: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569507) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10563) Set name for FlushHandler thread
[
https://issues.apache.org/jira/browse/HBASE-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905030#comment-13905030
]
Hudson commented on HBASE-10563:
FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/])
HBASE-10563 Set name for FlushHandler thread (Himanshu Vashishtha) (mbertozzi:
rev 1569370)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
> Set name for FlushHandler thread
>
>
> Key: HBASE-10563
> URL: https://issues.apache.org/jira/browse/HBASE-10563
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.98.0
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: 10563-v1.patch
>
>
> The FlushHandler thread in MemStoreFlusher class uses default thread name
> (Thread -XX). This is un-intentional and also confusing in case when there
> are multiple handlers.
> Current Stack trace looks like this:
> {code}
> "Thread-18" prio=10 tid=0x7f4e8cb21800 nid=0x356e waiting on condition
> [0x7f4e6d49a000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0004e5684b00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:39)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:228)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10564) HRegionServer.nextLong should be removed since it's not used anywhere, or should be used somewhere it meant to
[
https://issues.apache.org/jira/browse/HBASE-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905029#comment-13905029
]
Hudson commented on HBASE-10564:
FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/])
HBASE-10564 HRegionServer.nextLong should be removed since it's not used
anywhere, or should be used somewhere it meant to (stack: rev 1569467)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
> HRegionServer.nextLong should be removed since it's not used anywhere, or
> should be used somewhere it meant to
> --
>
> Key: HBASE-10564
> URL: https://issues.apache.org/jira/browse/HBASE-10564
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Feng Honghua
>Assignee: Feng Honghua
>Priority: Minor
> Fix For: 0.98.1, 0.99.0
>
> Attachments: HBASE-10564-trunk_v1.patch
>
>
> HRegionServer has its own nextLong method as below:
> {code}
> /**
>* Generate a random positive long number
>*
>* @return a random positive long number
>*/
> protected long nextLong() {
> long n = rand.nextLong();
> if (n == 0) {
> return nextLong();
> }
> if (n < 0) {
> n = -n;
> }
> return n;
> }
> {code}
> # It's never called anywhere, if we can ensure it's really useless, it should
> be removed
> # Looks likely below should use *nextLong* rather than *rand.nextLong*(but
> not sure):
> {code}
> protected long addScanner(RegionScanner s, HRegion r) throws
> LeaseStillHeldException {
> long scannerId = -1;
> while (true) {
> scannerId = Math.abs(rand.nextLong() << 24) ^ startcode;
> String scannerName = String.valueOf(scannerId);
> RegionScannerHolder existing =
> scanners.putIfAbsent(scannerName, new RegionScannerHolder(s, r));
> if (existing == null) {
> this.leases.createLease(scannerName, this.scannerLeaseTimeoutPeriod,
> new ScannerListener(scannerName));
> break;
> }
> }
> return scannerId;
> }
> {code}
> # Its implemenation would be better if
> {code}
> if (n == 0) {
> return nextLong();
> }
> {code}
> is changed to below (with better readability and (possible) less call-stacks
> by removing recursion)
> {code}
> while (n == 0) {
> n = rand.nextLong();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905032#comment-13905032 ] Hudson commented on HBASE-10546: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569500) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10552) HFilePerformanceEvaluation.GaussianRandomReadBenchmark fails sometimes.
[ https://issues.apache.org/jira/browse/HBASE-10552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905026#comment-13905026 ] Hudson commented on HBASE-10552: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/]) HBASE-10552 HFilePerformanceEvaluation.GaussianRandomReadBenchmark fails sometimes. (larsh: rev 1569126) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java > HFilePerformanceEvaluation.GaussianRandomReadBenchmark fails sometimes. > --- > > Key: HBASE-10552 > URL: https://issues.apache.org/jira/browse/HBASE-10552 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Minor > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10552-0.94.txt, 10552-trunk.txt > > > GaussianRandomReadBenchmark generates seek keys by using a Gaussian > distribution with the mean of N/2 and a sigma of N/10 (N = number of rows > used) and using this key directly to seek into the HFile. The HFile was > seeded with keys from 0-N. > This will fail if we ever generate a key < 0, which is rare, but by no means > impossible. We need to clamp the min and max values to 0 and N, resp. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905027#comment-13905027 ] Hudson commented on HBASE-10539: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/]) HBASE-10539. HRegion.addAndGetGlobalMemstoreSize returns previous size (Feng Honghua) (apurtell: rev 1569471) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10350) Master/AM/RegionStates changes to create and assign region replicas
[ https://issues.apache.org/jira/browse/HBASE-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905011#comment-13905011 ] Enis Soztutar commented on HBASE-10350: --- Forwarding my +1 from RB. > Master/AM/RegionStates changes to create and assign region replicas > --- > > Key: HBASE-10350 > URL: https://issues.apache.org/jira/browse/HBASE-10350 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Devaraj Das > Fix For: 0.99.0 > > Attachments: 10350-1.txt, 10350-3.txt, 10350-4-5.txt > > > As per design in the parent jira, this jira will capture the changes in the > master side (especially AM / RegionStates) for creating tables with region > replicas, and making sure the regions are assigned on create and failover. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10350) Master/AM/RegionStates changes to create and assign region replicas
[ https://issues.apache.org/jira/browse/HBASE-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904998#comment-13904998 ] Devaraj Das commented on HBASE-10350: - Could I get one more reviewer to look at this before I commit to the branch... > Master/AM/RegionStates changes to create and assign region replicas > --- > > Key: HBASE-10350 > URL: https://issues.apache.org/jira/browse/HBASE-10350 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Devaraj Das > Fix For: 0.99.0 > > Attachments: 10350-1.txt, 10350-3.txt, 10350-4-5.txt > > > As per design in the parent jira, this jira will capture the changes in the > master side (especially AM / RegionStates) for creating tables with region > replicas, and making sure the regions are assigned on create and failover. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904995#comment-13904995 ] Enis Soztutar commented on HBASE-10501: --- sorry, late +1. > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HBASE-10565) FavoredNodesPlan accidentally uses an internal Netty type
[ https://issues.apache.org/jira/browse/HBASE-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-10565. Resolution: Fixed Fix Version/s: 0.99.0 0.98.1 0.96.2 Committed > FavoredNodesPlan accidentally uses an internal Netty type > - > > Key: HBASE-10565 > URL: https://issues.apache.org/jira/browse/HBASE-10565 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2, 0.98.1, 0.99.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10565.patch > > > Obvious fix. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904900#comment-13904900 ] Hudson commented on HBASE-10501: FAILURE: Integrated in hbase-0.96-hadoop2 #207 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/207/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569505) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10563) Set name for FlushHandler thread
[
https://issues.apache.org/jira/browse/HBASE-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904897#comment-13904897
]
Hudson commented on HBASE-10563:
FAILURE: Integrated in hbase-0.96-hadoop2 #207 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/207/])
HBASE-10563 Set name for FlushHandler thread (Himanshu Vashishtha) (mbertozzi:
rev 1569371)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
> Set name for FlushHandler thread
>
>
> Key: HBASE-10563
> URL: https://issues.apache.org/jira/browse/HBASE-10563
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.98.0
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: 10563-v1.patch
>
>
> The FlushHandler thread in MemStoreFlusher class uses default thread name
> (Thread -XX). This is un-intentional and also confusing in case when there
> are multiple handlers.
> Current Stack trace looks like this:
> {code}
> "Thread-18" prio=10 tid=0x7f4e8cb21800 nid=0x356e waiting on condition
> [0x7f4e6d49a000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0004e5684b00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:39)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:228)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904903#comment-13904903 ] Hudson commented on HBASE-10546: FAILURE: Integrated in hbase-0.96-hadoop2 #207 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/207/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569493) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904895#comment-13904895 ] Hudson commented on HBASE-10539: FAILURE: Integrated in hbase-0.96-hadoop2 #207 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/207/]) HBASE-10539 HRegion.addAndGetGlobalMemstoreSize returns previous size (stack: rev 1569486) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10551) Change local mode back to one RS thread by default
[ https://issues.apache.org/jira/browse/HBASE-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904893#comment-13904893 ] Andrew Purtell commented on HBASE-10551: 0.98.0 went out with the change. Good reason to change it back? > Change local mode back to one RS thread by default > -- > > Key: HBASE-10551 > URL: https://issues.apache.org/jira/browse/HBASE-10551 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Blocker > Fix For: 0.94.17 > > Attachments: 10551.txt > > > HBASE-10340/HBASE-9892 changed the default number of RS threads from 1 to 3. > This is confusing. This is to set the default back to 1. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904887#comment-13904887 ] Hudson commented on HBASE-10546: SUCCESS: Integrated in HBase-TRUNK #4928 (See [https://builds.apache.org/job/HBase-TRUNK/4928/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569500) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10543) Two rare test failures with TestLogsCleaner and TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904889#comment-13904889 ] Hudson commented on HBASE-10543: SUCCESS: Integrated in HBase-TRUNK #4928 (See [https://builds.apache.org/job/HBase-TRUNK/4928/]) HBASE-10543 Two rare test failures with TestLogsCleaner and TestSplitLogWorker (jxiang: rev 1569496) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java > Two rare test failures with TestLogsCleaner and TestSplitLogWorker > -- > > Key: HBASE-10543 > URL: https://issues.apache.org/jira/browse/HBASE-10543 > Project: HBase > Issue Type: Test >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.1, 0.99.0 > > Attachments: hbase-10543.patch > > > TestSplitLogWorker#testPreemptTask timed out in waiting for a task prempted. > TestLogsCleaner#testLogCleaning failed to check the files remaining. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10565) FavoredNodesPlan accidentally uses an internal Netty type
[ https://issues.apache.org/jira/browse/HBASE-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10565: --- Attachment: 10565.patch Will commit obvious fix shortly to all affected branches. > FavoredNodesPlan accidentally uses an internal Netty type > - > > Key: HBASE-10565 > URL: https://issues.apache.org/jira/browse/HBASE-10565 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2, 0.98.1, 0.99.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 10565.patch > > > Obvious fix. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904885#comment-13904885 ] Hudson commented on HBASE-10501: SUCCESS: Integrated in HBase-TRUNK #4928 (See [https://builds.apache.org/job/HBase-TRUNK/4928/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569507) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10419) Add multiget support to PerformanceEvaluation
[ https://issues.apache.org/jira/browse/HBASE-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10419: - Attachment: HBASE-10419-v4-trunk.patch Once more, rebased onto trunk after cleaning up HBASE-10548. Patch v4 works for me with and without --multiGet in both MR and --nomapreduce modes. > Add multiget support to PerformanceEvaluation > - > > Key: HBASE-10419 > URL: https://issues.apache.org/jira/browse/HBASE-10419 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: HBASE-10419-v2-trunk.patch, HBASE-10419-v3-trunk.patch, > HBASE-10419-v4-trunk.patch, HBASE-10419.0.patch, HBASE-10419.1.patch > > > Folks planning to use multiget may find this useful. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HBASE-10565) FavoredNodesPlan accidentally uses an internal Netty type
Andrew Purtell created HBASE-10565: -- Summary: FavoredNodesPlan accidentally uses an internal Netty type Key: HBASE-10565 URL: https://issues.apache.org/jira/browse/HBASE-10565 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Obvious fix. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10554) Please delete old releases from mirroring system
[ https://issues.apache.org/jira/browse/HBASE-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904875#comment-13904875 ] Sebb commented on HBASE-10554: -- Thanks for the very prompt resolution! BTW, a good time to check if there are any further releases that can be deleted is when the Announce mail for a new release is sent. By that time, the mirrors should have caught up (and the download page will have been updated) so it should be safe to delete any superseded release. > Please delete old releases from mirroring system > > > Key: HBASE-10554 > URL: https://issues.apache.org/jira/browse/HBASE-10554 > Project: HBase > Issue Type: Task >Affects Versions: 0.96.0, 0.96.1, 0.94.14, 0.94.15 >Reporter: Sebb > > To reduce the load on the ASF mirrors, projects are required to delete old > releases [1] > Please can you remove all non-current releases? > Thanks! > [1] http://www.apache.org/dev/release.html#when-to-archive -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10551) Change local mode back to one RS thread by default
[ https://issues.apache.org/jira/browse/HBASE-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904863#comment-13904863 ] stack commented on HBASE-10551: --- So you will leave it at 3 in 0.98 [~andrew.purt...@gmail.com]? > Change local mode back to one RS thread by default > -- > > Key: HBASE-10551 > URL: https://issues.apache.org/jira/browse/HBASE-10551 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Blocker > Fix For: 0.94.17 > > Attachments: 10551.txt > > > HBASE-10340/HBASE-9892 changed the default number of RS threads from 1 to 3. > This is confusing. This is to set the default back to 1. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10551) Change local mode back to one RS thread by default
[ https://issues.apache.org/jira/browse/HBASE-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904860#comment-13904860 ] stack commented on HBASE-10551: --- So you will leave it at 3 in 0.98 [~andrew.purt...@gmail.com]? > Change local mode back to one RS thread by default > -- > > Key: HBASE-10551 > URL: https://issues.apache.org/jira/browse/HBASE-10551 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Blocker > Fix For: 0.94.17 > > Attachments: 10551.txt > > > HBASE-10340/HBASE-9892 changed the default number of RS threads from 1 to 3. > This is confusing. This is to set the default back to 1. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10359) Master/RS WebUI changes for region replicas
[ https://issues.apache.org/jira/browse/HBASE-10359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904861#comment-13904861 ] Enis Soztutar commented on HBASE-10359: --- +1 > Master/RS WebUI changes for region replicas > > > Key: HBASE-10359 > URL: https://issues.apache.org/jira/browse/HBASE-10359 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Devaraj Das > Fix For: 0.99.0 > > Attachments: 10359-2.txt, 10359-3.txt > > > Some UI changes to make region replicas more visible. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10511) Add latency percentiles on PerformanceEvaluation
[ https://issues.apache.org/jira/browse/HBASE-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904852#comment-13904852 ] Enis Soztutar commented on HBASE-10511: --- My bad. Applied the patch, and waiting for the build to finish before pushing the code. Thanks Nick for spotting. > Add latency percentiles on PerformanceEvaluation > > > Key: HBASE-10511 > URL: https://issues.apache.org/jira/browse/HBASE-10511 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.99.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.96.2, 0.98.1, 0.99.0, hbase-10070 > > Attachments: 10511.v1.patch > > > The latency is reported as an array or float. It's easier to deal with > percentiles :-) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HBASE-10554) Please delete old releases from mirroring system
[ https://issues.apache.org/jira/browse/HBASE-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-10554. --- Resolution: Fixed I removed 0.96.0 and 0.96.1. Resolving as addressed. Sorry about that [~s...@apache.org] We'll keep a better eye out in future. > Please delete old releases from mirroring system > > > Key: HBASE-10554 > URL: https://issues.apache.org/jira/browse/HBASE-10554 > Project: HBase > Issue Type: Task >Affects Versions: 0.96.0, 0.96.1, 0.94.14, 0.94.15 >Reporter: Sebb > > To reduce the load on the ASF mirrors, projects are required to delete old > releases [1] > Please can you remove all non-current releases? > Thanks! > [1] http://www.apache.org/dev/release.html#when-to-archive -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10548) Correct commons-math dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-10548:
-
Attachment: HBASE-10548.01-0.98.patch
HBASE-10548.01-0.96.patch
Attaching patches for what was committed to 0.96 and 0.98 since the trunk patch
did not apply to the pom cleanly.
> Correct commons-math dependency version
> ---
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: HBASE-10548.00.patch, HBASE-10548.01-0.96.patch,
> HBASE-10548.01-0.98.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HBASE-10548) Correct commons-math dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-10548:
-
Resolution: Fixed
Fix Version/s: 0.99.0
0.98.1
0.96.2
Release Note: Downgrades HBase's dependency on commons-math to 2.1, which
is consistent with all stable releases of Hadoop.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to all 3 branches. Thanks for the reviews folks!
> Correct commons-math dependency version
> ---
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904849#comment-13904849 ] stack commented on HBASE-10531: --- lgtm Internally, we take the byte array, presume it a serialized KeyValue and then reconstitute the row for a compare over in a method named compareFlatKey. > Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo > > > Key: HBASE-10531 > URL: https://issues.apache.org/jira/browse/HBASE-10531 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0 > > > Currently the byte[] key passed to HFileScanner.seekTo and > HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And > the caller forms this by using kv.getBuffer, which is actually deprecated. > So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10511) Add latency percentiles on PerformanceEvaluation
[ https://issues.apache.org/jira/browse/HBASE-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904845#comment-13904845 ] Nick Dimiduk commented on HBASE-10511: -- Looks like the commit to 0.96 was missed. Corrected. > Add latency percentiles on PerformanceEvaluation > > > Key: HBASE-10511 > URL: https://issues.apache.org/jira/browse/HBASE-10511 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.99.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.96.2, 0.98.1, 0.99.0, hbase-10070 > > Attachments: 10511.v1.patch > > > The latency is reported as an array or float. It's easier to deal with > percentiles :-) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10529) Make Cell extend Cloneable
[ https://issues.apache.org/jira/browse/HBASE-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904841#comment-13904841 ] stack commented on HBASE-10529: --- lgtm but [~mcorgan], you good w/ this? > Make Cell extend Cloneable > -- > > Key: HBASE-10529 > URL: https://issues.apache.org/jira/browse/HBASE-10529 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0 > > Attachments: HBSE-10529.patch > > > Refer to the parent JIRA for discussion on making extending Cloneable. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10516) Refactor code where Threads.sleep is called within a while/for loop
[
https://issues.apache.org/jira/browse/HBASE-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904835#comment-13904835
]
stack commented on HBASE-10516:
---
Nice writeup @nkeywal (and nice commentary [~fenghh] -- especially the bit
about resetting interrupt is useless unless it checked in upper layers). We
should extract your wisdom and put on dev list? It is good stuff. I can write
it up in refguide after we are agreed.
> Refactor code where Threads.sleep is called within a while/for loop
> ---
>
> Key: HBASE-10516
> URL: https://issues.apache.org/jira/browse/HBASE-10516
> Project: HBase
> Issue Type: Bug
> Components: Client, master, regionserver
>Reporter: Feng Honghua
>Assignee: Feng Honghua
> Attachments: HBASE-10516-trunk_v1.patch, HBASE-10516-trunk_v2.patch
>
>
> Threads.sleep implementation:
> {code}
> public static void sleep(long millis) {
> try {
> Thread.sleep(millis);
> } catch (InterruptedException e) {
> e.printStackTrace();
> Thread.currentThread().interrupt();
> }
> }
> {code}
> From above implementation, the current thread's interrupt status is
> restored/reset when InterruptedException is caught and handled. If this
> method is called within a while/for loop, if a first InterruptedException is
> thrown during sleep, it will make the Threads.sleep in next loop immediately
> throw InterruptedException without expected sleep. This behavior breaks the
> intention for independent sleep in each loop
> I mentioned above in HBASE-10497 and this jira is created to handle it
> separately per [~nkeywal]'s suggestion
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10559) Autogenerate CHANGES.txt
[ https://issues.apache.org/jira/browse/HBASE-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904833#comment-13904833 ] Enis Soztutar commented on HBASE-10559: --- I am happy to test it :) > Autogenerate CHANGES.txt > > > Key: HBASE-10559 > URL: https://issues.apache.org/jira/browse/HBASE-10559 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Priority: Minor > > Investigate if it's possible to autogenerate the CHANGES.txt file from the > 'release' Maven profile, using the JIRA API. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10548) Correct commons-math dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-10548:
-
Summary: Correct commons-math dependency version (was: PerfEval work
around wrong runtime dependency version)
> Correct commons-math dependency version
> ---
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10532) Make KeyValueComparator in KeyValue to accept Cell instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904828#comment-13904828 ] Nick Dimiduk commented on HBASE-10532: -- bq. Shall we do a CellComparator, which will not have any KV bits, and start using that internally? +1 > Make KeyValueComparator in KeyValue to accept Cell instead of KeyValue. > --- > > Key: HBASE-10532 > URL: https://issues.apache.org/jira/browse/HBASE-10532 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0 > > Attachments: HBASE-10532.patch, HBASE-10532.patch > > > public int compareRows(final KeyValue left, final KeyValue right) > public boolean matchingRows(final KeyValue left, final KeyValue right) > We can make them to use Cells instead of KeyValue incase we need to use them > for comparison of any type of cell in future. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10003) OnlineMerge should be extended to allow bulk merging
[ https://issues.apache.org/jira/browse/HBASE-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904822#comment-13904822 ] takeshi.miao commented on HBASE-10003: -- Tks [~heathcd] , I will take the previous suggestions into account, and to study the bulk online merge feature ;) > OnlineMerge should be extended to allow bulk merging > > > Key: HBASE-10003 > URL: https://issues.apache.org/jira/browse/HBASE-10003 > Project: HBase > Issue Type: Improvement > Components: Admin, Usability >Affects Versions: 0.98.0, 0.94.6 >Reporter: Clint Heath >Priority: Critical > Labels: noob > > Now that we have Online Merge capabilities, the function of that tool should > be extended to make it much easier for HBase operations folks to use. > Currently it is a very manual process (one fraught with confusion) to hand > pick two regions that are contiguous to each other in the META table such > that the admin can manually request those two regions to be merged. > In the real world, when admins find themselves wanting to merge regions, it's > usually because they've greatly increased their hbase.hregion.max.filesize > property and they have way too many regions on a table and want to reduce the > region count for that entire table quickly and easily. > Why can't the OnlineMerge command just take a "-max" argument along with a > table name which tells it to go ahead and merge all regions of said table > until the resulting regions are all of max size? This takes the voodoo out > of the process and quickly gets the admin what they're looking for. > As part of this improvement, I also suggest a "-regioncount" argument for > OnlineMerge, which will attempt to reduce the table's region count down to > the specified #. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10523) Correct wrong handling and add proper handling for swallowed InterruptedException thrown by Thread.sleep in util
[ https://issues.apache.org/jira/browse/HBASE-10523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904821#comment-13904821 ] stack commented on HBASE-10523: --- +1 from me. [~nkeywal] You good w/ this? > Correct wrong handling and add proper handling for swallowed > InterruptedException thrown by Thread.sleep in util > > > Key: HBASE-10523 > URL: https://issues.apache.org/jira/browse/HBASE-10523 > Project: HBase > Issue Type: Sub-task > Components: util >Reporter: Feng Honghua >Assignee: Feng Honghua > Attachments: HBASE-10523-trunk_v1.patch, HBASE-10523-trunk_v2.patch, > HBASE-10523-trunk_v3.patch > > > A sub-task of HBASE-10497 > # correct wrong handling of InterruptedException where > Thread.currentThread.interrupt() is called within while loops > # add proper handling for swallowed InterruptedException -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10356) Failover RPC's for multi-get
[ https://issues.apache.org/jira/browse/HBASE-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-10356: - Resolution: Fixed Status: Resolved (was: Patch Available) committed to hbase-10070 branch > Failover RPC's for multi-get > - > > Key: HBASE-10356 > URL: https://issues.apache.org/jira/browse/HBASE-10356 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Enis Soztutar >Assignee: Sergey Shelukhin > Fix For: hbase-10070 > > Attachments: HBASE-10356.01.patch, HBASE-10356.01.patch, > HBASE-10356.02.patch, HBASE-10356.patch, HBASE-10356.reference.patch, > HBASE-10356.reference.patch > > > This is extension of HBASE-10355 to add failover support for multi-gets. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9383) Have Cell interface extend HeapSize interface
[ https://issues.apache.org/jira/browse/HBASE-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904809#comment-13904809 ] Enis Soztutar commented on HBASE-9383: -- Agreed with Stack's comments on HBASE-7320. I don't think HeapSize belongs to the Cell interface. Maybe we can have a CellContainer interface, where CellContainer, and KV and encoders implement CellContainer. Also, since Cell is a public facing API, I think we have to be careful to not include methods. We actually want to hide the internal methods in Cell. > Have Cell interface extend HeapSize interface > - > > Key: HBASE-9383 > URL: https://issues.apache.org/jira/browse/HBASE-9383 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0 >Reporter: Jonathan Hsieh > > From the review of HBASE-9359. > bq. Stack: Cell should implement HeapSize? That sounds right. > bq. Ram: +1 for Cell extending HeapSize. > bq. Jon: I'll look into Cell extending HeapSize as a follow on. It doesn't > interfere with the signature and if a older 0.96 client talks to a 0.96.1 rs > that has the change, it won't matter. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10541) Make the file system properties customizable per table/column family
[ https://issues.apache.org/jira/browse/HBASE-10541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904811#comment-13904811 ] stack commented on HBASE-10541: --- How is FS_BLOCKSIZE different from BLOCKSIZE we already have in HColumnDescriptor [~vasu.mariy...@gmail.com]? The FS_ prefix would seem to indicate these settings would work for any filesystem implementation but is that true? Do they only work for HDFS? If so, should they have the HDFS_ prefix instead? Maybe say where the buffer is located so it is more clear what is being sized in below javadoc: + /** Set the amount of data that needs to be buffered during read/write operations. */ ... or rename the variables from FS_BLOCKSIZE to HFILE_HDFS_BLOCKSIZE so there can be no mistake. Otherwise patch looks good. Thanks for working on this. > Make the file system properties customizable per table/column family > > > Key: HBASE-10541 > URL: https://issues.apache.org/jira/browse/HBASE-10541 > Project: HBase > Issue Type: New Feature >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Attachments: trunk-HBASE-10541-rev1.patch, > trunk-HBASE-10541-rev2.patch, trunk-HBASE-10541.patch > > > The file system properties like replication (the number of nodes to which the > hfile needs to be replicated), block size need to be customizable per > table/column family. This is important especially in the testing scenarios or > for test tables where we don't want the hfile to be replicated 3 times. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10356) Failover RPC's for multi-get
[ https://issues.apache.org/jira/browse/HBASE-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-10356: - Attachment: HBASE-10356.02.patch Patch with conflicts resolved > Failover RPC's for multi-get > - > > Key: HBASE-10356 > URL: https://issues.apache.org/jira/browse/HBASE-10356 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Enis Soztutar >Assignee: Sergey Shelukhin > Fix For: hbase-10070 > > Attachments: HBASE-10356.01.patch, HBASE-10356.01.patch, > HBASE-10356.02.patch, HBASE-10356.patch, HBASE-10356.reference.patch, > HBASE-10356.reference.patch > > > This is extension of HBASE-10355 to add failover support for multi-gets. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10356) Failover RPC's for multi-get
[ https://issues.apache.org/jira/browse/HBASE-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-10356: - Fix Version/s: (was: 0.99.0) hbase-10070 > Failover RPC's for multi-get > - > > Key: HBASE-10356 > URL: https://issues.apache.org/jira/browse/HBASE-10356 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Enis Soztutar >Assignee: Sergey Shelukhin > Fix For: hbase-10070 > > Attachments: HBASE-10356.01.patch, HBASE-10356.01.patch, > HBASE-10356.02.patch, HBASE-10356.patch, HBASE-10356.reference.patch, > HBASE-10356.reference.patch > > > This is extension of HBASE-10355 to add failover support for multi-gets. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10526) Using Cell instead of KeyValue in HFileOutputFormat
[
https://issues.apache.org/jira/browse/HBASE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904798#comment-13904798
]
Hadoop QA commented on HBASE-10526:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629622/hbase-10526_v2.patch
against trunk revision .
ATTACHMENT ID: 12629622
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+TaskAttemptContext context = hadoop.createTestTaskAttemptContext(job,
"attempt_201402131733_0001_m_00_0");
+Map familyToCompression = new
HashMap();
+ "(reader: " + reader + ")", hcd.getCompression(),
reader.getFileContext().getCompression());
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestHBaseFsck
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8738//console
This message is automatically generated.
> Using Cell instead of KeyValue in HFileOutputFormat
> ---
>
> Key: HBASE-10526
> URL: https://issues.apache.org/jira/browse/HBASE-10526
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: hbase-10526.patch, hbase-10526_v1.1.patch,
> hbase-10526_v2.patch
>
>
> HFileOutputFormat/KeyValueSortReducer use KeyValue. We should deprecate them
> and use Cell instead.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10548) PerfEval work around wrong runtime dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904794#comment-13904794
]
stack commented on HBASE-10548:
---
I'm good for 0.96
> PerfEval work around wrong runtime dependency version
> -
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10543) Two rare test failures with TestLogsCleaner and TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904792#comment-13904792 ] Hudson commented on HBASE-10543: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #154 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/154/]) HBASE-10543 Two rare test failures with TestLogsCleaner and TestSplitLogWorker (jxiang: rev 1569498) * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java > Two rare test failures with TestLogsCleaner and TestSplitLogWorker > -- > > Key: HBASE-10543 > URL: https://issues.apache.org/jira/browse/HBASE-10543 > Project: HBase > Issue Type: Test >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.1, 0.99.0 > > Attachments: hbase-10543.patch > > > TestSplitLogWorker#testPreemptTask timed out in waiting for a task prempted. > TestLogsCleaner#testLogCleaning failed to check the files remaining. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10564) HRegionServer.nextLong should be removed since it's not used anywhere, or should be used somewhere it meant to
[
https://issues.apache.org/jira/browse/HBASE-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904789#comment-13904789
]
Hudson commented on HBASE-10564:
SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #154 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/154/])
HBASE-10564 HRegionServer.nextLong should be removed (Feng Honghua) (apurtell:
rev 1569476)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
> HRegionServer.nextLong should be removed since it's not used anywhere, or
> should be used somewhere it meant to
> --
>
> Key: HBASE-10564
> URL: https://issues.apache.org/jira/browse/HBASE-10564
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Feng Honghua
>Assignee: Feng Honghua
>Priority: Minor
> Fix For: 0.98.1, 0.99.0
>
> Attachments: HBASE-10564-trunk_v1.patch
>
>
> HRegionServer has its own nextLong method as below:
> {code}
> /**
>* Generate a random positive long number
>*
>* @return a random positive long number
>*/
> protected long nextLong() {
> long n = rand.nextLong();
> if (n == 0) {
> return nextLong();
> }
> if (n < 0) {
> n = -n;
> }
> return n;
> }
> {code}
> # It's never called anywhere, if we can ensure it's really useless, it should
> be removed
> # Looks likely below should use *nextLong* rather than *rand.nextLong*(but
> not sure):
> {code}
> protected long addScanner(RegionScanner s, HRegion r) throws
> LeaseStillHeldException {
> long scannerId = -1;
> while (true) {
> scannerId = Math.abs(rand.nextLong() << 24) ^ startcode;
> String scannerName = String.valueOf(scannerId);
> RegionScannerHolder existing =
> scanners.putIfAbsent(scannerName, new RegionScannerHolder(s, r));
> if (existing == null) {
> this.leases.createLease(scannerName, this.scannerLeaseTimeoutPeriod,
> new ScannerListener(scannerName));
> break;
> }
> }
> return scannerId;
> }
> {code}
> # Its implemenation would be better if
> {code}
> if (n == 0) {
> return nextLong();
> }
> {code}
> is changed to below (with better readability and (possible) less call-stacks
> by removing recursion)
> {code}
> while (n == 0) {
> n = rand.nextLong();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904790#comment-13904790 ] Hudson commented on HBASE-10501: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #154 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/154/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569506) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904791#comment-13904791 ] Hudson commented on HBASE-10546: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #154 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/154/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569495) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904788#comment-13904788 ] Hudson commented on HBASE-10539: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #154 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/154/]) HBASE-10539. HRegion.addAndGetGlobalMemstoreSize returns previous size (Feng Honghua) (apurtell: rev 1569472) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904782#comment-13904782 ] Hudson commented on HBASE-10546: FAILURE: Integrated in hbase-0.96 #299 (See [https://builds.apache.org/job/hbase-0.96/299/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569493) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904781#comment-13904781 ] Hudson commented on HBASE-10501: FAILURE: Integrated in hbase-0.96 #299 (See [https://builds.apache.org/job/hbase-0.96/299/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569505) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904780#comment-13904780 ] Hudson commented on HBASE-10539: FAILURE: Integrated in hbase-0.96 #299 (See [https://builds.apache.org/job/hbase-0.96/299/]) HBASE-10539 HRegion.addAndGetGlobalMemstoreSize returns previous size (stack: rev 1569486) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10532) Make KeyValueComparator in KeyValue to accept Cell instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904774#comment-13904774 ] stack commented on HBASE-10532: --- I like suggestion of adding a CellComparator and using it internally. It underlines the direction we are heading in. Patch looks good. > Make KeyValueComparator in KeyValue to accept Cell instead of KeyValue. > --- > > Key: HBASE-10532 > URL: https://issues.apache.org/jira/browse/HBASE-10532 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0 > > Attachments: HBASE-10532.patch, HBASE-10532.patch > > > public int compareRows(final KeyValue left, final KeyValue right) > public boolean matchingRows(final KeyValue left, final KeyValue right) > We can make them to use Cells instead of KeyValue incase we need to use them > for comparison of any type of cell in future. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10559) Autogenerate CHANGES.txt
[ https://issues.apache.org/jira/browse/HBASE-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904769#comment-13904769 ] stack commented on HBASE-10559: --- I do like it when Lars puts the issues fixed into the email he sends out announcing the release (Works when you have not too many issues). A little script that formatted for CHANGES.txt from JIRA release notes would be sweet (I should have a go at it) > Autogenerate CHANGES.txt > > > Key: HBASE-10559 > URL: https://issues.apache.org/jira/browse/HBASE-10559 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Priority: Minor > > Investigate if it's possible to autogenerate the CHANGES.txt file from the > 'release' Maven profile, using the JIRA API. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10532) Make KeyValueComparator in KeyValue to accept Cell instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904765#comment-13904765 ] Enis Soztutar commented on HBASE-10532: --- This looks good. In matchingRowColumn() we can just call compareRows() and compareColumns() directly. Just a suggestion. Shall we do a CellComparator, which will not have any KV bits, and start using that internally? What do you think. > Make KeyValueComparator in KeyValue to accept Cell instead of KeyValue. > --- > > Key: HBASE-10532 > URL: https://issues.apache.org/jira/browse/HBASE-10532 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0 > > Attachments: HBASE-10532.patch, HBASE-10532.patch > > > public int compareRows(final KeyValue left, final KeyValue right) > public boolean matchingRows(final KeyValue left, final KeyValue right) > We can make them to use Cells instead of KeyValue incase we need to use them > for comparison of any type of cell in future. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10559) Autogenerate CHANGES.txt
[ https://issues.apache.org/jira/browse/HBASE-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904754#comment-13904754 ] Lars Hofhansl commented on HBASE-10559: --- I think it's nice to have the authoritative changes as part of the release. Jira might go away or have issues (or Stack might delete all the version tags *smile*). It's not really that much work to cut and paste from the release notes into CHANGES.txt and then commit that file. Just my $0.02. I'm fine either way. > Autogenerate CHANGES.txt > > > Key: HBASE-10559 > URL: https://issues.apache.org/jira/browse/HBASE-10559 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Priority: Minor > > Investigate if it's possible to autogenerate the CHANGES.txt file from the > 'release' Maven profile, using the JIRA API. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10564) HRegionServer.nextLong should be removed since it's not used anywhere, or should be used somewhere it meant to
[
https://issues.apache.org/jira/browse/HBASE-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904745#comment-13904745
]
Hudson commented on HBASE-10564:
SUCCESS: Integrated in HBase-0.98 #165 (See
[https://builds.apache.org/job/HBase-0.98/165/])
HBASE-10564 HRegionServer.nextLong should be removed (Feng Honghua) (apurtell:
rev 1569476)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
> HRegionServer.nextLong should be removed since it's not used anywhere, or
> should be used somewhere it meant to
> --
>
> Key: HBASE-10564
> URL: https://issues.apache.org/jira/browse/HBASE-10564
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Feng Honghua
>Assignee: Feng Honghua
>Priority: Minor
> Fix For: 0.98.1, 0.99.0
>
> Attachments: HBASE-10564-trunk_v1.patch
>
>
> HRegionServer has its own nextLong method as below:
> {code}
> /**
>* Generate a random positive long number
>*
>* @return a random positive long number
>*/
> protected long nextLong() {
> long n = rand.nextLong();
> if (n == 0) {
> return nextLong();
> }
> if (n < 0) {
> n = -n;
> }
> return n;
> }
> {code}
> # It's never called anywhere, if we can ensure it's really useless, it should
> be removed
> # Looks likely below should use *nextLong* rather than *rand.nextLong*(but
> not sure):
> {code}
> protected long addScanner(RegionScanner s, HRegion r) throws
> LeaseStillHeldException {
> long scannerId = -1;
> while (true) {
> scannerId = Math.abs(rand.nextLong() << 24) ^ startcode;
> String scannerName = String.valueOf(scannerId);
> RegionScannerHolder existing =
> scanners.putIfAbsent(scannerName, new RegionScannerHolder(s, r));
> if (existing == null) {
> this.leases.createLease(scannerName, this.scannerLeaseTimeoutPeriod,
> new ScannerListener(scannerName));
> break;
> }
> }
> return scannerId;
> }
> {code}
> # Its implemenation would be better if
> {code}
> if (n == 0) {
> return nextLong();
> }
> {code}
> is changed to below (with better readability and (possible) less call-stacks
> by removing recursion)
> {code}
> while (n == 0) {
> n = rand.nextLong();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904746#comment-13904746 ] Hudson commented on HBASE-10501: SUCCESS: Integrated in HBase-0.98 #165 (See [https://builds.apache.org/job/HBase-0.98/165/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569506) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904747#comment-13904747 ] Hudson commented on HBASE-10546: SUCCESS: Integrated in HBase-0.98 #165 (See [https://builds.apache.org/job/HBase-0.98/165/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569495) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904744#comment-13904744 ] Hudson commented on HBASE-10539: SUCCESS: Integrated in HBase-0.98 #165 (See [https://builds.apache.org/job/HBase-0.98/165/]) HBASE-10539. HRegion.addAndGetGlobalMemstoreSize returns previous size (Feng Honghua) (apurtell: rev 1569472) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10543) Two rare test failures with TestLogsCleaner and TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904748#comment-13904748 ] Hudson commented on HBASE-10543: SUCCESS: Integrated in HBase-0.98 #165 (See [https://builds.apache.org/job/HBase-0.98/165/]) HBASE-10543 Two rare test failures with TestLogsCleaner and TestSplitLogWorker (jxiang: rev 1569498) * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java > Two rare test failures with TestLogsCleaner and TestSplitLogWorker > -- > > Key: HBASE-10543 > URL: https://issues.apache.org/jira/browse/HBASE-10543 > Project: HBase > Issue Type: Test >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.98.1, 0.99.0 > > Attachments: hbase-10543.patch > > > TestSplitLogWorker#testPreemptTask timed out in waiting for a task prempted. > TestLogsCleaner#testLogCleaning failed to check the files remaining. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10559) Autogenerate CHANGES.txt
[ https://issues.apache.org/jira/browse/HBASE-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904720#comment-13904720 ] Andrew Purtell commented on HBASE-10559: I sent out a release notes link with the 0.98.0 release announcement. JIRA will produce them on demand for the curious. Why not simply nuke CHANGES.txt? > Autogenerate CHANGES.txt > > > Key: HBASE-10559 > URL: https://issues.apache.org/jira/browse/HBASE-10559 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Priority: Minor > > Investigate if it's possible to autogenerate the CHANGES.txt file from the > 'release' Maven profile, using the JIRA API. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (HBASE-10548) PerfEval work around wrong runtime dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904716#comment-13904716
]
Andrew Purtell edited comment on HBASE-10548 at 2/18/14 10:30 PM:
--
I think the change is needed to rectify the issue for both Hadoop 1 and Hadoop
2 (at least 2.2.x is also affected). There is no reason for us to diverge from
Hadoop on the version of this dependency.
was (Author: apurtell):
I think the change is needed to unbreak MR for both Hadoop 1 and Hadoop 2 (at
least 2.2.x is also affected)
> PerfEval work around wrong runtime dependency version
> -
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10548) PerfEval work around wrong runtime dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904716#comment-13904716
]
Andrew Purtell commented on HBASE-10548:
I think the change is needed to unbreak MR for both Hadoop 1 and Hadoop 2 (at
least 2.2.x is also affected)
> PerfEval work around wrong runtime dependency version
> -
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10526) Using Cell instead of KeyValue in HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904713#comment-13904713 ] Jimmy Xiang commented on HBASE-10526: - It's better not to simply replace it, which may impact the upgrade path of some existing old applications. As to the class name, I am open for suggestions. > Using Cell instead of KeyValue in HFileOutputFormat > --- > > Key: HBASE-10526 > URL: https://issues.apache.org/jira/browse/HBASE-10526 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: hbase-10526.patch, hbase-10526_v1.1.patch, > hbase-10526_v2.patch > > > HFileOutputFormat/KeyValueSortReducer use KeyValue. We should deprecate them > and use Cell instead. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10559) Autogenerate CHANGES.txt
[ https://issues.apache.org/jira/browse/HBASE-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904705#comment-13904705 ] Lars Hofhansl commented on HBASE-10559: --- The release notes link shows everything, no? (too long in 0.94 land with < 50 changes per release :) ) > Autogenerate CHANGES.txt > > > Key: HBASE-10559 > URL: https://issues.apache.org/jira/browse/HBASE-10559 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Priority: Minor > > Investigate if it's possible to autogenerate the CHANGES.txt file from the > 'release' Maven profile, using the JIRA API. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10559) Autogenerate CHANGES.txt
[ https://issues.apache.org/jira/browse/HBASE-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904700#comment-13904700 ] stack commented on HBASE-10559: --- I copy from JIRA but then have to do vim hackery to make them fit the CHANGES.txt. IIRC, JIRA has upper bound of 100 JIRAs at a time which is a PITA so in the past had to get a 100 at a time. I've been putting the URL for the JIRA release into the CHANGES.txt. This has a downside in that JIRA is the authority and cannot be 'damaged' (I suppose we should 'close' issues as soon as release is done so they are no longer editable up in JIRA). > Autogenerate CHANGES.txt > > > Key: HBASE-10559 > URL: https://issues.apache.org/jira/browse/HBASE-10559 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Priority: Minor > > Investigate if it's possible to autogenerate the CHANGES.txt file from the > 'release' Maven profile, using the JIRA API. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10526) Using Cell instead of KeyValue in HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904695#comment-13904695 ] Nick Dimiduk commented on HBASE-10526: -- I'm not a fan of maintaining both versions of this thing -- and especially not a fan of the new class's name because it confuses the file format version. Can we instead deprecate the KeyValue version in 0.96, 0.98 and replace it with the Cell version on trunk? > Using Cell instead of KeyValue in HFileOutputFormat > --- > > Key: HBASE-10526 > URL: https://issues.apache.org/jira/browse/HBASE-10526 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: hbase-10526.patch, hbase-10526_v1.1.patch, > hbase-10526_v2.patch > > > HFileOutputFormat/KeyValueSortReducer use KeyValue. We should deprecate them > and use Cell instead. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10537) Let the ExportSnapshot mapper fail and retry on error
[ https://issues.apache.org/jira/browse/HBASE-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904688#comment-13904688 ] Jonathan Hsieh commented on HBASE-10537: +1, lgtm. > Let the ExportSnapshot mapper fail and retry on error > - > > Key: HBASE-10537 > URL: https://issues.apache.org/jira/browse/HBASE-10537 > Project: HBase > Issue Type: Bug > Components: snapshots >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.98.1, 0.99.0 > > Attachments: HBASE-10537-v1.patch, HBASE-10537-v2.patch > > > Instead of completing the job, and force the user to re-run the export if > something failed, let the Mapper fail and retry automatically based on the > job.getMaxMapAttempts() -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10537) Let the ExportSnapshot mapper fail and retry on error
[
https://issues.apache.org/jira/browse/HBASE-10537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904681#comment-13904681
]
Hadoop QA commented on HBASE-10537:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629615/HBASE-10537-v2.patch
against trunk revision .
ATTACHMENT ID: 12629615
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8737//console
This message is automatically generated.
> Let the ExportSnapshot mapper fail and retry on error
> -
>
> Key: HBASE-10537
> URL: https://issues.apache.org/jira/browse/HBASE-10537
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 0.98.1, 0.99.0
>
> Attachments: HBASE-10537-v1.patch, HBASE-10537-v2.patch
>
>
> Instead of completing the job, and force the user to re-run the export if
> something failed, let the Mapper fail and retry automatically based on the
> job.getMaxMapAttempts()
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10547) TestFixedLengthWrapper#testReadWrite occasionally fails with the IBM JDK
[
https://issues.apache.org/jira/browse/HBASE-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904662#comment-13904662
]
Nick Dimiduk commented on HBASE-10547:
--
Ugh. Let me see what I can uncover.
> TestFixedLengthWrapper#testReadWrite occasionally fails with the IBM JDK
>
>
> Key: HBASE-10547
> URL: https://issues.apache.org/jira/browse/HBASE-10547
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
> Environment: IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64
> Compressed References 20131114_175264 (JIT enabled, AOT enabled)
>Reporter: Andrew Purtell
>Priority: Minor
>
> Here's the trace.
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 29.288 sec
> <<< FAILURE!
> testReadWrite(org.apache.hadoop.hbase.types.TestFixedLengthWrapper) Time
> elapsed: 0.025 sec <<< FAILURE!
> arrays first differed at element [8]; expected:<-40> but was:<0>
> at
> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
> at org.junit.Assert.internalArrayEquals(Assert.java:473)
> at org.junit.Assert.assertArrayEquals(Assert.java:294)
> at org.junit.Assert.assertArrayEquals(Assert.java:305)
> at
> org.apache.hadoop.hbase.types.TestFixedLengthWrapper.testReadWrite(TestFixedLengthWrapper.java:60)
> {noformat}
> This is with 0.98.0.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10562) Fix TestMultiTableInputFormat for Hadoop 2 in 0.94
[ https://issues.apache.org/jira/browse/HBASE-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904654#comment-13904654 ] Lars Hofhansl commented on HBASE-10562: --- Yeah, I am seeing the same in 0.94 and Hadoop 2. Passes fine locally, but fails frequently on the Jenkins build. It seems it actually starts a lot of processes/threads, so it might just be too much for the poor jenkins boxes. > Fix TestMultiTableInputFormat for Hadoop 2 in 0.94 > -- > > Key: HBASE-10562 > URL: https://issues.apache.org/jira/browse/HBASE-10562 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.17 > > Attachments: 10562.txt > > > See parent. This test was removed from 0.95.2. > While investigating while it fails in 0.94 with Hadoop2 I found it is > extremely slow and uses up a *lot* of threads. > [~stack], any input? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10548) PerfEval work around wrong runtime dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904655#comment-13904655
]
Nick Dimiduk commented on HBASE-10548:
--
Build passes. This is the patch I'd like to apply to trunk.
[~stack] [~apurtell] what's your preference re: the pom change for your
respective release lines? I think it would be "unusual" to downgrade a
dependency on a patch release, however from everything I can tell, the
production mapreduce runtimes are using 2.1. Anyone using commons-math in their
MR jobs will have encoutered this incompatibility on their own. I'm thinking of
folks running the client outside of a MR runtime.
> PerfEval work around wrong runtime dependency version
> -
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904657#comment-13904657 ] Hudson commented on HBASE-10501: ABORTED: Integrated in HBase-0.94-security #418 (See [https://builds.apache.org/job/HBase-0.94-security/418/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569504) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-8329) Limit compaction speed
[ https://issues.apache.org/jira/browse/HBASE-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904652#comment-13904652 ] Jean-Daniel Cryans commented on HBASE-8329: --- [~sershe] Looking back at my unread emails I found that I didn't reply to your latest request so here it is. PeakCompactionsThrottle could use some more complete javadoc, also it might be good to note somewhere that we're limiting speed _per_ compaction, not RS-wide. Bumping up the number of concurrent compaction handlers won't make throttling more aggressive. Could be counter-intuitive for some folks. Also this comment has an extra white space "Limit peak hour compaction speed , slow down when it is too fast." Can be fixed on commit, +1. > Limit compaction speed > -- > > Key: HBASE-8329 > URL: https://issues.apache.org/jira/browse/HBASE-8329 > Project: HBase > Issue Type: Improvement > Components: Compaction >Reporter: binlijin >Assignee: Sergey Shelukhin > Fix For: 0.99.0 > > Attachments: HBASE-8329-2-trunk.patch, HBASE-8329-3-trunk.patch, > HBASE-8329-4-trunk.patch, HBASE-8329-5-trunk.patch, HBASE-8329-6-trunk.patch, > HBASE-8329-7-trunk.patch, HBASE-8329-8-trunk.patch, HBASE-8329-9-trunk.patch, > HBASE-8329-trunk.patch > > > There is no speed or resource limit for compaction，I think we should add this > feature especially when request burst. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904647#comment-13904647 ] Hudson commented on HBASE-10501: FAILURE: Integrated in HBase-0.94 #1294 (See [https://builds.apache.org/job/HBase-0.94/1294/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569504) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10562) Fix TestMultiTableInputFormat for Hadoop 2 in 0.94
[ https://issues.apache.org/jira/browse/HBASE-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904642#comment-13904642 ] stack commented on HBASE-10562: --- It was disabled because couldn't get it to pass reliably. > Fix TestMultiTableInputFormat for Hadoop 2 in 0.94 > -- > > Key: HBASE-10562 > URL: https://issues.apache.org/jira/browse/HBASE-10562 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.17 > > Attachments: 10562.txt > > > See parent. This test was removed from 0.95.2. > While investigating while it fails in 0.94 with Hadoop2 I found it is > extremely slow and uses up a *lot* of threads. > [~stack], any input? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10548) PerfEval work around wrong runtime dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904649#comment-13904649
]
Hadoop QA commented on HBASE-10548:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629610/HBASE-10548.01.patch
against trunk revision .
ATTACHMENT ID: 12629610
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8736//console
This message is automatically generated.
> PerfEval work around wrong runtime dependency version
> -
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noform
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904646#comment-13904646 ] Hudson commented on HBASE-10539: FAILURE: Integrated in HBase-0.94 #1294 (See [https://builds.apache.org/job/HBase-0.94/1294/]) HBASE-10539 HRegion.addAndGetGlobalMemstoreSize returns previous size (stack: rev 1569480) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904648#comment-13904648 ] Hudson commented on HBASE-10546: FAILURE: Integrated in HBase-0.94 #1294 (See [https://builds.apache.org/job/HBase-0.94/1294/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10548) PerfEval work around wrong runtime dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-10548:
-
Status: Open (was: Patch Available)
> PerfEval work around wrong runtime dependency version
> -
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HBASE-10548) PerfEval work around wrong runtime dependency version
[
https://issues.apache.org/jira/browse/HBASE-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-10548:
-
Status: Patch Available (was: Open)
> PerfEval work around wrong runtime dependency version
> -
>
> Key: HBASE-10548
> URL: https://issues.apache.org/jira/browse/HBASE-10548
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-10548.00.patch, HBASE-10548.01.patch
>
>
> From my
> [comment|https://issues.apache.org/jira/browse/HBASE-10511?focusedCommentId=13902238&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13902238]
> on HBASE-10511:
> I have hadoop-1.2.1 installed from tgz, which packages commons-math-2.1. This
> is *different* from the listed maven dependency, 2.2.
> {noformat}
> $ tar tvf hadoop-1.2.1.tar.gz | grep commons-math
> -rw-rw-r-- 0 0 0 832410 Jul 22 2013
> hadoop-1.2.1/lib/commons-math-2.1.jar
> $ mvn -f pom.xml.hadoop1 dependency:tree | grep commons-math
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile (version managed
> from 2.1)
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile
> {noformat}
> This is a problem because the 2.1 version of
> [DescriptiveStatistics|http://commons.apache.org/proper/commons-math/javadocs/api-2.1/org/apache/commons/math/stat/descriptive/DescriptiveStatistics.html]
> doesn't have a double[] constructor. Running the MR job, mappers fail:
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.commons.math.stat.descriptive.DescriptiveStatistics.([D)V
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1163)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:984)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1401)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:522)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:474)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10549) when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
[ https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904636#comment-13904636 ] Nick Dimiduk commented on HBASE-10549: -- Thanks for the report [~Auraro]! Would you mind attaching a patch vs trunk for the sake of our QA bot? We can work with it from there to ensure the fix is backported to all active versions. We would also appreciate an updated test that covers this scenario. TestLoadIncrementalHFiles and TestLoadIncrementalHFilesSplitRecovery look like a good place to start. > when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop. > > > Key: HBASE-10549 > URL: https://issues.apache.org/jira/browse/HBASE-10549 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11 >Reporter: yuanxinen > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904626#comment-13904626 ] Hudson commented on HBASE-10546: FAILURE: Integrated in HBase-0.94-JDK7 #58 (See [https://builds.apache.org/job/HBase-0.94-JDK7/58/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904625#comment-13904625 ] Hudson commented on HBASE-10501: FAILURE: Integrated in HBase-0.94-JDK7 #58 (See [https://builds.apache.org/job/HBase-0.94-JDK7/58/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569504) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10541) Make the file system properties customizable per table/column family
[ https://issues.apache.org/jira/browse/HBASE-10541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10541: - Assignee: Vasu Mariyala > Make the file system properties customizable per table/column family > > > Key: HBASE-10541 > URL: https://issues.apache.org/jira/browse/HBASE-10541 > Project: HBase > Issue Type: New Feature >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Attachments: trunk-HBASE-10541-rev1.patch, > trunk-HBASE-10541-rev2.patch, trunk-HBASE-10541.patch > > > The file system properties like replication (the number of nodes to which the > hfile needs to be replicated), block size need to be customizable per > table/column family. This is important especially in the testing scenarios or > for test tables where we don't want the hfile to be replicated 3 times. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904624#comment-13904624 ] Hudson commented on HBASE-10539: FAILURE: Integrated in HBase-0.94-JDK7 #58 (See [https://builds.apache.org/job/HBase-0.94-JDK7/58/]) HBASE-10539 HRegion.addAndGetGlobalMemstoreSize returns previous size (stack: rev 1569480) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10526) Using Cell instead of KeyValue in HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-10526: Attachment: hbase-10526_v2.patch Attached patch v2 that is good with hadoop 1.0. > Using Cell instead of KeyValue in HFileOutputFormat > --- > > Key: HBASE-10526 > URL: https://issues.apache.org/jira/browse/HBASE-10526 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: hbase-10526.patch, hbase-10526_v1.1.patch, > hbase-10526_v2.patch > > > HFileOutputFormat/KeyValueSortReducer use KeyValue. We should deprecate them > and use Cell instead. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904580#comment-13904580 ] Hudson commented on HBASE-10539: FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #29 (See [https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/29/]) HBASE-10539 HRegion.addAndGetGlobalMemstoreSize returns previous size (stack: rev 1569480) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10501) Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions
[ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904581#comment-13904581 ] Hudson commented on HBASE-10501: FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #29 (See [https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/29/]) HBASE-10501 Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions. (larsh: rev 1569504) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Improve IncreasingToUpperBoundRegionSplitPolicy to avoid too many regions > - > > Key: HBASE-10501 > URL: https://issues.apache.org/jira/browse/HBASE-10501 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, > 10501-0.94-v5.txt, 10501-0.94.txt, 10501-trunk-v2.txt, 10501-trunk.txt > > > During some (admittedly artificial) load testing we found a large amount > split activity, which we tracked down the > IncreasingToUpperBoundRegionSplitPolicy. > The current logic is this (from the comments): > "regions that are on this server that all are of the same table, squared, > times the region flush size OR the maximum region split size, whichever is > smaller" > So with a flush size of 128mb and max file size of 20gb, we'd need 13 region > of the same table on an RS to reach the max size. > With 10gb file sized it is still 9 regions of the same table. > Considering that the number of regions that an RS can carry is limited and > there might be multiple tables, this should be more configurable. > I think the squaring is smart and we do not need to change it. > We could > * Make the start size configurable and default it to the flush size > * Add multiplier for the initial size, i.e. start with n * flushSize > * Also change the default to start with 2*flush size > Of course one can override the default split policy, but these seem like > simple tweaks. > Or we could instead set the goal of how many regions of the same table would > need to be present in order to reach the max size. In that case we'd start > with maxSize/goal^2. So if max size is 20gb and the goal is three we'd start > with 20g/9 = 2.2g for the initial region size. > [~stack], I'm especially interested in your opinion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904582#comment-13904582 ] Hudson commented on HBASE-10546: FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #29 (See [https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/29/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10564) HRegionServer.nextLong should be removed since it's not used anywhere, or should be used somewhere it meant to
[
https://issues.apache.org/jira/browse/HBASE-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904572#comment-13904572
]
Hudson commented on HBASE-10564:
SUCCESS: Integrated in HBase-TRUNK #4927 (See
[https://builds.apache.org/job/HBase-TRUNK/4927/])
HBASE-10564 HRegionServer.nextLong should be removed since it's not used
anywhere, or should be used somewhere it meant to (stack: rev 1569467)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
> HRegionServer.nextLong should be removed since it's not used anywhere, or
> should be used somewhere it meant to
> --
>
> Key: HBASE-10564
> URL: https://issues.apache.org/jira/browse/HBASE-10564
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Feng Honghua
>Assignee: Feng Honghua
>Priority: Minor
> Fix For: 0.98.1, 0.99.0
>
> Attachments: HBASE-10564-trunk_v1.patch
>
>
> HRegionServer has its own nextLong method as below:
> {code}
> /**
>* Generate a random positive long number
>*
>* @return a random positive long number
>*/
> protected long nextLong() {
> long n = rand.nextLong();
> if (n == 0) {
> return nextLong();
> }
> if (n < 0) {
> n = -n;
> }
> return n;
> }
> {code}
> # It's never called anywhere, if we can ensure it's really useless, it should
> be removed
> # Looks likely below should use *nextLong* rather than *rand.nextLong*(but
> not sure):
> {code}
> protected long addScanner(RegionScanner s, HRegion r) throws
> LeaseStillHeldException {
> long scannerId = -1;
> while (true) {
> scannerId = Math.abs(rand.nextLong() << 24) ^ startcode;
> String scannerName = String.valueOf(scannerId);
> RegionScannerHolder existing =
> scanners.putIfAbsent(scannerName, new RegionScannerHolder(s, r));
> if (existing == null) {
> this.leases.createLease(scannerName, this.scannerLeaseTimeoutPeriod,
> new ScannerListener(scannerName));
> break;
> }
> }
> return scannerId;
> }
> {code}
> # Its implemenation would be better if
> {code}
> if (n == 0) {
> return nextLong();
> }
> {code}
> is changed to below (with better readability and (possible) less call-stacks
> by removing recursion)
> {code}
> while (n == 0) {
> n = rand.nextLong();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904571#comment-13904571 ] Hudson commented on HBASE-10539: SUCCESS: Integrated in HBase-TRUNK #4927 (See [https://builds.apache.org/job/HBase-TRUNK/4927/]) HBASE-10539. HRegion.addAndGetGlobalMemstoreSize returns previous size (Feng Honghua) (apurtell: rev 1569471) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10553) [Site] Regenerate with new announcement
[ https://issues.apache.org/jira/browse/HBASE-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904542#comment-13904542 ] Andrew Purtell commented on HBASE-10553: Yes, it took 2 hours to commit that change. > [Site] Regenerate with new announcement > --- > > Key: HBASE-10553 > URL: https://issues.apache.org/jira/browse/HBASE-10553 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: status.txt.gz, xdoc.diff > > > See attached as 'xdoc.diff' the change made to the trunk site source. > I also regenerated the 0.94 docs using 0.94.16 since they haven't been > updated since 0.94.6. > See attached as 'status.txt.gz' the output of the 'svn status' command for > the resulting staged commit. I am posting this JIRA first because I'm > surprised at the number of deletions. Either I haven't enabled all site > options correctly or there have not been cleanups for some time. Browsing the > generated doc locally looks fine as far as I can tell. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HBASE-10553) [Site] Regenerate with new announcement
[ https://issues.apache.org/jira/browse/HBASE-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-10553. Resolution: Fixed Committed as r1569519. > [Site] Regenerate with new announcement > --- > > Key: HBASE-10553 > URL: https://issues.apache.org/jira/browse/HBASE-10553 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: status.txt.gz, xdoc.diff > > > See attached as 'xdoc.diff' the change made to the trunk site source. > I also regenerated the 0.94 docs using 0.94.16 since they haven't been > updated since 0.94.6. > See attached as 'status.txt.gz' the output of the 'svn status' command for > the resulting staged commit. I am posting this JIRA first because I'm > surprised at the number of deletions. Either I haven't enabled all site > options correctly or there have not been cleanups for some time. Browsing the > generated doc locally looks fine as far as I can tell. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10539) HRegion.addAndGetGlobalMemstoreSize returns previous size
[ https://issues.apache.org/jira/browse/HBASE-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904537#comment-13904537 ] Hudson commented on HBASE-10539: FAILURE: Integrated in HBase-0.94-security #417 (See [https://builds.apache.org/job/HBase-0.94-security/417/]) HBASE-10539 HRegion.addAndGetGlobalMemstoreSize returns previous size (stack: rev 1569480) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion.addAndGetGlobalMemstoreSize returns previous size > - > > Key: HBASE-10539 > URL: https://issues.apache.org/jira/browse/HBASE-10539 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Feng Honghua >Assignee: Feng Honghua > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: HBASE-10539-trunk_v1.patch > > > HRegion.addAndGetGlobalMemstoreSize(addedSize) is called once some write > succeeds and 'addedSize' is the size of the edits newly put to the memstore, > the returned value of HRegion.addAndGetGlobalMemstoreSize(addedSize) is then > checked against the flush threshold to determine if a flush for the region > should be triggered. > By design the returned value should be the updated memstore size after adding > 'addedSize', but current implementation uses this.memstoreSize.getAndAdd > which returns the previous size before adding, actually 'addAndGet' rather > than 'getAndAdd' should be used here. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10546) Two scanner objects are open for each hbase map task but only one scanner object is closed
[ https://issues.apache.org/jira/browse/HBASE-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904538#comment-13904538 ] Hudson commented on HBASE-10546: FAILURE: Integrated in HBase-0.94-security #417 (See [https://builds.apache.org/job/HBase-0.94-security/417/]) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java > Two scanner objects are open for each hbase map task but only one scanner > object is closed > -- > > Key: HBASE-10546 > URL: https://issues.apache.org/jira/browse/HBASE-10546 > Project: HBase > Issue Type: Bug >Reporter: Vasu Mariyala >Assignee: Vasu Mariyala > Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17 > > Attachments: 0.94-HBASE-10546.patch, trunk-HBASE-10546.patch > > > Map reduce framework calls createRecordReader of the > TableInputFormat/MultiTableInputFormat to get the record reader instance. In > this method, we are initializing the TableRecordReaderImpl (restart method). > This initializes the scanner object. After this, map reduce framework calls > initialize on the RecordReader. In our case, this calls restart of the > TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the > end of the task, only the second scanner object is closed. Because of this, > the smallest read point of HRegion is affected. > We don't need to initialize the RecordReader in the createRecordReader method > and we need to close the scanner object in the restart method. (incase if the > restart method is called because of exceptions in the nextKeyValue method) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
