[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158204#comment-16158204
]
xinxin fan commented on HBASE-16894:
Creating splits per region for live table will not face the parallel region
opening issue, because the table scan doesn't need to open region. it's
different with snapshot.
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158204#comment-16158204
]
xinxin fan edited comment on HBASE-16894 at 9/8/17 7:01 AM:
Creating splits per region for live table will not face the parallel region
opening issue, because the table scan doesn't need to open regions parallelly
at client side. it's different with snapshot.
was (Author: xinxin fan):
Creating splits per region for live table will not face the parallel region
opening issue, because the table scan doesn't need to open region. it's
different with snapshot.
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18769) Make CompareFilter use generic CompareOperator instead of internal enum
[ https://issues.apache.org/jira/browse/HBASE-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18769: -- Attachment: HBASE-18769.branch-2.003.patch > Make CompareFilter use generic CompareOperator instead of internal enum > --- > > Key: HBASE-18769 > URL: https://issues.apache.org/jira/browse/HBASE-18769 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18769.branch-2.001.patch, > HBASE-18769.branch-2.002.patch, HBASE-18769.branch-2.003.patch > > > Should probably do this so only the one enum in the codebase. Means a bunch > of boilerplate deprecating the old enum for the new. Suggested by > [~anoop.hbase]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18769) Make CompareFilter use generic CompareOperator instead of internal enum
[ https://issues.apache.org/jira/browse/HBASE-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158207#comment-16158207 ] stack commented on HBASE-18769: --- .003 incorporates nice review by [~ram_krish], [~anoop.hbase], and [~chia7712] up on rb. Thanks boys. > Make CompareFilter use generic CompareOperator instead of internal enum > --- > > Key: HBASE-18769 > URL: https://issues.apache.org/jira/browse/HBASE-18769 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18769.branch-2.001.patch, > HBASE-18769.branch-2.002.patch, HBASE-18769.branch-2.003.patch > > > Should probably do this so only the one enum in the codebase. Means a bunch > of boilerplate deprecating the old enum for the new. Suggested by > [~anoop.hbase]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18771) Incorrect StoreFileRefresh leading to split and compaction failures
[ https://issues.apache.org/jira/browse/HBASE-18771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158217#comment-16158217 ] Abhishek Singh Chouhan commented on HBASE-18771: Thanks [~ram_krish] bq. One quesiton on the test case - does this test case fail always without patch or you need to introduce some locking / new threads to introduce the actual bug? Because all are sequential in the tests now and what if closeAndArchiveCompactedFiles() gets completed and then the hr.compact(false) gets executed? Test case always fails on branch-1.3 without the fix. In the test case closeAndArchiveCompactedFiles() explicitly gets completed before he.compact(false) gets called which does not find the file since its already archived. bq. The refreshStoreFiles() API how ever needs the fix in 2.0 and trunk also right? So lets get it in branch-2 and above also? Yes. I checked master and refreshStoreFiles() there is also the same, so i think we should fix there also. The test case for split failure passes on master due to differences in the way splits are done in the new assignment manager. [~lhofhansl] had some good suggestions on using Collections.emptySet() inplace of new ArrayList. Putting up a new patch with those in. > Incorrect StoreFileRefresh leading to split and compaction failures > --- > > Key: HBASE-18771 > URL: https://issues.apache.org/jira/browse/HBASE-18771 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.4.0, 1.3.2, 1.5.0 > > Attachments: HBASE-18771.branch-1.3.001.patch, > HBASE-18771.branch-1.3.002.patch > > > We ran into issues of compaction and split failures with 1.3 similar to > HBASE-18186 and HBASE-17406. Here's what i believe is happening - > Lets say we have 4 store files that are compacted to form a new one. At this > point we now have 5 store files, however only 1(the newly formed) is open now > for the store and rest are waiting to get archived by HFileArchiver > Now before the files are archived we get a FNFE in a scanner. This results in > HRegion.RegionScannerImpl.handleFileNotFound(FileNotFoundException fnfe) > being called which results in region.refreshStoreFiles(true) -> > HStore.refreshStoreFiles() > HStore.refreshStoreFiles now checks the hdfs dir and adds the previously > compacted files back to the store, however these files are also present in > StoreFileManager's compactedFiles list. Now at this point HFileArchiver runs, > checks compactedFiles list and moves these files into the archive directory. > Now when compaction runs it gets: > 2017-09-04 12:30:13,899 ERROR [ctions-1504505399609] > regionserver.CompactSplitThread - Compaction selection failed regionName = > , storeName = 0, priority = 26, time = 1504528213899 > java.io.FileNotFoundException: File does not exist: hdfs:// > at > org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1337) > at > org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1329) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1329) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:422) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360) > at > org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:325) > at > org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63) > at > org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:65) > at > org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82) > at > org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107) > at > org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1679) > Similarly if a split happens after archival we fail after PONR while opening > daughter regions due to FNFE. This results in parent offline and daughters > also in a limbo since they're unable to open. Since we get the error after > PONR we
[jira] [Updated] (HBASE-18771) Incorrect StoreFileRefresh leading to split and compaction failures
[ https://issues.apache.org/jira/browse/HBASE-18771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-18771: --- Attachment: HBASE-18771.branch-1.3.003.patch Patch with changes incorporated. Will put up a patch for other branches if this looks ok [~lhofhansl] [~ram_krish] [~apurtell] > Incorrect StoreFileRefresh leading to split and compaction failures > --- > > Key: HBASE-18771 > URL: https://issues.apache.org/jira/browse/HBASE-18771 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.4.0, 1.3.2, 1.5.0 > > Attachments: HBASE-18771.branch-1.3.001.patch, > HBASE-18771.branch-1.3.002.patch, HBASE-18771.branch-1.3.003.patch > > > We ran into issues of compaction and split failures with 1.3 similar to > HBASE-18186 and HBASE-17406. Here's what i believe is happening - > Lets say we have 4 store files that are compacted to form a new one. At this > point we now have 5 store files, however only 1(the newly formed) is open now > for the store and rest are waiting to get archived by HFileArchiver > Now before the files are archived we get a FNFE in a scanner. This results in > HRegion.RegionScannerImpl.handleFileNotFound(FileNotFoundException fnfe) > being called which results in region.refreshStoreFiles(true) -> > HStore.refreshStoreFiles() > HStore.refreshStoreFiles now checks the hdfs dir and adds the previously > compacted files back to the store, however these files are also present in > StoreFileManager's compactedFiles list. Now at this point HFileArchiver runs, > checks compactedFiles list and moves these files into the archive directory. > Now when compaction runs it gets: > 2017-09-04 12:30:13,899 ERROR [ctions-1504505399609] > regionserver.CompactSplitThread - Compaction selection failed regionName = > , storeName = 0, priority = 26, time = 1504528213899 > java.io.FileNotFoundException: File does not exist: hdfs:// > at > org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1337) > at > org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1329) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1329) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:422) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360) > at > org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:325) > at > org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63) > at > org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:65) > at > org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82) > at > org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107) > at > org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1679) > Similarly if a split happens after archival we fail after PONR while opening > daughter regions due to FNFE. This results in parent offline and daughters > also in a limbo since they're unable to open. Since we get the error after > PONR we also end up aborting the RS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18779) Move CompareOperator to hbase-client module
stack created HBASE-18779: - Summary: Move CompareOperator to hbase-client module Key: HBASE-18779 URL: https://issues.apache.org/jira/browse/HBASE-18779 Project: HBase Issue Type: Sub-task Reporter: stack Assignee: stack Priority: Critical Comes of [~enis] review over in HBASE-14997. Do this review feedback after HBASE-18769 goes in. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-7219) Make it so can connect remotely to a standalone hbase
[ https://issues.apache.org/jira/browse/HBASE-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158223#comment-16158223 ] stack commented on HBASE-7219: -- This works IIRC [~vlefort] Does it not work for you sir? > Make it so can connect remotely to a standalone hbase > - > > Key: HBASE-7219 > URL: https://issues.apache.org/jira/browse/HBASE-7219 > Project: HBase > Issue Type: Bug >Reporter: stack > Labels: beginner > > Should be able to connect from a remote client to a standalone instance. > HBase has 'localhost' in regionservers file and will write 'localhost' to > znode for master location which remote client can't use. Fix. This comes up > on mailing list w/ some frequency. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158231#comment-16158231 ] xinxin fan commented on HBASE-18090: I cann't assign the issue to me, someone can help? > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18778: -- Summary: Use Comparator for StealJobQueue (was: Use Comparator to compare CompactionRequest) > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 3.0.0, 2.0.0-alpha-2 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18778: -- Attachment: HBASE-18778.patch Introduce comparator for CompactSplit.CompactionRunner and HFileCleaner.HFileDeleteTask. > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 3.0.0, 2.0.0-alpha-2 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18778.patch > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18778: -- Status: Patch Available (was: Open) > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 2.0.0-alpha-2, 3.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18778.patch > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158238#comment-16158238 ] Lars George commented on HBASE-17584: - [~Apache9] If this is pushed to branch-1, could you please update the fix versions in this JIRA? > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584-branch-1.patch, HBASE-17584.patch, > HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John reassigned HBASE-18090: -- Assignee: xinxin fan > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17584: -- Fix Version/s: 1.4.0 > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17584-branch-1.patch, HBASE-17584.patch, > HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158245#comment-16158245 ] Duo Zhang commented on HBASE-17584: --- Added 1.4.0 to fix versions, we do not branch-1.4 at that time I believe. Thanks [~larsgeorge] for the reminding. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17584-branch-1.patch, HBASE-17584.patch, > HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18683) Upgrade hbase to commons-math 3
[ https://issues.apache.org/jira/browse/HBASE-18683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-18683: -- Attachment: HBASE-18683.master.003.patch > Upgrade hbase to commons-math 3 > --- > > Key: HBASE-18683 > URL: https://issues.apache.org/jira/browse/HBASE-18683 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0-alpha-2 >Reporter: Peter Somogyi >Assignee: Peter Somogyi > Fix For: 2.0.0 > > Attachments: HBASE-18683.master.001.patch, > HBASE-18683.master.001.patch, HBASE-18683.master.002.patch, > HBASE-18683.master.003.patch, HBASE-18683.master.003.patch, LICENSE.txt, > LICENSE-v3, NOTICE.txt, NOTICE-v3 > > > Upgrade hbase to use commons-math 3.6. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18453) CompactionRequest should not be exposed to user directly
[
https://issues.apache.org/jira/browse/HBASE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158268#comment-16158268
]
Duo Zhang commented on HBASE-18453:
---
The {{CompactionRequester}} seems useless, it is marked as IA.Private but we
never use it in our own code. Will remove it to make the code a bit clean. Will
add a request compaction method to the {{Region}} interface to give CP users
the ability to request a compaction on a region.
> CompactionRequest should not be exposed to user directly
>
>
> Key: HBASE-18453
> URL: https://issues.apache.org/jira/browse/HBASE-18453
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
>
> It is an implementation class. And we need to find another to let user know
> the compaction start and end.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-13312) SmallScannerCallable does not increment scan metrics
[
https://issues.apache.org/jira/browse/HBASE-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158282#comment-16158282
]
Lars George commented on HBASE-13312:
-
[~apurtell] When running my book example (see
https://github.com/larsgeorge/hbase-book/blob/master/ch03/src/main/java/client/ScanCacheBatchExample.java,
just run it against a locally started HBase instance) against a 1.3.1 today, I
noticed the small scanner still reports zero RPC, but should say one, no? Am I
missing something? Do we need to revisit?
> SmallScannerCallable does not increment scan metrics
>
>
> Key: HBASE-13312
> URL: https://issues.apache.org/jira/browse/HBASE-13312
> Project: HBase
> Issue Type: Bug
> Components: Client, Scanners
>Affects Versions: 1.0.0
>Reporter: Lars George
>Assignee: Andrew Purtell
> Fix For: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1
>
> Attachments: HBASE-13312-0.98.patch, HBASE-13312.patch
>
>
> The subclass of {{ScannerCallable}}, called {{SmallScannerCallable}} seems to
> miss calling any of the increment methods of the {{ScanMetrics}}. The
> superclass does so, but the super methods are not invoked. It emits the
> metrics dutifully at the end of {{next()}}, but there are no useful numbers
> in it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158287#comment-16158287 ] Guanghao Zhang commented on HBASE-15809: [~appy] Any progess? It will be great if we can add it to hbase 2.0 release. Thanks. > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.5.0 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17980) Any HRegionInfo we give out should be immutable
[ https://issues.apache.org/jira/browse/HBASE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158302#comment-16158302 ] Kuan-Po Tseng commented on HBASE-17980: --- Hello stack, Now I make MutableRegionInfo public is because I make nearly all methods inside HRegionInfo delegate to MutableRegionInfo, so MutableRegion must be public in this patch. My thought is let MutableRegionInfo be a private class only for RegionInfoBuilder to use after HRegionInfo is totally deleted. Or make MutableRegionInfo private right now and let HRegionInfo remain it used to be ? > Any HRegionInfo we give out should be immutable > --- > > Key: HBASE-17980 > URL: https://issues.apache.org/jira/browse/HBASE-17980 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Kuan-Po Tseng > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-17980.master.v0.patch, HBASE-17980.master.v1.patch > > > This is similar to HBASE-15583. > # Introduce RegionInfo class. HRegionInfo will extend RegionInfo. > # Deprecate HRegionInfo to be removed in 3.0 > # RegionInfo contain all of the read-only methods of HRegionInfo > # Add "RegionInfo Builder" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18753) Introduce the unsynchronized TimeRangeTracker
[ https://issues.apache.org/jira/browse/HBASE-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18753: --- Status: Open (was: Patch Available) > Introduce the unsynchronized TimeRangeTracker > - > > Key: HBASE-18753 > URL: https://issues.apache.org/jira/browse/HBASE-18753 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18753.v0.patch, HBASE-18753.v1.patch > > > If HBASE-18752 is pushed, it will be better to introduce the unsync > TimeRangeTracker for reducing the cost of recalculation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18753) Introduce the unsynchronized TimeRangeTracker
[ https://issues.apache.org/jira/browse/HBASE-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18753: --- Attachment: HBASE-18753.v2.patch v2: ASYNC -> NON_SYNC > Introduce the unsynchronized TimeRangeTracker > - > > Key: HBASE-18753 > URL: https://issues.apache.org/jira/browse/HBASE-18753 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18753.v0.patch, HBASE-18753.v1.patch, > HBASE-18753.v2.patch > > > If HBASE-18752 is pushed, it will be better to introduce the unsync > TimeRangeTracker for reducing the cost of recalculation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18753) Introduce the unsynchronized TimeRangeTracker
[ https://issues.apache.org/jira/browse/HBASE-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18753: --- Status: Patch Available (was: Open) > Introduce the unsynchronized TimeRangeTracker > - > > Key: HBASE-18753 > URL: https://issues.apache.org/jira/browse/HBASE-18753 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18753.v0.patch, HBASE-18753.v1.patch, > HBASE-18753.v2.patch > > > If HBASE-18752 is pushed, it will be better to introduce the unsync > TimeRangeTracker for reducing the cost of recalculation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-13333) Renew Scanner Lease without advancing the RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158344#comment-16158344 ] Lars George commented on HBASE-1: - [~mantonov] [~lhofhansl] This was not committed to branch-1.3 it seems, which is an oversight? All other versions of HBase 1.x have it, as does 2.0. It should be in 1.3 too, no? > Renew Scanner Lease without advancing the RegionScanner > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1 > > Attachments: 1-0.98.txt, 1-master.txt > > > We have a usecase (for Phoenix) where we want to let the server know that the > client is still around. Like a client-side heartbeat. > Doing a full heartbeat is complicated, but we could add the ability to make > scanner call with caching set to 0. The server already does the right thing > (it renews the lease, but does not advance the scanner). > It looks like the client (ScannerCallable) also does the right thing. We > cannot break ResultScanner before HBase 2.0, but we can add a renewLease() > method to AbstractClientScaner. Phoenix (or any other caller) can then cast > to ClientScanner and call that method to ensure we renew the lease on the > server. > It would be a simple and fully backwards compatible change. [~giacomotaylor] > Comments? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-13312) SmallScannerCallable does not increment scan metrics
[
https://issues.apache.org/jira/browse/HBASE-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158365#comment-16158365
]
Andrew Purtell commented on HBASE-13312:
I don't remember this one [~larsgeorge] . Please file a new issue.
> SmallScannerCallable does not increment scan metrics
>
>
> Key: HBASE-13312
> URL: https://issues.apache.org/jira/browse/HBASE-13312
> Project: HBase
> Issue Type: Bug
> Components: Client, Scanners
>Affects Versions: 1.0.0
>Reporter: Lars George
>Assignee: Andrew Purtell
> Fix For: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1
>
> Attachments: HBASE-13312-0.98.patch, HBASE-13312.patch
>
>
> The subclass of {{ScannerCallable}}, called {{SmallScannerCallable}} seems to
> miss calling any of the increment methods of the {{ScanMetrics}}. The
> superclass does so, but the super methods are not invoked. It emits the
> metrics dutifully at the end of {{next()}}, but there are no useful numbers
> in it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18771) Incorrect StoreFileRefresh leading to split and compaction failures
[
https://issues.apache.org/jira/browse/HBASE-18771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158367#comment-16158367
]
Hadoop QA commented on HBASE-18771:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 47s{color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m
23s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:b3a2a00 |
| JIRA Issue | HBASE-18771 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886008/HBASE-18771.branch-1.3.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 413198e531bd 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1.3 / 6fcb15f |
| Default Java | 1.7.0_131 |
| Multi-JDK versions |
[jira] [Commented] (HBASE-18778) Use Comparator for StealJobQueue
[
https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158368#comment-16158368
]
Chia-Ping Tsai commented on HBASE-18778:
How about using the largeThreads and smallThreads as the initial capacity for
the stealJobQueue?
{code}
-StealJobQueue stealJobQueue = new StealJobQueue<>();
+StealJobQueue stealJobQueue = new
StealJobQueue(COMPARATOR);
{code}
> Use Comparator for StealJobQueue
>
>
> Key: HBASE-18778
> URL: https://issues.apache.org/jira/browse/HBASE-18778
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Affects Versions: 3.0.0, 2.0.0-alpha-2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 3.0.0, 2.0.0-alpha-3
>
> Attachments: HBASE-18778.patch
>
>
> To eliminate the confusing compareTo and equals implementation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18621) Refactor ClusterOptions before applying to code base
[ https://issues.apache.org/jira/browse/HBASE-18621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158370#comment-16158370 ] Chia-Ping Tsai commented on HBASE-18621: +1. Will commit it tomorrow if no objections. > Refactor ClusterOptions before applying to code base > > > Key: HBASE-18621 > URL: https://issues.apache.org/jira/browse/HBASE-18621 > Project: HBase > Issue Type: Improvement >Reporter: Reid Chan >Assignee: Reid Chan > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18621.master.001.patch, > HBASE-18621.master.002.patch, HBASE-18621.master.003.patch, > HBASE-18621.master.004.patch, HBASE-18621.master.005.patch, > HBASE-18621.master.006.patch, HBASE-18621.master.007.patch, > HBASE-18621.master.008.patch, HBASE-18621.master.009.patch, > HBASE-18621.master.010.patch > > > So far, ClusterStatus.Options is not so clean that can be applied to code > base. > Refactoring it before next move. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-14998) Unify synchronous and asynchronous methods in Admin and cleanup
[
https://issues.apache.org/jira/browse/HBASE-14998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros reassigned HBASE-14998:
---
Assignee: Balazs Meszaros
> Unify synchronous and asynchronous methods in Admin and cleanup
> ---
>
> Key: HBASE-14998
> URL: https://issues.apache.org/jira/browse/HBASE-14998
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Balazs Meszaros
>Priority: Blocker
> Fix For: 2.0.0-alpha-3
>
>
> Admin has a bunch of methods, some are asnyc, some are sync. Needs some
> unification in method naming, and method signatures.
> - We use modify and alter interchangeably. Pick one and stick with it
> (modifyTable(), versus getAlterStatus()). Shell uses {{alter}}.
> - Remove getAlterStatus(), should not be needed.
> - remove already deprecated methods
> - isTableAvailable(TableName tableName, byte[][] splitKeys) should be
> removed.
> - Consistently use Aysnc as a prefix for all async methods.
> - Other ideas?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18764) add slow read block log entry to alert slow datanodeinfo when reading a block is slow
[
https://issues.apache.org/jira/browse/HBASE-18764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158378#comment-16158378
]
Wang, Xinglong commented on HBASE-18764:
[~elserj]
Thanks for the response.
IMO, for a system with SLA requirement, like such a call path: User request -->
Service A ---> Service B. if request to Service A out of SLA, then Service A
have to determine whether the out of SLA is caused by underlaying Service B or
not, in this case service A has to get the knowledge about responsiveness of
Service B.
When hbase is slow, our first instinct is that hbase is not stable. And then
our first action will be doing check on hbase regionserver. And then hdfs
metrics. It's just hard to correlate these 2 parts.
It’s a headache for us to determine what slow down hbase from responding to
users’ request. Our customers complained a lot about this.
For example, in one of our cluster we have ambari grafana with a lot predefined
metics monitor both for hbase and hdfs.
We can tell that some region servers are with high get latency sometime.
However we can’t tell which underlying datanode is performing bad. Even we find
a datanode with an abnormal metrics like high fsyncNanoTime. We are still
uncertain that this node is the root cause. All we can do is to guess, that
this might be a naughty node that it impacted the whole cluster. If we have
some specific log entry that tells us which datanode is slow while hbase
accessing datanode, then it will be very straight forward.
And most of time, we find some region servers with high get latency or put
latency, however we could not find anything abnormal from datanode metrics. In
this case, either the issue is with that region server, or the issue is with a
remote datanode, and the remote datanode is not always slow(So we can’t tell
from datanode metrics)
In another shared hdfs case, due to the underlying hdfs is shared hdfs. It is
even harder to find out the slow datanode. Even we find out a bad datanode then
how to correlate that this datanode is the one datanode that impacted hbase
request and then impacted user requests to hbase. It is because a datanode is
serving a lot of traffic in same time, we can’t distinguish traffic from hbase
and other user. And if only hbase user is impacted due to like a sector on disk
is bad, while other users are fine because they don’t use this sector. This
case will be difficult to triage the issue, because we can’t find the path from
a specific hbase request —> datanode —> disk. In a slow read case, the path is
unvisiable to users and even to hbase admin. And if we have no clue which is
the slow datanode, then we can’t proceed.
IMO, It will make life easier if we have some kind of waring for us to alert
about possible slow datanode other than for us to guess which one is slow and
might contribute to the slow hbase performance.
When we talk about hdfs metrics, we will see 95th, 90th, mean, median etc.
Usually in case a datanode is only slow to access a small set of data(a small
number of bad sector on local disk), then this kind of issue will be ommited
and we can't see it from aggeragated metrics.
> add slow read block log entry to alert slow datanodeinfo when reading a block
> is slow
> -
>
> Key: HBASE-18764
> URL: https://issues.apache.org/jira/browse/HBASE-18764
> Project: HBase
> Issue Type: Improvement
> Components: HFile
>Affects Versions: 1.1.2
>Reporter: Wang, Xinglong
>Priority: Minor
> Attachments: HBASE-18764.rev1.1.2.patch
>
>
> HBASE is on top of HDFS and both are distributed system. HBASE will also get
> impacted when there is struggler datanode due to network/disk/cpu issue. All
> HBASE read/scan towards that datanode will be slowdown. It's not easy for
> hbase admin to find out the struggler datanode in such case.
> While we have a log entry known as slow sync. One such entry is like the
> following. It will help hbase admin to quickly identify the slow datanode in
> the pipline in case of network/disk/cup issue with one of the 3 datanods in
> pipeline.
> {noformat}
> 2017-07-08 19:11:30,538 INFO [sync.3] wal.FSHLog: Slow sync cost: 490189 ms,
> current pipeline:
> [DatanodeInfoWithStorage[xx.xx.xx.xx:50010,DS-c391299a-aa9f-4146-ac7e-a493ae536bff,DISK],
> DatanodeInfoWithSt
> orage[xx.xx.xx.xx:50010,DS-21a85f8b-f389-4f9e-95a8-b711945fd210,DISK],
> DatanodeInfoWithStorage[xx.xx.xx.xx:50010,DS-aa48cef2-3554-482f-b49d-be4763f4d8b8,DISK]]
> {noformat}
> Inspired by slow sync log entry, I think it will also be beneficial for us to
> print out such kind of entry when we encounter slow read case. So that it
> will be easy to identify the slow datanode.
--
This message was sent by Atlassian J
[jira] [Commented] (HBASE-13333) Renew Scanner Lease without advancing the RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158383#comment-16158383
]
Chia-Ping Tsai commented on HBASE-1:
{code}
$ git status
On branch branch-1.3
Your branch is up-to-date with 'origin/branch-1.3'.
nothing to commit, working tree clean
$ git log --grep=1
commit 0e76cf5be41485849c7330a0dec281cadad7d135
Author: Lars Hofhansl
Date: Sat May 2 17:52:27 2015 -0700
HBASE-1 Renew Scanner Lease without advancing the RegionScanner.
{code}
It was already in 1.3.
> Renew Scanner Lease without advancing the RegionScanner
> ---
>
> Key: HBASE-1
> URL: https://issues.apache.org/jira/browse/HBASE-1
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1
>
> Attachments: 1-0.98.txt, 1-master.txt
>
>
> We have a usecase (for Phoenix) where we want to let the server know that the
> client is still around. Like a client-side heartbeat.
> Doing a full heartbeat is complicated, but we could add the ability to make
> scanner call with caching set to 0. The server already does the right thing
> (it renews the lease, but does not advance the scanner).
> It looks like the client (ScannerCallable) also does the right thing. We
> cannot break ResultScanner before HBase 2.0, but we can add a renewLease()
> method to AbstractClientScaner. Phoenix (or any other caller) can then cast
> to ClientScanner and call that method to ensure we renew the lease on the
> server.
> It would be a simple and fully backwards compatible change. [~giacomotaylor]
> Comments?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xinxin fan updated HBASE-18090: --- Attachment: HBASE-18090-branch-1.3-v2.patch > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158394#comment-16158394 ] xinxin fan commented on HBASE-18090: patch v2 for review > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18778) Use Comparator for StealJobQueue
[
https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158397#comment-16158397
]
Hadoop QA commented on HBASE-18778:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
16s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 31s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Class org.apache.hadoop.hbase.util.StealJobQueue defines non-transient
non-serializable instance field notEmpty In StealJobQueue.java:instance field
notEmpty In StealJobQueue.java |
| Failed junit tests | hadoop.hbase.regionserver.TestCompactionWithCoprocessor |
| | hadoop.hbase.regionserver.TestCompaction |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18778 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886009/HBASE-18778.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b1fcd398f0e5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e69b05d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/8514/artifact/patchprocess/new-findbugs-hbase-server.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/8514/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8514/testReport/ |
| modules | C: hbase-server U: hbase-ser
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[
https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158399#comment-16158399
]
Hadoop QA commented on HBASE-18090:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HBASE-18090 does not apply to branch-1.3. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.4.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-18090 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886028/HBASE-18090-branch-1.3-v2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8517/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Improve TableSnapshotInputFormat to allow more multiple mappers per region
> --
>
> Key: HBASE-18090
> URL: https://issues.apache.org/jira/browse/HBASE-18090
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 1.4.0
>Reporter: Mikhail Antonov
>Assignee: xinxin fan
> Attachments: HBASE-18090-branch-1.3-v1.patch,
> HBASE-18090-branch-1.3-v2.patch
>
>
> TableSnapshotInputFormat runs one map task per region in the table snapshot.
> This places unnecessary restriction that the region layout of the original
> table needs to take the processing resources available to MR job into
> consideration. Allowing to run multiple mappers per region (assuming
> reasonably even key distribution) would be useful.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-13312) SmallScannerCallable does not increment scan metrics
[
https://issues.apache.org/jira/browse/HBASE-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158401#comment-16158401
]
Lars George commented on HBASE-13312:
-
Was that not committed to branch-1.3?
> SmallScannerCallable does not increment scan metrics
>
>
> Key: HBASE-13312
> URL: https://issues.apache.org/jira/browse/HBASE-13312
> Project: HBase
> Issue Type: Bug
> Components: Client, Scanners
>Affects Versions: 1.0.0
>Reporter: Lars George
>Assignee: Andrew Purtell
> Fix For: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1
>
> Attachments: HBASE-13312-0.98.patch, HBASE-13312.patch
>
>
> The subclass of {{ScannerCallable}}, called {{SmallScannerCallable}} seems to
> miss calling any of the increment methods of the {{ScanMetrics}}. The
> superclass does so, but the super methods are not invoked. It emits the
> metrics dutifully at the end of {{next()}}, but there are no useful numbers
> in it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18780) Remove HLogPrettyPrinter
Chia-Ping Tsai created HBASE-18780: -- Summary: Remove HLogPrettyPrinter Key: HBASE-18780 URL: https://issues.apache.org/jira/browse/HBASE-18780 Project: HBase Issue Type: Sub-task Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai Priority: Minor WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18780) Remove HLogPrettyPrinter
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18780: --- Labels: beginner (was: ) > Remove HLogPrettyPrinter > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18780) Remove HLogPrettyPrinter
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18780: --- Component/s: wal > Remove HLogPrettyPrinter > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158404#comment-16158404 ] Duo Zhang commented on HBASE-18778: --- For ThreadPoolExecutor, if there are available core threads, then the task will be executed directly without inserting into the queue, so I do not think the size of the queue should be calculated by thread count. Thanks. > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 3.0.0, 2.0.0-alpha-2 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18778.patch > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HBASE-14998) Unify synchronous and asynchronous methods in Admin and cleanup
[
https://issues.apache.org/jira/browse/HBASE-14998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-14998 started by Balazs Meszaros.
---
> Unify synchronous and asynchronous methods in Admin and cleanup
> ---
>
> Key: HBASE-14998
> URL: https://issues.apache.org/jira/browse/HBASE-14998
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Balazs Meszaros
>Priority: Blocker
> Fix For: 2.0.0-alpha-3
>
>
> Admin has a bunch of methods, some are asnyc, some are sync. Needs some
> unification in method naming, and method signatures.
> - We use modify and alter interchangeably. Pick one and stick with it
> (modifyTable(), versus getAlterStatus()). Shell uses {{alter}}.
> - Remove getAlterStatus(), should not be needed.
> - remove already deprecated methods
> - isTableAvailable(TableName tableName, byte[][] splitKeys) should be
> removed.
> - Consistently use Aysnc as a prefix for all async methods.
> - Other ideas?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18765) The value of balancerRan is true even though no plans are executed
[
https://issues.apache.org/jira/browse/HBASE-18765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158413#comment-16158413
]
Hudson commented on HBASE-18765:
FAILURE: Integrated in Jenkins build HBase-2.0 #477 (See
[https://builds.apache.org/job/HBase-2.0/477/])
HBASE-18765 The value of balancerRan is true even though no plans are
(chia7712: rev 7b2ce38b4bb714be508b39bc9f1f7354a1691884)
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
> The value of balancerRan is true even though no plans are executed
> --
>
> Key: HBASE-18765
> URL: https://issues.apache.org/jira/browse/HBASE-18765
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0-alpha-3
>
> Attachments: HBASE-18765.v0.patch
>
>
> {code}
> //We balance per group instead of per table
> List plans = new ArrayList<>();
> for(Map.Entry>> tableMap:
> getRSGroupAssignmentsByTable(groupName).entrySet()) {
> LOG.info("Creating partial plan for table " + tableMap.getKey() + ": "
> + tableMap.getValue());
> List partialPlans =
> balancer.balanceCluster(tableMap.getValue());
> LOG.info("Partial plan for table " + tableMap.getKey() + ": " +
> partialPlans);
> if (partialPlans != null) {
> plans.addAll(partialPlans);
> }
> }
> long startTime = System.currentTimeMillis();
> balancerRan = plans != null;
> {code}
> The *plans* is never null.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18778: -- Attachment: HBASE-18778-v1.patch Fix the fingbugs warnings and failed UTs. I misunderstood the priority of compaction... It should be less first... > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 3.0.0, 2.0.0-alpha-2 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18778.patch, HBASE-18778-v1.patch > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18771) Incorrect StoreFileRefresh leading to split and compaction failures
[ https://issues.apache.org/jira/browse/HBASE-18771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-18771: --- Attachment: HBASE-18771.master.001.patch Patch for master, we should fix refreshStoreFiles there too. > Incorrect StoreFileRefresh leading to split and compaction failures > --- > > Key: HBASE-18771 > URL: https://issues.apache.org/jira/browse/HBASE-18771 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.4.0, 1.3.2, 1.5.0 > > Attachments: HBASE-18771.branch-1.3.001.patch, > HBASE-18771.branch-1.3.002.patch, HBASE-18771.branch-1.3.003.patch, > HBASE-18771.master.001.patch > > > We ran into issues of compaction and split failures with 1.3 similar to > HBASE-18186 and HBASE-17406. Here's what i believe is happening - > Lets say we have 4 store files that are compacted to form a new one. At this > point we now have 5 store files, however only 1(the newly formed) is open now > for the store and rest are waiting to get archived by HFileArchiver > Now before the files are archived we get a FNFE in a scanner. This results in > HRegion.RegionScannerImpl.handleFileNotFound(FileNotFoundException fnfe) > being called which results in region.refreshStoreFiles(true) -> > HStore.refreshStoreFiles() > HStore.refreshStoreFiles now checks the hdfs dir and adds the previously > compacted files back to the store, however these files are also present in > StoreFileManager's compactedFiles list. Now at this point HFileArchiver runs, > checks compactedFiles list and moves these files into the archive directory. > Now when compaction runs it gets: > 2017-09-04 12:30:13,899 ERROR [ctions-1504505399609] > regionserver.CompactSplitThread - Compaction selection failed regionName = > , storeName = 0, priority = 26, time = 1504528213899 > java.io.FileNotFoundException: File does not exist: hdfs:// > at > org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1337) > at > org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1329) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1329) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:422) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360) > at > org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:325) > at > org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63) > at > org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:65) > at > org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82) > at > org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107) > at > org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1679) > Similarly if a split happens after archival we fail after PONR while opening > daughter regions due to FNFE. This results in parent offline and daughters > also in a limbo since they're unable to open. Since we get the error after > PONR we also end up aborting the RS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18106) Redo ProcedureInfo and LockInfo
[
https://issues.apache.org/jira/browse/HBASE-18106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158451#comment-16158451
]
Hadoop QA commented on HBASE-18106:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 27 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 75 new + 328 unchanged - 70 fixed
= 403 total (was 398) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
9s{color} | {color:red} The patch generated 78 new + 756 unchanged - 79 fixed =
834 total (was 835) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
41m 12s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
52s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}121m
25s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 21s{color}
| {color:red} hbase-shell in the patch f
[jira] [Updated] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xinxin fan updated HBASE-18090: --- Attachment: (was: HBASE-18090-branch-1.3-v2.patch) > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xinxin fan updated HBASE-18090: --- Attachment: HBASE-18090-branch-1.3-v2.patch > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158394#comment-16158394 ] xinxin fan edited comment on HBASE-18090 at 9/8/17 11:20 AM: - patch v2 for review was (Author: xinxin fan): patch v2 for review > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158394#comment-16158394 ] xinxin fan edited comment on HBASE-18090 at 9/8/17 11:25 AM: - patch v2 (rebased branch-1.3) for review was (Author: xinxin fan): patch v2 for review > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158505#comment-16158505 ] xinxin fan commented on HBASE-18090: rebase branch-1.3, patch v2 for review > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xinxin fan updated HBASE-18090: --- Comment: was deleted (was: patch v2 (rebased branch-1.3) for review) > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15607) Remove PB references from Admin for 2.0
[ https://issues.apache.org/jira/browse/HBASE-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158519#comment-16158519 ] ramkrishna.s.vasudevan commented on HBASE-15607: Pushed to branch-1.3 and branch-1.4 Thanks Stack. > Remove PB references from Admin for 2.0 > --- > > Key: HBASE-15607 > URL: https://issues.apache.org/jira/browse/HBASE-15607 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 2.0.0-alpha-3 > > Attachments: HBASE-15607_1.patch, HBASE-15607_2.patch, > HBASE-15607_3.patch, HBASE-15607_3.patch, HBASE-15607_4.patch, > HBASE-15607_4.patch, HBASE-15607_branch-1.3.patch, > HBASE-15607_branch-1.patch, HBASE-15607.patch > > > This is a sub-task for HBASE-15174. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18781) Accessing table.jsp directly returns Exception to user.
Janos Gub created HBASE-18781: - Summary: Accessing table.jsp directly returns Exception to user. Key: HBASE-18781 URL: https://issues.apache.org/jira/browse/HBASE-18781 Project: HBase Issue Type: Bug Reporter: Janos Gub Assignee: Janos Gub Discovered by [~stack] while accessing table.jsp directly. Moved a function up introduced in https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message to be more descriptive. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18781) Accessing table.jsp directly returns Exception to user.
[ https://issues.apache.org/jira/browse/HBASE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janos Gub updated HBASE-18781: -- Attachment: HBASE-18781.patch > Accessing table.jsp directly returns Exception to user. > --- > > Key: HBASE-18781 > URL: https://issues.apache.org/jira/browse/HBASE-18781 > Project: HBase > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HBASE-18781.patch > > > Discovered by [~stack] while accessing table.jsp directly. > Moved a function up introduced in > https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message > to be more descriptive. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18781) Accessing table.jsp directly returns Exception to user.
[ https://issues.apache.org/jira/browse/HBASE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janos Gub updated HBASE-18781: -- Status: Patch Available (was: Open) > Accessing table.jsp directly returns Exception to user. > --- > > Key: HBASE-18781 > URL: https://issues.apache.org/jira/browse/HBASE-18781 > Project: HBase > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HBASE-18781.patch > > > Discovered by [~stack] while accessing table.jsp directly. > Moved a function up introduced in > https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message > to be more descriptive. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18106) Redo ProcedureInfo and LockInfo
[ https://issues.apache.org/jira/browse/HBASE-18106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158526#comment-16158526 ] Balazs Meszaros commented on HBASE-18106: - Test runs locally. > Redo ProcedureInfo and LockInfo > --- > > Key: HBASE-18106 > URL: https://issues.apache.org/jira/browse/HBASE-18106 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: stack >Assignee: Balazs Meszaros >Priority: Critical > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18106.master.001.patch, > HBASE-18106.master.002.patch, HBASE-18106.master.003.patch, > HBASE-18106.master.004.patch, HBASE-18106.master.005.patch, > HBASE-18106.master.006.patch, HBASE-18106.master.007.patch, > HBASE-18106.master.008.patch, HBASE-18106.master.009.patch, > HBASE-18106.master.010.patch, HBASE-18106.master.011.patch, > HBASE-18106.master.011.patch, HBASE-18106.master.012.patch, > HBASE-18106.master.012.patch > > > ProcedureInfo was introduced as a lowest-common-denominator POJO that could > be used as a facade on PB Procedures. It was good for showing state of > Procedure framework in shell and UI. > Its a bit weird though. Its up in hbase-common rather than in Procedure and > it can only ever show a subset of the Procedure info. > I was thinking we could use the pb3.1 pb->JSON utility instead and emit a > JSON String wherever we need to export a view on procedure internals. > This issue is about exploring this possibility. Would depend on our having an > upgraded guava (so probably depends on the 'pre-build' project). > From ProcedureInfo and LockInfo need fixing in > https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.kid1jzo114xw -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18781) Accessing table.jsp directly returns Exception to user.
[ https://issues.apache.org/jira/browse/HBASE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janos Gub updated HBASE-18781: -- Description: Discovered by [~stack] while accessing table.jsp directly. Moved a function up introduced in https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message to be more descriptive. See more on HBASE-17436 was: Discovered by [~stack] while accessing table.jsp directly. Moved a function up introduced in https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message to be more descriptive. > Accessing table.jsp directly returns Exception to user. > --- > > Key: HBASE-18781 > URL: https://issues.apache.org/jira/browse/HBASE-18781 > Project: HBase > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HBASE-18781.patch > > > Discovered by [~stack] while accessing table.jsp directly. > Moved a function up introduced in > https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message > to be more descriptive. See more on HBASE-17436 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158527#comment-16158527 ] Janos Gub commented on HBASE-17436: --- Created separate ticket for the NPE. [~stack] could you please have a look on that too? Also, for me the tablesDetailed.jsp is working. Could you please help me out a bit how to reproduce this error? What user tables you have? I mean what are their name or any hint how to reproduce. > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: HBASE-17436.patch, HBASE-17436-v2.patch, > HBASE-17436-v5.patch, HBASE-17436-v6.patch, HBASE-17436-v6.patch, > HBASE-17436-v7.patch, HBASE-17436-v8.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, initial.patch, regionstates.png, Screen Shot 2017-04-24 > at 8.43.31 PM.png, Screen Shot 2017-04-26 at 4.39.49 PM.png, > tableregions.png, table-selected.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15607) Remove PB references from Admin for 2.0
[ https://issues.apache.org/jira/browse/HBASE-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158530#comment-16158530 ] Hudson commented on HBASE-15607: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #275 (See [https://builds.apache.org/job/HBase-1.3-JDK7/275/]) HBASE-15607 - Deprecating SnapShotInfo (Ram) (ramkrishna: rev ae6ff50dc4ff0ccae92b6ad3eed12916675920cc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java > Remove PB references from Admin for 2.0 > --- > > Key: HBASE-15607 > URL: https://issues.apache.org/jira/browse/HBASE-15607 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 2.0.0-alpha-3 > > Attachments: HBASE-15607_1.patch, HBASE-15607_2.patch, > HBASE-15607_3.patch, HBASE-15607_3.patch, HBASE-15607_4.patch, > HBASE-15607_4.patch, HBASE-15607_branch-1.3.patch, > HBASE-15607_branch-1.patch, HBASE-15607.patch > > > This is a sub-task for HBASE-15174. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158527#comment-16158527 ] Janos Gub edited comment on HBASE-17436 at 9/8/17 12:15 PM: Created separate ticket for the NPE. [~stack] could you please have a look on that too (https://issues.apache.org/jira/browse/HBASE-18781)? Also, for me the tablesDetailed.jsp is working. Could you please help me out a bit how to reproduce this error? What user tables you have? I mean what are their name or any hint how to reproduce. was (Author: gubjanos): Created separate ticket for the NPE. [~stack] could you please have a look on that too? Also, for me the tablesDetailed.jsp is working. Could you please help me out a bit how to reproduce this error? What user tables you have? I mean what are their name or any hint how to reproduce. > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: HBASE-17436.patch, HBASE-17436-v2.patch, > HBASE-17436-v5.patch, HBASE-17436-v6.patch, HBASE-17436-v6.patch, > HBASE-17436-v7.patch, HBASE-17436-v8.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, initial.patch, regionstates.png, Screen Shot 2017-04-24 > at 8.43.31 PM.png, Screen Shot 2017-04-26 at 4.39.49 PM.png, > tableregions.png, table-selected.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158527#comment-16158527 ] Janos Gub edited comment on HBASE-17436 at 9/8/17 12:20 PM: Created separate ticket for the NPE. [~stack] could you please have a look on that too (https://issues.apache.org/jira/browse/HBASE-18781 )? Also, for me the tablesDetailed.jsp is working. Could you please help me out a bit how to reproduce this error? What user tables you have? I mean what are their name or any hint how to reproduce. was (Author: gubjanos): Created separate ticket for the NPE. [~stack] could you please have a look on that too (https://issues.apache.org/jira/browse/HBASE-18781)? Also, for me the tablesDetailed.jsp is working. Could you please help me out a bit how to reproduce this error? What user tables you have? I mean what are their name or any hint how to reproduce. > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: HBASE-17436.patch, HBASE-17436-v2.patch, > HBASE-17436-v5.patch, HBASE-17436-v6.patch, HBASE-17436-v6.patch, > HBASE-17436-v7.patch, HBASE-17436-v8.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, initial.patch, regionstates.png, Screen Shot 2017-04-24 > at 8.43.31 PM.png, Screen Shot 2017-04-26 at 4.39.49 PM.png, > tableregions.png, table-selected.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18781) Accessing table.jsp directly returns Exception to user.
[ https://issues.apache.org/jira/browse/HBASE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janos Gub updated HBASE-18781: -- Description: Discovered by [~stack] while accessing table.jsp directly. Moved a function up introduced in https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message to be more descriptive. See more on https://issues.apache.org/jira/browse/HBASE-17436?focusedCommentId=16142040&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16142040 was: Discovered by [~stack] while accessing table.jsp directly. Moved a function up introduced in https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message to be more descriptive. See more on HBASE-17436 > Accessing table.jsp directly returns Exception to user. > --- > > Key: HBASE-18781 > URL: https://issues.apache.org/jira/browse/HBASE-18781 > Project: HBase > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HBASE-18781.patch > > > Discovered by [~stack] while accessing table.jsp directly. > Moved a function up introduced in > https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message > to be more descriptive. See more on > https://issues.apache.org/jira/browse/HBASE-17436?focusedCommentId=16142040&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16142040 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18753) Introduce the unsynchronized TimeRangeTracker
[
https://issues.apache.org/jira/browse/HBASE-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158544#comment-16158544
]
Hadoop QA commented on HBASE-18753:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 32s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}120m
33s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
18s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}197m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18753 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886023/HBASE-18753.v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f3911d1ff657 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e69b05d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8516/testReport/ |
| modules | C: hbase-common hbase-server hbase-mapreduce U: . |
| Console output |
https://builds.apache.org/job/PreCommit-H
[jira] [Commented] (HBASE-15607) Remove PB references from Admin for 2.0
[ https://issues.apache.org/jira/browse/HBASE-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158546#comment-16158546 ] Hudson commented on HBASE-15607: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #204 (See [https://builds.apache.org/job/HBase-1.3-IT/204/]) HBASE-15607 - Deprecating SnapShotInfo (Ram) (ramkrishna: rev ae6ff50dc4ff0ccae92b6ad3eed12916675920cc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java > Remove PB references from Admin for 2.0 > --- > > Key: HBASE-15607 > URL: https://issues.apache.org/jira/browse/HBASE-15607 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 2.0.0-alpha-3 > > Attachments: HBASE-15607_1.patch, HBASE-15607_2.patch, > HBASE-15607_3.patch, HBASE-15607_3.patch, HBASE-15607_4.patch, > HBASE-15607_4.patch, HBASE-15607_branch-1.3.patch, > HBASE-15607_branch-1.patch, HBASE-15607.patch > > > This is a sub-task for HBASE-15174. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18778) Use Comparator for StealJobQueue
[
https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158556#comment-16158556
]
Hadoop QA commented on HBASE-18778:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 46s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m
46s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18778 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886037/HBASE-18778-v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 66d04d91c4d1 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e69b05d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8518/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8518/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Use Comparator for StealJobQueue
>
>
> Key: HBASE-18778
> URL: https://issues.apache.org/jira/browse/HBASE-18778
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Affects Versions: 3.0.0, 2.0.0-alpha-2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 3.0.0, 2.0.0-alpha-3
>
> Attachments: HBASE-1
[jira] [Commented] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158565#comment-16158565 ] Duo Zhang commented on HBASE-18778: --- OK, all green. Any other concerns [~chia7712]? Thanks. > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 3.0.0, 2.0.0-alpha-2 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18778.patch, HBASE-18778-v1.patch > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18683) Upgrade hbase to commons-math 3
[
https://issues.apache.org/jira/browse/HBASE-18683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158576#comment-16158576
]
Hadoop QA commented on HBASE-18683:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hbase-resource-bundle . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
38m 41s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hbase-resource-bundle . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m
2s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
18s{color} | {color:green} hbase-resource-bundle in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 48s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
4s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}298m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestAvoidCellReferencesIntoShippedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18683 |
| JIRA Patch URL |
ht
[jira] [Commented] (HBASE-15607) Remove PB references from Admin for 2.0
[ https://issues.apache.org/jira/browse/HBASE-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158583#comment-16158583 ] Hudson commented on HBASE-15607: FAILURE: Integrated in Jenkins build HBase-1.4 #907 (See [https://builds.apache.org/job/HBase-1.4/907/]) HBASE-15607 - Deprecating SnapShotInfo (Ram) (ramkrishna: rev 93fbf25e3162d79a697450a64c7a56b56c3d3179) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java > Remove PB references from Admin for 2.0 > --- > > Key: HBASE-15607 > URL: https://issues.apache.org/jira/browse/HBASE-15607 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 2.0.0-alpha-3 > > Attachments: HBASE-15607_1.patch, HBASE-15607_2.patch, > HBASE-15607_3.patch, HBASE-15607_3.patch, HBASE-15607_4.patch, > HBASE-15607_4.patch, HBASE-15607_branch-1.3.patch, > HBASE-15607_branch-1.patch, HBASE-15607.patch > > > This is a sub-task for HBASE-15174. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18683) Upgrade hbase to commons-math 3
[ https://issues.apache.org/jira/browse/HBASE-18683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158584#comment-16158584 ] Peter Somogyi commented on HBASE-18683: --- TestAvoidCellReferencesIntoShippedBlocks passes locally. > Upgrade hbase to commons-math 3 > --- > > Key: HBASE-18683 > URL: https://issues.apache.org/jira/browse/HBASE-18683 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0-alpha-2 >Reporter: Peter Somogyi >Assignee: Peter Somogyi > Fix For: 2.0.0 > > Attachments: HBASE-18683.master.001.patch, > HBASE-18683.master.001.patch, HBASE-18683.master.002.patch, > HBASE-18683.master.003.patch, HBASE-18683.master.003.patch, LICENSE.txt, > LICENSE-v3, NOTICE.txt, NOTICE-v3 > > > Upgrade hbase to use commons-math 3.6. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158600#comment-16158600 ] Chia-Ping Tsai commented on HBASE-18778: +1 > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 3.0.0, 2.0.0-alpha-2 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18778.patch, HBASE-18778-v1.patch > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18771) Incorrect StoreFileRefresh leading to split and compaction failures
[
https://issues.apache.org/jira/browse/HBASE-18771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158614#comment-16158614
]
Hadoop QA commented on HBASE-18771:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 49s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 17s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestSplitLogWorker |
| | org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy
|
| | org.apache.hadoop.hbase.regionserver.TestCompaction |
| | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| | org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook |
| | org.apache.hadoop.hbase.regionserver.TestTimestampFilterSeekHint |
| | org.apache.hadoop.hbase.wal.TestWALFiltering |
| | org.apache.hadoop.hbase.master.TestGetInfoPort |
| | org.apache.hadoop.hbase.regionserver.TestColumnSeeking |
| | org.apache.hadoop.hbase.regionserver.TestRegionServerAbort |
| | org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort |
| | org.apache.hadoop.hbase.regionserver.TestRegionIncrement |
| | org.apache.hadoop.hbase.regionserver.TestWALLockup |
| | org.apache.hadoop.hbase.master.TestTableStateManager |
| | org.apache.hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush |
| | org.apache.hadoop.hbase.master.assignment.TestAssignmentOnRSCrash |
| | org.apache.hadoop.hbase.master.TestSplitLogManager |
| | org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport |
| | org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss |
| | org.apache.hadoop.hbase.master.TestRollingRestart |
| | org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalance
[jira] [Commented] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158626#comment-16158626 ] Duo Zhang commented on HBASE-18778: --- Will commit shortly. > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 3.0.0, 2.0.0-alpha-2 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18778.patch, HBASE-18778-v1.patch > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18780) Remove HLogPrettyPrinter
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18780: --- Hadoop Flags: Incompatible change > Remove HLogPrettyPrinter > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18780.v0.patch > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18780) Remove HLogPrettyPrinter
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18780: --- Attachment: HBASE-18780.v0.patch > Remove HLogPrettyPrinter > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18780.v0.patch > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18780) Remove HLogPrettyPrinter
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18780: --- Component/s: documentation > Remove HLogPrettyPrinter > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18780.v0.patch > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18765) The value of balancerRan is true even though no plans are executed
[
https://issues.apache.org/jira/browse/HBASE-18765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158631#comment-16158631
]
Hudson commented on HBASE-18765:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3679 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3679/])
HBASE-18765 The value of balancerRan is true even though no plans are
(chia7712: rev e69b05d109d4afa4bc6201197407e3f40044c522)
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
> The value of balancerRan is true even though no plans are executed
> --
>
> Key: HBASE-18765
> URL: https://issues.apache.org/jira/browse/HBASE-18765
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0-alpha-3
>
> Attachments: HBASE-18765.v0.patch
>
>
> {code}
> //We balance per group instead of per table
> List plans = new ArrayList<>();
> for(Map.Entry>> tableMap:
> getRSGroupAssignmentsByTable(groupName).entrySet()) {
> LOG.info("Creating partial plan for table " + tableMap.getKey() + ": "
> + tableMap.getValue());
> List partialPlans =
> balancer.balanceCluster(tableMap.getValue());
> LOG.info("Partial plan for table " + tableMap.getKey() + ": " +
> partialPlans);
> if (partialPlans != null) {
> plans.addAll(partialPlans);
> }
> }
> long startTime = System.currentTimeMillis();
> balancerRan = plans != null;
> {code}
> The *plans* is never null.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18780) Remove HLogPrettyPrinter and hlog command
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18780: --- Status: Patch Available (was: Open) > Remove HLogPrettyPrinter and hlog command > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18780.v0.patch > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18780) Remove HLogPrettyPrinter and hlog command
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18780: --- Summary: Remove HLogPrettyPrinter and hlog command (was: Remove HLogPrettyPrinter ) > Remove HLogPrettyPrinter and hlog command > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18780.v0.patch > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18778) Use Comparator for StealJobQueue
[ https://issues.apache.org/jira/browse/HBASE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18778: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks [~chia7712] for reviewing. > Use Comparator for StealJobQueue > > > Key: HBASE-18778 > URL: https://issues.apache.org/jira/browse/HBASE-18778 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 3.0.0, 2.0.0-alpha-2 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18778.patch, HBASE-18778-v1.patch > > > To eliminate the confusing compareTo and equals implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18769) Make CompareFilter use generic CompareOperator instead of internal enum
[
https://issues.apache.org/jira/browse/HBASE-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158696#comment-16158696
]
Hadoop QA commented on HBASE-18769:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
44s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
30s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 24s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
13s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}146m
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
7s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
55s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
30s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}157m 24s{color}
| {color:red} root in the patch failed. {color} |
| {color:gre
[jira] [Commented] (HBASE-15607) Remove PB references from Admin for 2.0
[ https://issues.apache.org/jira/browse/HBASE-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158746#comment-16158746 ] Hudson commented on HBASE-15607: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #285 (See [https://builds.apache.org/job/HBase-1.3-JDK8/285/]) HBASE-15607 - Deprecating SnapShotInfo (Ram) (ramkrishna: rev ae6ff50dc4ff0ccae92b6ad3eed12916675920cc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java > Remove PB references from Admin for 2.0 > --- > > Key: HBASE-15607 > URL: https://issues.apache.org/jira/browse/HBASE-15607 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 2.0.0-alpha-3 > > Attachments: HBASE-15607_1.patch, HBASE-15607_2.patch, > HBASE-15607_3.patch, HBASE-15607_3.patch, HBASE-15607_4.patch, > HBASE-15607_4.patch, HBASE-15607_branch-1.3.patch, > HBASE-15607_branch-1.patch, HBASE-15607.patch > > > This is a sub-task for HBASE-15174. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18298) RegionServerServices Interface cleanup for CP expose
[ https://issues.apache.org/jira/browse/HBASE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-18298: --- Attachment: HBASE-18298_V4.patch > RegionServerServices Interface cleanup for CP expose > > > Key: HBASE-18298 > URL: https://issues.apache.org/jira/browse/HBASE-18298 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18298.patch, HBASE-18298_V2.patch, > HBASE-18298_V3.patch, HBASE-18298_V4.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18781) Accessing table.jsp directly returns Exception to user.
[
https://issues.apache.org/jira/browse/HBASE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158759#comment-16158759
]
Hadoop QA commented on HBASE-18781:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
56s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}110m
16s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886061/HBASE-18781.patch |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux 8ed530a36d94 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e69b05d |
| Default Java | 1.8.0_144 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8521/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8521/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Accessing table.jsp directly returns Exception to user.
> ---
>
> Key: HBASE-18781
> URL: https://issues.apache.org/jira/browse/HBASE-18781
> Project: HBase
> Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HBASE-18781.patch
>
>
> Discovered by [~stack] while accessing table.jsp directly.
> Moved a function up introduced in
> https://issues.apache.org/jira/browse/HBASE-15675. Also changed error message
> to be more descriptive. See more on
> https://issues.apache.org/jira/browse/HBASE-17436?focusedCommentId=16142040&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16142040
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18764) add slow read block log entry to alert slow datanodeinfo when reading a block is slow
[
https://issues.apache.org/jira/browse/HBASE-18764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158789#comment-16158789
]
Josh Elser commented on HBASE-18764:
Thanks for the reply!
This is definitely one of those tricky kinds of situations. To use your same
argument against you, RS logging would still not give us a definitive clue as
to where the problem lies, just a hint of where to look. When a RegionServer is
processing 15+ RPCs at one point in time, it requires a bit of human effort to
figure out which handler thread was actually tied to the user whose request
missed the SLA. That's the crux of why I suggested looking at HDFS metrics
would be a better approach.
To that point, if you want to figure out why some specific request was slow,
and specifically what part of the request was slow, we'd instead want to use
something like HTrace to aggregate this. However, in practice, this too is
difficult due to the overhead of the tracing as well as the intermittent
slowness that typically comes with these kinds of problems.
bq. When we talk about hdfs metrics, we will see 95th, 90th, mean, median etc
Yep, that is the downside of using aggregations like percentiles :). In most
cases, we know that we don't want to look at tail of operation times because of
"expected" slowness (e.g. JVM garbage collection or compactions); however,
there are certainly cases when looking at all of the metrics is necessary to
identify a culprit. Specifically, the case you have identified when a small
percent of requests to HDFS or, even worse, a small percent of requests to a
DataNode fall outside the bounds of normalcy.
At the end of the day, I'm not trying to levy a veto on this kind of change,
but instead trying to outline some cases in which this will just have you
blocked on the next question :). Since I've already spent this much time
thinking about the subject, let me look closer at your patch too:
{noformat}
+ this.conf = new HBaseConfiguration();
{noformat}
The no-args constructor is deprecated.
{noformat}
+ this.slowReadNs =
+100 * conf.getInt("hbase.regionserver.hlog.slowsync.ms",
+DEFAULT_SLOW_READ_TIME_MS);
{noformat}
Create a new configuration property, don't reuse the slowsync key.
{noformat}
+ long timeInNanos = System.nanoTime() - startNano;
+ if (timeInNanos > this.slowReadNs) {
{noformat}
Nit: combine these two lines
{noformat}
+ DFSInputStream dfsInputStream = (DFSInputStream)
is.getWrappedStream();
{noformat}
You need to check if this is an instance of {{DFSInputStream}} instead of
blindly casting it.
{noformat}
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
{noformat}
We do not print to stdout/stderr directly. You use a logger to do that. In this
case, since this is not a critical codepath, I'd suggest you make a {{debug}}
log call, with a message making clear that the exception was caught trying to
collect this slowread information. We want to make sure that if/when this block
happens, an admin/user would not interpret the exception as something they need
to take action on.
> add slow read block log entry to alert slow datanodeinfo when reading a block
> is slow
> -
>
> Key: HBASE-18764
> URL: https://issues.apache.org/jira/browse/HBASE-18764
> Project: HBase
> Issue Type: Improvement
> Components: HFile
>Affects Versions: 1.1.2
>Reporter: Wang, Xinglong
>Priority: Minor
> Attachments: HBASE-18764.rev1.1.2.patch
>
>
> HBASE is on top of HDFS and both are distributed system. HBASE will also get
> impacted when there is struggler datanode due to network/disk/cpu issue. All
> HBASE read/scan towards that datanode will be slowdown. It's not easy for
> hbase admin to find out the struggler datanode in such case.
> While we have a log entry known as slow sync. One such entry is like the
> following. It will help hbase admin to quickly identify the slow datanode in
> the pipline in case of network/disk/cup issue with one of the 3 datanods in
> pipeline.
> {noformat}
> 2017-07-08 19:11:30,538 INFO [sync.3] wal.FSHLog: Slow sync cost: 490189 ms,
> current pipeline:
> [DatanodeInfoWithStorage[xx.xx.xx.xx:50010,DS-c391299a-aa9f-4146-ac7e-a493ae536bff,DISK],
> DatanodeInfoWithSt
> orage[xx.xx.xx.xx:50010,DS-21a85f8b-f389-4f9e-95a8-b711945fd210,DISK],
> DatanodeInfoWithStorage[xx.xx.xx.xx:50010,DS-aa48cef2-3554-482f-b49d-be4763f4d8b8,DISK]]
> {noformat}
> Inspired by slow sync log entry, I think it will also be beneficial for us to
> print out such kind of entry when we encounter slow read case. So that it
> will be easy to identify the slow datanode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029
[jira] [Commented] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158898#comment-16158898 ] Appy commented on HBASE-15809: -- I have a great UI for monitoring and javascript for it, but it needs a backend for metrics collection. Unfortunately, don't have bandwidth for it right now. > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.5.0 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-13333) Renew Scanner Lease without advancing the RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-1: -- Release Note: Adds a renewLease call to ClientScanner > Renew Scanner Lease without advancing the RegionScanner > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 0.98.13, 1.0.2, 1.2.0, 1.1.1 > > Attachments: 1-0.98.txt, 1-master.txt > > > We have a usecase (for Phoenix) where we want to let the server know that the > client is still around. Like a client-side heartbeat. > Doing a full heartbeat is complicated, but we could add the ability to make > scanner call with caching set to 0. The server already does the right thing > (it renews the lease, but does not advance the scanner). > It looks like the client (ScannerCallable) also does the right thing. We > cannot break ResultScanner before HBase 2.0, but we can add a renewLease() > method to AbstractClientScaner. Phoenix (or any other caller) can then cast > to ClientScanner and call that method to ensure we renew the lease on the > server. > It would be a simple and fully backwards compatible change. [~giacomotaylor] > Comments? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17980) Any HRegionInfo we give out should be immutable
[ https://issues.apache.org/jira/browse/HBASE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158920#comment-16158920 ] stack commented on HBASE-17980: --- [~brandboat] Can MutableRI be Audience Private? Can we get away with HRI and RI being Audience public only? Do we have to let MutableRI be Public? Thank you for working on this real nice cleanup. > Any HRegionInfo we give out should be immutable > --- > > Key: HBASE-17980 > URL: https://issues.apache.org/jira/browse/HBASE-17980 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Kuan-Po Tseng > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-17980.master.v0.patch, HBASE-17980.master.v1.patch > > > This is similar to HBASE-15583. > # Introduce RegionInfo class. HRegionInfo will extend RegionInfo. > # Deprecate HRegionInfo to be removed in 3.0 > # RegionInfo contain all of the read-only methods of HRegionInfo > # Add "RegionInfo Builder" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18780) Remove HLogPrettyPrinter and hlog command
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158936#comment-16158936 ] stack commented on HBASE-18780: --- +1 > Remove HLogPrettyPrinter and hlog command > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18780.v0.patch > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18769) Make CompareFilter use generic CompareOperator instead of internal enum
[ https://issues.apache.org/jira/browse/HBASE-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18769: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks for the reviews lads. Helped. > Make CompareFilter use generic CompareOperator instead of internal enum > --- > > Key: HBASE-18769 > URL: https://issues.apache.org/jira/browse/HBASE-18769 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18769.branch-2.001.patch, > HBASE-18769.branch-2.002.patch, HBASE-18769.branch-2.003.patch > > > Should probably do this so only the one enum in the codebase. Means a bunch > of boilerplate deprecating the old enum for the new. Suggested by > [~anoop.hbase]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18779) Move CompareOperator to hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-18779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-18779. --- Resolution: Fixed Pushed to master and branch-2. Moved o.a.h.h.CompareOperator from hbase-common to hbase-client. > Move CompareOperator to hbase-client module > --- > > Key: HBASE-18779 > URL: https://issues.apache.org/jira/browse/HBASE-18779 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-alpha-3 > > > Comes of [~enis] review over in HBASE-14997. Do this review feedback after > HBASE-18769 goes in. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18780) Remove HLogPrettyPrinter and hlog command
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18780: -- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Status: Resolved (was: Patch Available) Pushed to master and branch-2. > Remove HLogPrettyPrinter and hlog command > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18780.v0.patch > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18780) Remove HLogPrettyPrinter and hlog command
[ https://issues.apache.org/jira/browse/HBASE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158977#comment-16158977 ] Chia-Ping Tsai commented on HBASE-18780: Thanks for the reviews. [~stack] > Remove HLogPrettyPrinter and hlog command > - > > Key: HBASE-18780 > URL: https://issues.apache.org/jira/browse/HBASE-18780 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Labels: beginner > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18780.v0.patch > > > WALPrettyPrinter was substituted for HLogPrettyPrinter. Remove it for 2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18773) Add utility to determine if a TableName is a meta table
[ https://issues.apache.org/jira/browse/HBASE-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158989#comment-16158989 ] Zach York commented on HBASE-18773: --- bq. Yeah. That looks about right. Make them the same. [~stack] I think there is some miscommunication here. While I think that making a dynamic way of accessing this makes sense, I also need to have both implementations of this function. (1) One that checks that this is the correct meta table name EXACTLY (including meta suffix if applicable) and (2) one that checks if it is a meta table (without any suffixes applied). I can change the funtionality of isMeta() and isMetaTableName() to (1) and then create a new method isMetaTableNameWithSuffix() for (2). What do you think of this approach? > Add utility to determine if a TableName is a meta table > --- > > Key: HBASE-18773 > URL: https://issues.apache.org/jira/browse/HBASE-18773 > Project: HBase > Issue Type: Sub-task >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18773.HBASE-18477.001.patch > > > HBASE-18444 adds a method of specifying a meta table suffix. To continue work > on HBASE-18477, we need a way to determine if a TableName is a meta table. > This patch adds this method and a unit test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158998#comment-16158998
]
Yi Liang commented on HBASE-16894:
--
Yes, and create splits for live table does not deal with HDFS
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18753) Introduce the unsynchronized TimeRangeTracker
[ https://issues.apache.org/jira/browse/HBASE-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18753: --- Status: Open (was: Patch Available) > Introduce the unsynchronized TimeRangeTracker > - > > Key: HBASE-18753 > URL: https://issues.apache.org/jira/browse/HBASE-18753 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18753.v0.patch, HBASE-18753.v1.patch, > HBASE-18753.v2.patch > > > If HBASE-18752 is pushed, it will be better to introduce the unsync > TimeRangeTracker for reducing the cost of recalculation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158998#comment-16158998
]
Yi Liang edited comment on HBASE-16894 at 9/8/17 5:51 PM:
--
Yes, and create splits for live table does not deal with WALs
was (Author: easyliangjob):
Yes, and create splits for live table does not deal with HDFS
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17980) Any HRegionInfo we give out should be immutable
[ https://issues.apache.org/jira/browse/HBASE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16159008#comment-16159008 ] Kuan-Po Tseng commented on HBASE-17980: --- Hello stack, 1. Yes, MutableRI is set to IA.Private. 2. I will correct the ImmutableRI to be IA.Private. Sorry about this mistake. 3. MutableRI is public because HRI delegate all things to MutableRI in this patch, if not so, we have to let MutableRI be private. Do you think let MutabableRI be private is better ??? > Any HRegionInfo we give out should be immutable > --- > > Key: HBASE-17980 > URL: https://issues.apache.org/jira/browse/HBASE-17980 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Kuan-Po Tseng > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-17980.master.v0.patch, HBASE-17980.master.v1.patch > > > This is similar to HBASE-15583. > # Introduce RegionInfo class. HRegionInfo will extend RegionInfo. > # Deprecate HRegionInfo to be removed in 3.0 > # RegionInfo contain all of the read-only methods of HRegionInfo > # Add "RegionInfo Builder" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16159007#comment-16159007 ] Ted Yu commented on HBASE-18090: This is a relatively large patch. Can you create patch for master branch and put it on review board ? > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18753) Introduce the unsynchronized TimeRangeTracker
[ https://issues.apache.org/jira/browse/HBASE-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18753: --- Attachment: HBASE-18753.v3.patch v3: AsyncTimeRangeTracker -> NonSyncTimeRangeTracker > Introduce the unsynchronized TimeRangeTracker > - > > Key: HBASE-18753 > URL: https://issues.apache.org/jira/browse/HBASE-18753 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18753.v0.patch, HBASE-18753.v1.patch, > HBASE-18753.v2.patch, HBASE-18753.v3.patch > > > If HBASE-18752 is pushed, it will be better to introduce the unsync > TimeRangeTracker for reducing the cost of recalculation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
