[jira] [Updated] (HBASE-14876) Provide maven archetypes
[ https://issues.apache.org/jira/browse/HBASE-14876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Vimont updated HBASE-14876: -- Attachment: archetype_shaded_prototype01.zip A new prototype of an archetype, this one to fulfill the second requirement: to produce client application with dependency upon *hbase-shaded-client*. This one may also be installed in a sandbox environment to work with, by following the same instructions that were listed above for the first prototype. > Provide maven archetypes > > > Key: HBASE-14876 > URL: https://issues.apache.org/jira/browse/HBASE-14876 > Project: HBase > Issue Type: New Feature > Components: build, Usability >Reporter: Nick Dimiduk >Assignee: Daniel Vimont > Labels: beginner > Attachments: archetype_prototype.zip, archetype_prototype02.zip, > archetype_shaded_prototype01.zip > > > To help onboard new users, we should provide maven archetypes for hbase > client applications. Off the top of my head, we should have templates for > - hbase client application with all dependencies > - hbase client application using client-shaded jar > - mapreduce application with hbase as input and output (ie, copy table) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041234#comment-15041234
]
Duo Zhang edited comment on HBASE-14790 at 12/4/15 8:02 AM:
Considering these features:
Hflush is much faster than hsync, especially in pipeline mode. So we have to
use hflush for hbase writing.
The data in DN that is hflushed but not hsynced may only in memory not disk,
but it can be read by client.
So if we hflush data to DNs, and it is read by ReplicationSource and
transferred to slave cluster, then three DNs and RS in master cluster crash.
And after replaying WALs, slave will have data that master loses...
The only way to prevent any data losses is hsync every time but it is too slow,
and I think most users can bear data lose to speed up writing operation but can
not bear slave has more data than master.
Therefore, I think we can do these:
hflush every time, not fsync;
hsync periodically, for example, default per 1000ms? It can be configured by
users, and users can also configure that we every each time, so there will not
have any data loses unless all DNs disk fail...
RS tells "acked length" to ReplicationSource which is the data we hsynced, not
hflushed.
ReplicationSource only transfer data which is not larger than acked length. So
the slave cluster will never have inconsistency.
WAL reading can handle duplicate entries.
On WAL logging, if we get error on hflush, we open a new file and rewrite this
entry, and recover/hsync/close old file asynchronously.
was (Author: yangzhe1991):
Considering these features:
Hflush is much faster than hsync, especially in pipeline mode. So we have to
use hflush for hbase writing.
The data in DN that is hflushed but not hsynced may only in memory not disk,
but it can be read by client.
So if we hflush data to DNs, and it is read by ReplicationSource and
transferred to slave cluster, then three DNs and RS in master cluster crash.
And after replaying WALs, slave will have data that master loses...
The only way to prevent any data losses is hsync every time but it is too slow,
and I think most users can bear data lose to speed up writing operation but can
not bear slave has more data than master.
Therefore, I think we can do these:
hflush every time, not fsync;
hfsync periodically, for example, default per 1000ms? It can be configured by
users, and users can also configure that we hfsync each time, so there will not
have any data loses unless all DNs disk fail...
RS tells "acked length" to ReplicationSource which is the data we hsynced, not
hflushed.
ReplicationSource only transfer data which is not larger than acked length. So
the slave cluster will never have inconsistency.
WAL reading can handle duplicate entries.
On WAL logging, if we get error on hflush, we open a new file and rewrite this
entry, and recover/hsync/close old file asynchronously.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041234#comment-15041234
]
Duo Zhang edited comment on HBASE-14790 at 12/4/15 8:06 AM:
Considering these features:
Hflush is much faster than hsync, especially in pipeline mode. So we have to
use hflush for hbase writing.
The data in DN that is hflushed but not hsynced may only in memory not disk,
but it can be read by client.
So if we hflush data to DNs, and it is read by ReplicationSource and
transferred to slave cluster, then three DNs and RS in master cluster crash.
And after replaying WALs, slave will have data that master loses...
The only way to prevent any data losses is hsync every time but it is too slow,
and I think most users can bear data lose to speed up writing operation but can
not bear slave has more data than master.
Therefore, I think we can do these:
hflush every time, not hsync;
hsync periodically, for example, default per 1000ms? It can be configured by
users, and users can also configure that we hsync every time, so there will not
have any data loses unless all DNs disk fail...
RS tells "acked length" to ReplicationSource which is the data we hsynced, not
hflushed.
ReplicationSource only transfer data which is not larger than acked length. So
the slave cluster will never have inconsistency.
WAL reading can handle duplicate entries.
On WAL logging, if we get error on hflush, we open a new file and rewrite this
entry, and recover/hsync/close old file asynchronously.
was (Author: yangzhe1991):
Considering these features:
Hflush is much faster than hsync, especially in pipeline mode. So we have to
use hflush for hbase writing.
The data in DN that is hflushed but not hsynced may only in memory not disk,
but it can be read by client.
So if we hflush data to DNs, and it is read by ReplicationSource and
transferred to slave cluster, then three DNs and RS in master cluster crash.
And after replaying WALs, slave will have data that master loses...
The only way to prevent any data losses is hsync every time but it is too slow,
and I think most users can bear data lose to speed up writing operation but can
not bear slave has more data than master.
Therefore, I think we can do these:
hflush every time, not fsync;
hsync periodically, for example, default per 1000ms? It can be configured by
users, and users can also configure that we hsync every time, so there will not
have any data loses unless all DNs disk fail...
RS tells "acked length" to ReplicationSource which is the data we hsynced, not
hflushed.
ReplicationSource only transfer data which is not larger than acked length. So
the slave cluster will never have inconsistency.
WAL reading can handle duplicate entries.
On WAL logging, if we get error on hflush, we open a new file and rewrite this
entry, and recover/hsync/close old file asynchronously.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041234#comment-15041234
]
Duo Zhang edited comment on HBASE-14790 at 12/4/15 8:06 AM:
Considering these features:
Hflush is much faster than hsync, especially in pipeline mode. So we have to
use hflush for hbase writing.
The data in DN that is hflushed but not hsynced may only in memory not disk,
but it can be read by client.
So if we hflush data to DNs, and it is read by ReplicationSource and
transferred to slave cluster, then three DNs and RS in master cluster crash.
And after replaying WALs, slave will have data that master loses...
The only way to prevent any data losses is hsync every time but it is too slow,
and I think most users can bear data lose to speed up writing operation but can
not bear slave has more data than master.
Therefore, I think we can do these:
hflush every time, not fsync;
hsync periodically, for example, default per 1000ms? It can be configured by
users, and users can also configure that we hsync every time, so there will not
have any data loses unless all DNs disk fail...
RS tells "acked length" to ReplicationSource which is the data we hsynced, not
hflushed.
ReplicationSource only transfer data which is not larger than acked length. So
the slave cluster will never have inconsistency.
WAL reading can handle duplicate entries.
On WAL logging, if we get error on hflush, we open a new file and rewrite this
entry, and recover/hsync/close old file asynchronously.
was (Author: yangzhe1991):
Considering these features:
Hflush is much faster than hsync, especially in pipeline mode. So we have to
use hflush for hbase writing.
The data in DN that is hflushed but not hsynced may only in memory not disk,
but it can be read by client.
So if we hflush data to DNs, and it is read by ReplicationSource and
transferred to slave cluster, then three DNs and RS in master cluster crash.
And after replaying WALs, slave will have data that master loses...
The only way to prevent any data losses is hsync every time but it is too slow,
and I think most users can bear data lose to speed up writing operation but can
not bear slave has more data than master.
Therefore, I think we can do these:
hflush every time, not fsync;
hsync periodically, for example, default per 1000ms? It can be configured by
users, and users can also configure that we every each time, so there will not
have any data loses unless all DNs disk fail...
RS tells "acked length" to ReplicationSource which is the data we hsynced, not
hflushed.
ReplicationSource only transfer data which is not larger than acked length. So
the slave cluster will never have inconsistency.
WAL reading can handle duplicate entries.
On WAL logging, if we get error on hflush, we open a new file and rewrite this
entry, and recover/hsync/close old file asynchronously.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14876) Provide maven archetypes
[ https://issues.apache.org/jira/browse/HBASE-14876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041244#comment-15041244 ] Daniel Vimont commented on HBASE-14876: --- This comment refers to the zip file I just uploaded: archetype_shaded_prototype01.zip > Provide maven archetypes > > > Key: HBASE-14876 > URL: https://issues.apache.org/jira/browse/HBASE-14876 > Project: HBase > Issue Type: New Feature > Components: build, Usability >Reporter: Nick Dimiduk >Assignee: Daniel Vimont > Labels: beginner > Attachments: archetype_prototype.zip, archetype_prototype02.zip, > archetype_shaded_prototype01.zip > > > To help onboard new users, we should provide maven archetypes for hbase > client applications. Off the top of my head, we should have templates for > - hbase client application with all dependencies > - hbase client application using client-shaded jar > - mapreduce application with hbase as input and output (ie, copy table) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
[
https://issues.apache.org/jira/browse/HBASE-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041249#comment-15041249
]
Hudson commented on HBASE-14904:
SUCCESS: Integrated in HBase-1.1-JDK8 #1700 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1700/])
HBASE-14904 Mark Base[En|De]coder LimitedPrivate and fix binary compat (enis:
rev f3d3bd9d3b8eca176166391ef078391816b34bed)
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseEncoder.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseDecoder.java
> Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
>
>
> Key: HBASE-14904
> URL: https://issues.apache.org/jira/browse/HBASE-14904
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: hbase-14904_v1.patch, hbase-14904_v2.patch
>
>
> PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary
> compatibility in Phoenix compiled with earlier versions of HBase and run
> agains later versions.
> This is one of the areas that the boundary is not clear, but it won't hurt us
> to fix it.
> The exception trace is:
> {code}
> Exception in thread "main" java.lang.NoSuchFieldError: in
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
> {code}
> Although {{BaseDecoder.in}} is still there, it got changed to be a class
> rather than an interface. BaseDecoder is marked Private, thus the binary
> compat check is not run at all. Not sure whether it would have caught this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14905) VerifyReplication does not honour versions option
[
https://issues.apache.org/jira/browse/HBASE-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041263#comment-15041263
]
Hudson commented on HBASE-14905:
FAILURE: Integrated in HBase-1.2 #421 (See
[https://builds.apache.org/job/HBase-1.2/421/])
HBASE-14905 VerifyReplication does not honour versions option (Vishal (tedyu:
rev eb777ef289827ae385c0cae71ea64cd6618e14af)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
> VerifyReplication does not honour versions option
> -
>
> Key: HBASE-14905
> URL: https://issues.apache.org/jira/browse/HBASE-14905
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Affects Versions: 2.0.0, 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14905-v2.txt, HBASE-14905.patch, HBASE-14905_v3.patch,
> HBASE-14905_v4.patch, test.patch
>
>
> source:
> hbase(main):001:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
>
> target:
> hbase(main):023:0> scan 't1', {RAW => true, VERSIONS => 100}
> ROW COLUMN+CELL
>
>
> r1 column=f1:, timestamp=1449030102091,
> value=value1112
>
> r1 column=f1:, timestamp=1449030090758,
> value=value1112
>
> r1 column=f1:, timestamp=1449029984282,
> value=value
>
> r1 column=f1:, timestamp=1449029774173,
> value=value1001
>
> r1 column=f1:, timestamp=1449029709974,
> value=value1002
> /bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --versions=100 1 t1
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters
> GOODROWS=1
> Does not show any mismatch. Ideally it should show. This is because in
> VerifyReplication Class maxVersion is not correctly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
[
https://issues.apache.org/jira/browse/HBASE-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041264#comment-15041264
]
Hudson commented on HBASE-14904:
FAILURE: Integrated in HBase-1.2 #421 (See
[https://builds.apache.org/job/HBase-1.2/421/])
HBASE-14904 Mark Base[En|De]coder LimitedPrivate and fix binary compat (enis:
rev a75a93f98ca003a172cc966464308b013b1769e4)
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseDecoder.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseEncoder.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
> Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
>
>
> Key: HBASE-14904
> URL: https://issues.apache.org/jira/browse/HBASE-14904
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: hbase-14904_v1.patch, hbase-14904_v2.patch
>
>
> PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary
> compatibility in Phoenix compiled with earlier versions of HBase and run
> agains later versions.
> This is one of the areas that the boundary is not clear, but it won't hurt us
> to fix it.
> The exception trace is:
> {code}
> Exception in thread "main" java.lang.NoSuchFieldError: in
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
> {code}
> Although {{BaseDecoder.in}} is still there, it got changed to be a class
> rather than an interface. BaseDecoder is marked Private, thus the binary
> compat check is not run at all. Not sure whether it would have caught this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14927) Backport HBASE-13014 and HBASE-14749 to branch-1 and 0.98
Abhishek Singh Chouhan created HBASE-14927: -- Summary: Backport HBASE-13014 and HBASE-14749 to branch-1 and 0.98 Key: HBASE-14927 URL: https://issues.apache.org/jira/browse/HBASE-14927 Project: HBase Issue Type: Improvement Reporter: Abhishek Singh Chouhan Assignee: Abhishek Singh Chouhan Fix For: 1.3.0, 0.98.17 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14873) Problems around BoundedByteBufferPool providing direct buffers
[
https://issues.apache.org/jira/browse/HBASE-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041304#comment-15041304
]
Hiroshi Ikeda commented on HBASE-14873:
---
In practice the most problematic thing seems unlimited Unsafe.copyMemory if we
worry about stop-the-world. Unsafe is mysteriously dangerous and I doubt the
performance improvement is practically effective at the cost of uncertain risk.
> Problems around BoundedByteBufferPool providing direct buffers
> --
>
> Key: HBASE-14873
> URL: https://issues.apache.org/jira/browse/HBASE-14873
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: HBASE-14873-V2.patch, HBASE-14873.patch,
> HBASE-14873.patch, HBASE-14873.patch
>
>
> HBASE-13819 made BoundedByteBufferPool provide direct buffers.
> See RpcServer.java:
> {code}
> ...
> class Call implements RpcCallContext {
> protected synchronized void setResponse(...) {
> ...
> this.cellBlock = ipcUtil.buildCellBlock(..., reservoir);
> ...
> bc = new BufferChain(..., this.cellBlock);
> if (connection.useWrap) {
> bc = wrapWithSasl(bc);
> }
> ...
> private BufferChain wrapWithSasl(BufferChain bc) throws IOException {
> ...
> byte[] responseBytes = bc.getBytes();
> ...
> {code}
> {{cellBlock}} is expected to be a direct buffer retrieved from {{reservoir}}
> (but not always), and {{bc}} may be composed of both direct and non-direct
> buffers.
> And then, see BufferChain.java:
> {code}
> byte [] getBytes() {
> ...
> for (ByteBuffer bb: this.buffers) {
> System.arraycopy(bb.array(), ...);
> {code}
> A direct buffer doesn't give its array, and will throw
> UnsupportedOperationException.
> Another problem; {{cellBlock}} is allowed to be a non-direct buffer, and
> after use it will be put to {{reservoir}}, mixing direct and non-direct
> buffers in the pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041307#comment-15041307
]
Heng Chen commented on HBASE-14790:
---
{quote}
hsync periodically, for example, default per 1000ms? It can be configured by
users, and users can also configure that we hsync every time, so there will not
have any data loses unless all DNs disk fail...
{quote}
Sounds reasonable. we update 'acked length' after hsync, I think 'acked
length' logical is completed. Any thoughts?
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041321#comment-15041321
]
Duo Zhang commented on HBASE-14790:
---
{quote}
This is clean up of a broken WAL? This is being able to ask each DN what it
thinks the length is? While this is going on, we would be holding on to the
hbase handlers not letting response go back to the client? Would we have to do
some weird accounting where three clients A, B, and C and each written an edit,
and then the length we get back from exisiting DNs after a crash say does not
include the edit written by client C... we'll have to figure out how to fail
client C's write (though we'd moved on from append and were trying to
sync/hflush the append)?
{quote}
I think we can implement in this way. When a WAL is broken(one datanode fail
means broken)
1. Open a new WAL synchronously.
2. Write all the un-acked WAL entries to the new WAL file(which means we should
keep all the un-acked WAL entries).
3. Schedule a background task to close the old WAL file.
We should hold the sync WAL request if we consider that some of WAL entries
after last sync has already been written out but not acked until we
successfully write them to new WAL file and get ack back.
And the task in phase 3
1. Doing standard pipeline recovery. Maybe a little difference is that, we can
truncate the block length to our acked length when negotiating with datanodes.
2. endBlock on each datanode.
3. complete file on namenode.
It is does not matter if the rs is crashed during the recovery because we can
make sure that the file length after lease recovery should be longer than acked
length(unless the 3 datanodes are all crashed, we can not handle this using
hflush).
Thanks. [~stack]
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
Ted Yu created HBASE-14928: -- Summary: Start row should be set for query through HBase REST gateway involving globbing option Key: HBASE-14928 URL: https://issues.apache.org/jira/browse/HBASE-14928 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu As Ben Sutton reported in the thread, Slow response on HBase REST api using globbing option, query through the Rest API with a globbing option i.e. http://:/table/key\* executes extremely slowly. Jerry He pointed out that PrefixFilter is used for query involving globbing option. This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14928: --- Attachment: 14928-v1.txt > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14928: --- Status: Patch Available (was: Open) > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14926) Hung ThriftServer; no timeout on read from client; if client crashes, worker thread gets stuck reading
[
https://issues.apache.org/jira/browse/HBASE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041376#comment-15041376
]
Hadoop QA commented on HBASE-14926:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775738/14926.patch
against master branch at commit 8b3d1f144408e4a7a014c5ac46418c9e91b9b0db.
ATTACHMENT ID: 12775738
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ 4. Here is a lazy example that just pulls in all hbase dependency
jars and that goes against default location on localhost.
+ {java -cp
./hbase-examples/target/hbase-examples-2.0.0-SNAPSHOT.jar:`./bin/hbase
classpath` org.apache.hadoop.hbase.thrift.DemoClient localhost 9090}
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16764//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16764//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16764//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16764//console
This message is automatically generated.
> Hung ThriftServer; no timeout on read from client; if client crashes, worker
> thread gets stuck reading
> --
>
> Key: HBASE-14926
> URL: https://issues.apache.org/jira/browse/HBASE-14926
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 1.0.3, 0.98.16
>Reporter: stack
>Assignee: stack
> Attachments: 14926.patch
>
>
> Thrift server is hung. All worker threads are doing this:
> {code}
> "thrift-worker-0" daemon prio=10 tid=0x7f0bb95c2800 nid=0xf6a7 runnable
> [0x7f0b956e]
>java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00066d859490> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
> at
> org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.jav
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041432#comment-15041432 ] Anoop Sam John commented on HBASE-13082: Can we have it in 1.3 also? > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, > HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, > HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, > HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, > HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, > HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, > HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, > LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14929) There is a space missing from Table "foo" is not currently available.
Ted Malaska created HBASE-14929:
---
Summary: There is a space missing from Table "foo" is not
currently available.
Key: HBASE-14929
URL: https://issues.apache.org/jira/browse/HBASE-14929
Project: HBase
Issue Type: Bug
Reporter: Ted Malaska
Priority: Trivial
Go to the following line in LoadIncrementalHFiles.java
throw new TableNotFoundException("Table " + table.getName() + "is not currently
available.");
and add a space before is and after '
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14895: --- Attachment: HBASE-14895.patch Patch for QA. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14895: --- Status: Patch Available (was: Open) > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
[
https://issues.apache.org/jira/browse/HBASE-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041519#comment-15041519
]
Hudson commented on HBASE-14904:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1140 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1140/])
HBASE-14904 Mark Base[En|De]coder LimitedPrivate and fix binary compat (enis:
rev b04a09da352a396bcbb2fbd4bbcbd7aceaa8fdfb)
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseDecoder.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseEncoder.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
> Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
>
>
> Key: HBASE-14904
> URL: https://issues.apache.org/jira/browse/HBASE-14904
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: hbase-14904_v1.patch, hbase-14904_v2.patch
>
>
> PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary
> compatibility in Phoenix compiled with earlier versions of HBase and run
> agains later versions.
> This is one of the areas that the boundary is not clear, but it won't hurt us
> to fix it.
> The exception trace is:
> {code}
> Exception in thread "main" java.lang.NoSuchFieldError: in
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
> {code}
> Although {{BaseDecoder.in}} is still there, it got changed to be a class
> rather than an interface. BaseDecoder is marked Private, thus the binary
> compat check is not run at all. Not sure whether it would have caught this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041527#comment-15041527 ] Anoop Sam John commented on HBASE-14895: After HBASE-13082, we will notify ChangedReadersObserver only after a flush operation. After bulk load and compaction we dont need inform the readers. After a flush, which results in exactly one file, the notify mechanism can pass that file's detail also? So on that we can do the reset. We will remove the scanner on the snapshot also (?).. So this can be achieved with lesser code change? Rather than maintain new Lists and all? > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[
https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041540#comment-15041540
]
Hadoop QA commented on HBASE-14928:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775755/14928-v1.txt
against master branch at commit 8b3d1f144408e4a7a014c5ac46418c9e91b9b0db.
ATTACHMENT ID: 12775755
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16765//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16765//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16765//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16765//console
This message is automatically generated.
> Start row should be set for query through HBase REST gateway involving
> globbing option
> --
>
> Key: HBASE-14928
> URL: https://issues.apache.org/jira/browse/HBASE-14928
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14928-v1.txt
>
>
> As Ben Sutton reported in the thread, Slow response on HBase REST api using
> globbing option, query through the Rest API with a globbing option i.e.
> http://:/table/key\* executes extremely slowly.
> Jerry He pointed out that PrefixFilter is used for query involving globbing
> option.
> This issue is to fix this bug by setting start row for such queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
[
https://issues.apache.org/jira/browse/HBASE-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041541#comment-15041541
]
Hudson commented on HBASE-14904:
FAILURE: Integrated in HBase-0.98-matrix #267 (See
[https://builds.apache.org/job/HBase-0.98-matrix/267/])
HBASE-14904 Mark Base[En|De]coder LimitedPrivate and fix binary compat (enis:
rev b04a09da352a396bcbb2fbd4bbcbd7aceaa8fdfb)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseDecoder.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseEncoder.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java
> Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
>
>
> Key: HBASE-14904
> URL: https://issues.apache.org/jira/browse/HBASE-14904
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: hbase-14904_v1.patch, hbase-14904_v2.patch
>
>
> PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary
> compatibility in Phoenix compiled with earlier versions of HBase and run
> agains later versions.
> This is one of the areas that the boundary is not clear, but it won't hurt us
> to fix it.
> The exception trace is:
> {code}
> Exception in thread "main" java.lang.NoSuchFieldError: in
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
> {code}
> Although {{BaseDecoder.in}} is still there, it got changed to be a class
> rather than an interface. BaseDecoder is marked Private, thus the binary
> compat check is not run at all. Not sure whether it would have caught this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13857) Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
[
https://issues.apache.org/jira/browse/HBASE-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041651#comment-15041651
]
Vrishal Kulkarni commented on HBASE-13857:
--
There's one failing test
org.apache.hadoop.hbase.ipc.TestAsyncIPC.testRTEDuringAsyncConnectionSetup[0]
The failure is unrelated to this change!
> Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
> -
>
> Key: HBASE-13857
> URL: https://issues.apache.org/jira/browse/HBASE-13857
> Project: HBase
> Issue Type: Bug
> Components: regionserver, UI
>Affects Versions: 0.98.0
>Reporter: Lars George
>Assignee: Vrishal Kulkarni
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-13857.patch
>
>
> The template has this:
> {noformat}
>
> ...
> Slow WAL Append Count
>
>
>
> <% 0 %>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13857) Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
[
https://issues.apache.org/jira/browse/HBASE-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041650#comment-15041650
]
Vrishal Kulkarni commented on HBASE-13857:
--
There's one failing test
org.apache.hadoop.hbase.ipc.TestAsyncIPC.testRTEDuringAsyncConnectionSetup[0]
The failure is unrelated to this change!
> Slow WAL Append count in ServerMetricsTmpl.jamon is hardcoded to zero
> -
>
> Key: HBASE-13857
> URL: https://issues.apache.org/jira/browse/HBASE-13857
> Project: HBase
> Issue Type: Bug
> Components: regionserver, UI
>Affects Versions: 0.98.0
>Reporter: Lars George
>Assignee: Vrishal Kulkarni
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-13857.patch
>
>
> The template has this:
> {noformat}
>
> ...
> Slow WAL Append Count
>
>
>
> <% 0 %>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14928: --- Fix Version/s: 1.1.3 1.3.0 1.2.0 2.0.0 > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14928: --- Fix Version/s: 0.98.17 > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041680#comment-15041680
]
Hadoop QA commented on HBASE-14895:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775763/HBASE-14895.patch
against master branch at commit 8b3d1f144408e4a7a014c5ac46418c9e91b9b0db.
ATTACHMENT ID: 12775763
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16766//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16766//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16766//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16766//console
This message is automatically generated.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14822) Renewing leases of scanners doesn't work
[ https://issues.apache.org/jira/browse/HBASE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041711#comment-15041711 ] Andrew Purtell commented on HBASE-14822: lgtm, and has the benefit of not needing to be concerned about the behavior of older server side code. New flag will simply be ignored. > Renewing leases of scanners doesn't work > > > Key: HBASE-14822 > URL: https://issues.apache.org/jira/browse/HBASE-14822 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.14 >Reporter: Samarth Jain >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4 > > Attachments: 14822-0.98-v2.txt, 14822-0.98-v3.txt, 14822-0.98.txt, > 14822-v3-0.98.txt, 14822-v4-0.98.txt, 14822.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14869) Better request latency histograms
[ https://issues.apache.org/jira/browse/HBASE-14869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041714#comment-15041714 ] Andrew Purtell commented on HBASE-14869: Ok, no further concerns here > Better request latency histograms > - > > Key: HBASE-14869 > URL: https://issues.apache.org/jira/browse/HBASE-14869 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: 14869-test-0.98.txt, 14869-v1-0.98.txt, > 14869-v1-2.0.txt, 14869-v2-0.98.txt, 14869-v2-2.0.txt, 14869-v3-0.98.txt, > 14869-v4-0.98.txt, 14869-v5-0.98.txt, AppendSizeTime.png, Get.png > > > I just discussed this with a colleague. > The get, put, etc, histograms that each region server keeps are somewhat > useless (depending on what you want to achieve of course), as they are > aggregated and calculated by each region server. > It would be better to record the number of requests in certainly latency > bands in addition to what we do now. > For example the number of gets that took 0-5ms, 6-10ms, 10-20ms, 20-50ms, > 50-100ms, 100-1000ms, > 1000ms, etc. (just as an example, should be > configurable). > That way we can do further calculations after the fact, and answer questions > like: How often did we miss our SLA? Percentage of requests that missed an > SLA, etc. > Comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14926) Hung ThriftServer; no timeout on read from client; if client crashes, worker thread gets stuck reading
[
https://issues.apache.org/jira/browse/HBASE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041735#comment-15041735
]
Andrew Purtell commented on HBASE-14926:
bq. I'm a bit stuck on how to manufacture this circumstance in a test
Yeah, hard to mock
It's an obvious improvement even without a test, I'd be +1 to commit as is
> Hung ThriftServer; no timeout on read from client; if client crashes, worker
> thread gets stuck reading
> --
>
> Key: HBASE-14926
> URL: https://issues.apache.org/jira/browse/HBASE-14926
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 1.0.3, 0.98.16
>Reporter: stack
>Assignee: stack
> Attachments: 14926.patch
>
>
> Thrift server is hung. All worker threads are doing this:
> {code}
> "thrift-worker-0" daemon prio=10 tid=0x7f0bb95c2800 nid=0xf6a7 runnable
> [0x7f0b956e]
>java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00066d859490> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
> at
> org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
> at
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> They never recover.
> I don't have client side logs.
> We've been here before: HBASE-4967 "connected client thrift sockets should
> have a server side read timeout" but this patch only got applied to fb branch
> (and thrift has changed since then).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14900) Make global config option for ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041764#comment-15041764 ] Cody Marcel commented on HBASE-14900: - I should have a patch today. I got side tracked. I was also trying to refactor the existing tests a little to leverage those. Unsurprisingly that's taking longer than the few lines for the code change ;) > Make global config option for ReplicationEndpoint > - > > Key: HBASE-14900 > URL: https://issues.apache.org/jira/browse/HBASE-14900 > Project: HBase > Issue Type: Sub-task > Components: Replication >Affects Versions: 2.0.0 >Reporter: Cody Marcel >Assignee: Cody Marcel >Priority: Minor > Fix For: 2.0.0 > > > Currently ReplicationEndpoint implementations can only be configured through > the HBase shell. We should be able to to use a property in the hbase-site.xml > to globally set an alternate default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041761#comment-15041761
]
Yu Li commented on HBASE-14790:
---
bq. hsync periodically, for example, default per 1000ms
AFAIK, DN will do this if {{dfs.datanode.sync.behind.writes}} is set to true,
and the window is 8MB, just correct me if I'm wrong.
Sorry for the interruption but is the discussion here back to the issue
HBASE-14004 reports, say preventing ReplicationSource to replicate any
non-persisted and later-lost data, due to the fact that we depends on hflush
not hsync currently? And it seems we could not resolve the issue by
implementing new DFSOutputStream?
Would it be a better idea to continue the discussion on master-slave
replication consistency issue in HBASE-14004, while focusing on the new
DFSOutputStream here, to make "an optimized DataOutputStream for small writes
like WAL entry"?
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-14929) There is a space missing from Table "foo" is not currently available.
[
https://issues.apache.org/jira/browse/HBASE-14929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carlos A. Morillo reassigned HBASE-14929:
-
Assignee: Carlos A. Morillo
> There is a space missing from Table "foo" is not currently available.
> -
>
> Key: HBASE-14929
> URL: https://issues.apache.org/jira/browse/HBASE-14929
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Malaska
>Assignee: Carlos A. Morillo
>Priority: Trivial
>
> Go to the following line in LoadIncrementalHFiles.java
> throw new TableNotFoundException("Table " + table.getName() + "is not
> currently available.");
> and add a space before is and after '
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
[
https://issues.apache.org/jira/browse/HBASE-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041782#comment-15041782
]
Hudson commented on HBASE-14904:
SUCCESS: Integrated in HBase-1.0 #1119 (See
[https://builds.apache.org/job/HBase-1.0/1119/])
HBASE-14904 Mark Base[En|De]coder LimitedPrivate and fix binary compat (enis:
rev 46536baa97d3c15a3c2233ea26e7b9034dfa520c)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseDecoder.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/codec/BaseEncoder.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java
> Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
>
>
> Key: HBASE-14904
> URL: https://issues.apache.org/jira/browse/HBASE-14904
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: hbase-14904_v1.patch, hbase-14904_v2.patch
>
>
> PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary
> compatibility in Phoenix compiled with earlier versions of HBase and run
> agains later versions.
> This is one of the areas that the boundary is not clear, but it won't hurt us
> to fix it.
> The exception trace is:
> {code}
> Exception in thread "main" java.lang.NoSuchFieldError: in
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
> {code}
> Although {{BaseDecoder.in}} is still there, it got changed to be a class
> rather than an interface. BaseDecoder is marked Private, thus the binary
> compat check is not run at all. Not sure whether it would have caught this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041781#comment-15041781
]
Hudson commented on HBASE-14223:
SUCCESS: Integrated in HBase-1.0 #1119 (See
[https://builds.apache.org/job/HBase-1.0/1119/])
Revert "HBASE-14223 Meta WALs are not cleared if meta region was closed (enis:
rev c4aab43a1ac1617890bb4535c9611fa0e31a59c2)
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestMetaWALsAreClosed.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/NoKillMonkeyFactory.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseMetaHandler.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/MoveMetaAction.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/StressAssignmentManagerMonkeyFactory.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogR
[jira] [Commented] (HBASE-14719) Add metric for number of MasterProcWALs
[ https://issues.apache.org/jira/browse/HBASE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041854#comment-15041854 ] Vrishal Kulkarni commented on HBASE-14719: -- Have you had a chance? > Add metric for number of MasterProcWALs > --- > > Key: HBASE-14719 > URL: https://issues.apache.org/jira/browse/HBASE-14719 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Vrishal Kulkarni > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-14719.1.patch, HBASE-14719.2.patch, > HBASE-14719.3.patch, HBASE-14719.patch > > > Lets add monitoring to this so that we can see when it starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041859#comment-15041859 ] Nick Dimiduk commented on HBASE-14928: -- Perhaps the use of PrefixFilter should be detected and a startkey be added automatically if one is not present. > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041865#comment-15041865 ] Ted Yu commented on HBASE-14928: I agree. Lars and I had some discussion a long time back but if I remember correctly, no consensus was reached how this should be done. Maybe do it in another JIRA ? Let's fix the regression reported by Ben first. > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14926) Hung ThriftServer; no timeout on read from client; if client crashes, worker thread gets stuck reading
[
https://issues.apache.org/jira/browse/HBASE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041870#comment-15041870
]
stack commented on HBASE-14926:
---
I tried hacking up a fail by hand. Let me try again... if I don't get
anywhere will commit as is. Thanks for taking a look [~apurtell]
> Hung ThriftServer; no timeout on read from client; if client crashes, worker
> thread gets stuck reading
> --
>
> Key: HBASE-14926
> URL: https://issues.apache.org/jira/browse/HBASE-14926
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 1.0.3, 0.98.16
>Reporter: stack
>Assignee: stack
> Attachments: 14926.patch
>
>
> Thrift server is hung. All worker threads are doing this:
> {code}
> "thrift-worker-0" daemon prio=10 tid=0x7f0bb95c2800 nid=0xf6a7 runnable
> [0x7f0b956e]
>java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00066d859490> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
> at
> org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
> at
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> They never recover.
> I don't have client side logs.
> We've been here before: HBASE-4967 "connected client thrift sockets should
> have a server side read timeout" but this patch only got applied to fb branch
> (and thrift has changed since then).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041873#comment-15041873 ] stack commented on HBASE-13082: --- Wahoo. +1 on 1.3. Probably too late for 1.2. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, > HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, > HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, > HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, > HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, > HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, > HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, > LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041897#comment-15041897 ] Jerry He commented on HBASE-14928: -- I am coping my comment in the mailing to here: There seems to be a change in the way the url table/rowkey* is executed on the Rest gateway. in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte of 255 on Rest gateway in the Scan sent to the servers. In the newer versions, table/rowkey* seems to treated with a prefixFilter. This may explain why you are seeing slower response. > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[
https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041907#comment-15041907
]
Jerry He commented on HBASE-14928:
--
I wonder if it is fine to go back to the old behavior completely.
In the new code,
{code}
@Path("{suffixglobbingspec: .*\\*/.+}")
{code}
{code}
@Path("{scanspec: .*[*]$}")
{code}
The url path table/rowkey* will be handled by the scanspec. Should it be
handled by the suffixglobbingspec? Notice that the suffixglobbingspec needs
table/rowkey*/column
The workaround I had is to table/rowkey*/family1. Then it would go back to the
old behavior.
> Start row should be set for query through HBase REST gateway involving
> globbing option
> --
>
> Key: HBASE-14928
> URL: https://issues.apache.org/jira/browse/HBASE-14928
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17
>
> Attachments: 14928-v1.txt
>
>
> As Ben Sutton reported in the thread, Slow response on HBase REST api using
> globbing option, query through the Rest API with a globbing option i.e.
> http://:/table/key\* executes extremely slowly.
> Jerry He pointed out that PrefixFilter is used for query involving globbing
> option.
> This issue is to fix this bug by setting start row for such queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041910#comment-15041910
]
Phil Yang commented on HBASE-14790:
---
Currently there are two scenarios which may result in inconsistency between two
clusters.
The first is master cluster crashes(for example, power failure) or three DNs
and RS crash at the same time and we lost all data that is not flushed to DNs'
disks but the data have been already synced to slave cluster.
The second is we will rollback memstore and response client an error if we get
a error on hflush but the log may indeed exists in WAL. This will not only
results in inconsistency between two clusters but also gives client a wrong
response because the data will "revive" after replaying WAL. This scenario has
been discussed in HBASE-14004
Comparing to the second, it is easier to solve the first scenario that we can
tell ReplicationSource it can only read the logs that is already saved on three
disks. We need to know the largest WAL entry id that has been synced. So HDFS's
sync logic for itself may be not useful for us and we must use hsync to let
HBase know the entry id. So we need a configurable periodically hsync here, and
if we have only one cluster it is also helpful to reduce data losses because of
data center power failure or unluckily crashing three DNs and RS at the same
time. I think this work can be done without the new DFSOutputStream?
For the second scenario, it is more complex because we can not rollback
memstore and tell client this operation failed unless we are very sure the data
will never exist in WAL, and mostly we are not sure... So we have to use a new
WAL logic that rewriting the entry to the new file rather than rollback. To
implement this we need to handle duplicate entries while replaying WAL. I think
this logic is not conflicting with pipeline DFSOutputStream so actually we can
fix it on currently WAL implementation?
And this issue HBASE-14790 may be only a performance improvement work that will
not fix any bugs? Of course, the FanOutOneBlockDFSOutputStream should implement
the new WAL logic directly.
[~Apache9] What do you think?
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14873) Problems around BoundedByteBufferPool providing direct buffers
[
https://issues.apache.org/jira/browse/HBASE-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041908#comment-15041908
]
stack commented on HBASE-14873:
---
bq. In practice the most problematic thing seems unlimited Unsafe.copyMemory if
we worry about stop-the-world.
bq. How copyFromBufferToArray can omit the logic?
Oversight. Thanks for turning it up. Lets fix.
bq. Another thing I should mention is that copyFromBufferToArray is forced to
use one-by-one copy for the other VM implementations, which is an almost bug.
Another issue. Again, lets fix. There is logic in
Bytes#LexicographicalComparerHolder that marches through the buffer long-wise,
then by int,... and finally by byte.
bq. Unsafe is mysteriously dangerous and I doubt the performance improvement is
practically effective at the cost of uncertain risk.
Yeah. Going this route is risky. The alternative though is all our eggs in the
G1GC basket which is also a risk. This route of offheaping is looking promising
(Anoop and Ram have a nice blog that will be up soon). We could end up in a new
world of unexplained stop-the-worlds because of these offheap machinations and
if they were unfixable, we'd have to call this (expensive) experiment a
faillure but so far the results look good.
Thanks [~ikeda]. Keep the feedback coming.
On this patch, do we want to make it use utils before we commit?
> Problems around BoundedByteBufferPool providing direct buffers
> --
>
> Key: HBASE-14873
> URL: https://issues.apache.org/jira/browse/HBASE-14873
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: HBASE-14873-V2.patch, HBASE-14873.patch,
> HBASE-14873.patch, HBASE-14873.patch
>
>
> HBASE-13819 made BoundedByteBufferPool provide direct buffers.
> See RpcServer.java:
> {code}
> ...
> class Call implements RpcCallContext {
> protected synchronized void setResponse(...) {
> ...
> this.cellBlock = ipcUtil.buildCellBlock(..., reservoir);
> ...
> bc = new BufferChain(..., this.cellBlock);
> if (connection.useWrap) {
> bc = wrapWithSasl(bc);
> }
> ...
> private BufferChain wrapWithSasl(BufferChain bc) throws IOException {
> ...
> byte[] responseBytes = bc.getBytes();
> ...
> {code}
> {{cellBlock}} is expected to be a direct buffer retrieved from {{reservoir}}
> (but not always), and {{bc}} may be composed of both direct and non-direct
> buffers.
> And then, see BufferChain.java:
> {code}
> byte [] getBytes() {
> ...
> for (ByteBuffer bb: this.buffers) {
> System.arraycopy(bb.array(), ...);
> {code}
> A direct buffer doesn't give its array, and will throw
> UnsupportedOperationException.
> Another problem; {{cellBlock}} is allowed to be a non-direct buffer, and
> after use it will be put to {{reservoir}}, mixing direct and non-direct
> buffers in the pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041914#comment-15041914 ] Jerry He commented on HBASE-14928: -- I don't know if it is intentional that the scanspec handles the rowkey suffixglobbing. The patch sets the startKey, but no endKey. It will still have slower response if the number of rows after the startKey is big. Right? > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041916#comment-15041916 ] Ted Yu commented on HBASE-14928: bq. table/rowkey\*/family1 What if user wants all column families ? I prefer the current patch. > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14516) categorize hadoop-compat tests
[ https://issues.apache.org/jira/browse/HBASE-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14516: Resolution: Fixed Fix Version/s: 1.0.4 0.98.17 1.1.3 1.3.0 1.2.0 2.0.0 Status: Resolved (was: Patch Available) > categorize hadoop-compat tests > -- > > Key: HBASE-14516 > URL: https://issues.apache.org/jira/browse/HBASE-14516 > Project: HBase > Issue Type: Task > Components: build, hadoop2, test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4 > > Attachments: HBASE-14516.0.98.v2.patch, HBASE-14516.1.patch > > > the hadoop-compat and hadoop2-compat modules do not rely on the hbase > annotations test-jar and their tests aren't categorized. > this causes things to fail if you attempt to specify one of our test > categories to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[
https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041930#comment-15041930
]
Ted Yu commented on HBASE-14928:
bq. The patch sets the startKey, but no endKey
If certain rows don't pass the prefix, PrefixFilter has this code:
{code}
public boolean filterAllRemaining() {
return passedPrefix;
{code}
> Start row should be set for query through HBase REST gateway involving
> globbing option
> --
>
> Key: HBASE-14928
> URL: https://issues.apache.org/jira/browse/HBASE-14928
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17
>
> Attachments: 14928-v1.txt
>
>
> As Ben Sutton reported in the thread, Slow response on HBase REST api using
> globbing option, query through the Rest API with a globbing option i.e.
> http://:/table/key\* executes extremely slowly.
> Jerry He pointed out that PrefixFilter is used for query involving globbing
> option.
> This issue is to fix this bug by setting start row for such queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041969#comment-15041969
]
stack commented on HBASE-14925:
---
Just some notes... yeah, would be good to have a nice dump of regions by table.
How to format output will be an issue. Need to do escaping if binary data as we
do in the shell currently. Unfortunately while you can do 'm =
get_table('hbase:meta')' and then 'm.scan...', we've not done the work so you
can do 'for result in m.scan' so you could loop the results in the shell
and format them however you wish.
Here is some ugly hack that gets the wanted cells from hbase:meta and then cuts
up the emission into a form perhaps digestible in shell (have to handle the
empty row that demarks start and stop of a table...)
{code}
echo "scan 'hbase:meta', {ROWPREFIXFILTER => 't2,', COLUMNS => ['info:server',
'info:regioninfo']"} | ./bin/hbase shell|sed -e 's/^.*value=//'|paste -d " " -
-|sed -e "s/^.*NAME => '//" -e "s/', STARTKEY => '/ /" -e "s/', ENDKEY => '/ /"
-e "s/'} / /"
{code}
...where 't2' in above is the tablename, would get you:
{code}
t2,a,1449181637707.cf2b5d8caa5a0a44d7f0b48273abb53e. a b localhost:61767
t2,b,1449181637707.80686f6b985b23885e06416ced5e9e1b. b c localhost:61767
t2,c,1449181637707.4c8237c47d72de053f1b8e50a28dd223. c d localhost:61767
t2,d,1449181637707.a5f1b474563b6d4ee2cabbd75d2ffd1e. d e localhost:61767
t2,e,1449181637707.14f09d26e6d98498473483ee479b0e61. e f localhost:61767
t2,f,1449181637707.a37b71a65c6a6395ffdf4af57d69382a. f localhost:61767
{code}
... but all is quoted and it needs more edit to get clean list.
Doing this in shell, you are getting the toStringBinary... which does escaping
which may or may not be wanted by the issue filer.
The shell is ruby so you could write a few lines to format however.
> Develop HBase shell command/tool to list table's region info through command
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Romil Choksi
>
> I am going through the hbase shell commands to see if there is anything I can
> use to get all the regions info just for a particular table. I don’t see any
> such command that provides me that information.
> It would be better to have a command that provides region info, start key,
> end key etc taking a table name as the input parameter. This is available
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a
> tabular structured output (that is machine readable)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14907) NPE of MobUtils.hasMobColumns in Build failed in Jenkins: HBase-Trunk_matrix » latest1.8,Hadoop #513
[
https://issues.apache.org/jira/browse/HBASE-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14907:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks for taking the time to look into the test failure
[~jingcheng...@intel.com]
> NPE of MobUtils.hasMobColumns in Build failed in Jenkins: HBase-Trunk_matrix
> » latest1.8,Hadoop #513
>
>
> Key: HBASE-14907
> URL: https://issues.apache.org/jira/browse/HBASE-14907
> Project: HBase
> Issue Type: Bug
> Components: mob
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-14907-V2.patch, HBASE-14907.patch
>
>
> NPE is thrown when rollback the failures of table creation.
> 1. Table is being created, get issues when creating fs layout.
> 2. Rollback this creation and trying to delete the data from fs. It tries to
> delete the mob dir and needs to ask HMaster about the HTableDescriptor, and
> at that time the table dir had been deleted and no HTableDescriptor can be
> found.
> The exception looks like:
> {noformat}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.mob.MobUtils.hasMobColumns(MobUtils.java:851)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteFromFs(DeleteTableProcedure.java:350)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.rollbackState(CreateTableProcedure.java:167)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.rollbackState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:134)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:467)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecut
> {noformat}
> In this patch, it directly checks if the mob directory is existing instead of
> checking the HTableDescriptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Status: Open (was: Patch Available) > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Attachment: (was: HBASE-14901.4.patch) > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Attachment: HBASE-14901.5.patch Didn't re-queue after cancel/subit; re-uploading the same patch (no change from .4) and resubmitting. > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch, HBASE-14901.5.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Status: Patch Available (was: Open) > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch, HBASE-14901.5.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14864) Add support for bucketing of keys into client library
[
https://issues.apache.org/jira/browse/HBASE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041994#comment-15041994
]
Sean Busbey commented on HBASE-14864:
-
[~anoop.hbase] are you thinking of HBASE-12853?
> Add support for bucketing of keys into client library
> -
>
> Key: HBASE-14864
> URL: https://issues.apache.org/jira/browse/HBASE-14864
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Reporter: Lars George
>
> This has been discussed and taught so many times, I believe it is time to
> support it properly. The idea is to be able to assign an optional _bucketing_
> strategy to a table, which translates the user given row keys into a bucketed
> version. This is done by either simple count, or by parts of the key.
> Possibly some simple functionality should help _compute_ bucket keys.
> For example, given a key {{\-\--...}} you could
> imagine that a rule can be defined that takes the _epoch_ part and chunks it
> into, for example, 5 minute buckets. This allows to store small time series
> together and make reading (especially over many servers) much more efficient.
> The client also supports the proper scan logic to fan a scan over the buckets
> as needed. There may be an executor service (implicitly or explicitly
> provided) that is used to fetch the original data with user visible ordering
> from the distributed buckets.
> Note that this has been attempted a few times to various extends out in the
> field, but then withered away. This is an essential feature that when present
> in the API will make users consider this earlier, instead of when it is too
> late (when hot spotting occurs for example).
> The selected bucketing strategy and settings could be stored in the table
> descriptor key/value pairs. This will allow any client to observe the
> strategy transparently. If not set the behaviour is the same as today, so the
> new feature is not touching any critical path in terms of code, and is fully
> client side. (But could be considered for say UI support as well - if needed).
> The strategies are pluggable using classes, but a few default implementations
> are supplied.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041995#comment-15041995 ] Jerry He commented on HBASE-14928: -- Good catch on the filterAllRemaining. It slipped from my mind. Patch looks good. bq. Perhaps the use of PrefixFilter should be detected and a startkey be added automatically if one is not present. Good point too. > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042022#comment-15042022
]
stack commented on HBASE-14925:
---
To be clear, the sed hackery above is not meant to replace the original wish
for a simple way of listing regions and servers. The hard part I think is how
to format the output (taking into consideration that keys may have binary
content..). The list suggestion with a means of specifying how to do the output
seems good ([~huaxiang]?)
Would be cool too if rather than solve the explicit ask here, instead we did
some generic changes -- such as making it so it easy to iterate row results in
the shell to do with the result as you wish -- so it was easy to satisfy
requests like those of the original posters. E.g: being able to iterate in the
shell on the rows returned out of a Scan:
{code}
hbase> for result in "hbase:meta".scan({FILTER, [COLUMNS]) do
puts(results.get('info:server') + " " + results.get("info:regioninfo")) end
{code}
... or some such ... but even just writing the above out, it is ugly... in
ruby+hbase shell DSL... and it is incomplete not writing escaped strings
So scratch this last suggestion (smile) at least for now.
> Develop HBase shell command/tool to list table's region info through command
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Romil Choksi
>
> I am going through the hbase shell commands to see if there is anything I can
> use to get all the regions info just for a particular table. I don’t see any
> such command that provides me that information.
> It would be better to have a command that provides region info, start key,
> end key etc taking a table name as the input parameter. This is available
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a
> tabular structured output (that is machine readable)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14530) surefire configuration ignores surefire.firstPartGroups
[ https://issues.apache.org/jira/browse/HBASE-14530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042023#comment-15042023 ] Sean Busbey commented on HBASE-14530: - no, not consistently. It doesn't work in e.g. hbase-common or hbase-client. > surefire configuration ignores surefire.firstPartGroups > --- > > Key: HBASE-14530 > URL: https://issues.apache.org/jira/browse/HBASE-14530 > Project: HBase > Issue Type: Bug > Components: build, test >Reporter: Sean Busbey > > all of our various test profiles rely on using JUnit's test categorization > groups. however, no part of our configuration properly sets the groups for > the initial run to use surefire.firstPartGroups. > This means that the first part always attempts to run all the tests. > It also means the profiles that try to break things up into first and second > runs then re-run tests in the second part. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun reassigned HBASE-14925: Assignee: huaxiang sun > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: huaxiang sun > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042042#comment-15042042 ] huaxiang sun commented on HBASE-14925: -- [~stack] I take it and will discuss more with you on Monday, :) > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: huaxiang sun > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13426) add java6 builds to 0.98 jenkins
[ https://issues.apache.org/jira/browse/HBASE-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-13426. - Resolution: Fixed build has been live since the last jenkins clean up. > add java6 builds to 0.98 jenkins > > > Key: HBASE-13426 > URL: https://issues.apache.org/jira/browse/HBASE-13426 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey > > make sure we have jenkins coverage of java6 builds for 0.98 so we can catch > things like HBASE-13422 earlier -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14930) check_compatibility.sh needs smarter exit codes
Dima Spivak created HBASE-14930: --- Summary: check_compatibility.sh needs smarter exit codes Key: HBASE-14930 URL: https://issues.apache.org/jira/browse/HBASE-14930 Project: HBase Issue Type: Bug Reporter: Dima Spivak Assignee: Dima Spivak The check_compatibility.sh tool in dev_support uses the Java API Compliance Checker to do static analysis of source/binary incompatibilties between two HBase branches. One problem, though, is that the script has a few instances where it may return an exit code of 1 (e.g. if Maven steps fail), but this is the same exit code that the Java ACC tool itself uses to denote that the tool succeeded, but found incompatibilities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14901) There is duplicated code to create/manage encryption keys
[
https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042064#comment-15042064
]
Hadoop QA commented on HBASE-14901:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775827/HBASE-14901.5.patch
against master branch at commit 03e4712f0ca08d57586b3fc4d93cf02c999515d8.
ATTACHMENT ID: 12775827
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16767//console
This message is automatically generated.
> There is duplicated code to create/manage encryption keys
> -
>
> Key: HBASE-14901
> URL: https://issues.apache.org/jira/browse/HBASE-14901
> Project: HBase
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.0.0
>Reporter: Nate Edel
>Assignee: Nate Edel
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch,
> HBASE-14901.3.patch, HBASE-14901.5.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> There is duplicated code from MobUtils.createEncryptionContext in HStore, and
> there is a subset of that code in HFileReaderImpl.
> Refactored key selection
> Moved both to EncryptionUtil.java
> Can't figure out how to write a unit test for this, but there's no new code
> just refactoring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14674) Rpc handler / task monitoring seems to be broken after 0.98
[
https://issues.apache.org/jira/browse/HBASE-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14674:
Fix Version/s: (was: 1.2.1)
1.2.0
> Rpc handler / task monitoring seems to be broken after 0.98
> ---
>
> Key: HBASE-14674
> URL: https://issues.apache.org/jira/browse/HBASE-14674
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 1.2.0, 1.3.0, 1.0.3, 1.1.4, 0.98.17
>
> Attachments: HBASE-14674.patch, HBASE-14674_v1.patch,
> HBASE-14674_v2.patch
>
>
> In 0.96, we have the RPC handlers listed as tasks and show them in the web UI
> as well:
> {code}
> Tasks:
> ===
> Task: RpcServer.handler=0,port=64231
> Status: WAITING:Waiting for a call
> Running for 932s
> Task: RpcServer.handler=1,port=64231
> Status: WAITING:Waiting for a call
> Running for 932s
> Task: RpcServer.handler=2,port=64231
> Status: WAITING:Waiting for a call
> Running for 932s
> {code}
> After pluggable RPC scheduler, the way the tasks work for the handlers got
> changed. We no longer list idle RPC handlers in the tasks, but we register
> them dynamically to {{TaskMonitor}} through {{CallRunner}}. However, the IPC
> readers are still registered the old way (meaning that idle readers are
> listed as tasks, but not idle handlers).
> From the javadoc of {{MonitoredRPCHandlerImpl}}, it seems that we are NOT
> optimizing the allocation for the MonitoredTask anymore, but instead allocate
> one for every RPC call breaking the pattern (See CallRunner.getStatus()).
> {code}
> /**
> * A MonitoredTask implementation designed for use with RPC Handlers
> * handling frequent, short duration tasks. String concatenations and object
> * allocations are avoided in methods that will be hit by every RPC call.
> */
> @InterfaceAudience.Private
> public class MonitoredRPCHandlerImpl extends MonitoredTaskImpl
> {code}
> There is also one more side affect that, since the CallRunner is a per-RPC
> object and created in the RPC listener thread, the created task ends up
> having a name "listener" although the actual processing happens in a handler
> thread. This is obviously very confusing during debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12988) [Replication]Parallel apply edits across regions
[ https://issues.apache.org/jira/browse/HBASE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12988: Fix Version/s: (was: 1.2.1) 1.2.0 > [Replication]Parallel apply edits across regions > > > Key: HBASE-12988 > URL: https://issues.apache.org/jira/browse/HBASE-12988 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: hongyu bi >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 12988-v2.txt, 12988-v3.txt, 12988-v4.txt, 12988-v5.txt, > 12988.txt, HBASE-12988-0.98.patch, ParallelReplication-v2.txt > > > we can apply edits to slave cluster in parallel on table-level to speed up > replication . > update : per conversation blow , it's better to apply edits on row-level in > parallel -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14680) Two configs for snapshot timeout and better defaults
[
https://issues.apache.org/jira/browse/HBASE-14680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14680:
Fix Version/s: (was: 1.2.1)
1.2.0
> Two configs for snapshot timeout and better defaults
>
>
> Key: HBASE-14680
> URL: https://issues.apache.org/jira/browse/HBASE-14680
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14680.patch, HBASE-14680_v1.patch,
> HBASE-14680_v2.patch, hbase-14680_v3.patch
>
>
> One of the clusters timed out taking a snapshot for a disabled table. The
> table is big enough, and the master operation takes more than 1 min to
> complete. However while trying to increase the timeout, we noticed that there
> are two parameters with very similar names configuring different things:
> {{hbase.snapshot.master.timeout.millis}} is defined in
> SnapshotDescriptionUtils and is send to client side and used in disabled
> table snapshot.
> {{hbase.snapshot.master.timeoutMillis}} is defined in SnapshotManager and
> used as the timeout for the procedure execution.
> So, there are a couple of improvements that we can do:
> - 1 min is too low for big tables. We need to set this to 5 min or 10 min by
> default. Even a 6T table which is medium sized fails.
> - Unify the two timeouts into one. Decide on either of them, and deprecate
> the other. Use the biggest one for BC.
> - Add the timeout to hbase-default.xml.
> - Why do we even have a timeout for disabled table snapshots? The master is
> doing the work so we should not timeout in any case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14014) Explore row-by-row grouping options
[ https://issues.apache.org/jira/browse/HBASE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042070#comment-15042070 ] Sean Busbey commented on HBASE-14014: - Parent was HBASE-12988 > Explore row-by-row grouping options > --- > > Key: HBASE-14014 > URL: https://issues.apache.org/jira/browse/HBASE-14014 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Lars Hofhansl > > See discussion in parent. > We need to considering the following attributes of WALKey: > * The cluster ids > * Table Name > * write time (here we could use the latest of any batch) > * seqNum > As long as we preserve these we can rearrange the cells between WALEdits. > Since seqNum is unique this will be a challenge. Currently it is not used, > but we shouldn't design anything that prevents us guaranteeing better > ordering guarantees using seqNum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14014) Explore row-by-row grouping options
[ https://issues.apache.org/jira/browse/HBASE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14014: Issue Type: Improvement (was: Sub-task) Parent: (was: HBASE-12988) > Explore row-by-row grouping options > --- > > Key: HBASE-14014 > URL: https://issues.apache.org/jira/browse/HBASE-14014 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Lars Hofhansl > > See discussion in parent. > We need to considering the following attributes of WALKey: > * The cluster ids > * Table Name > * write time (here we could use the latest of any batch) > * seqNum > As long as we preserve these we can rearrange the cells between WALEdits. > Since seqNum is unique this will be a challenge. Currently it is not used, > but we shouldn't design anything that prevents us guaranteeing better > ordering guarantees using seqNum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13330) Region left unassigned due to AM & SSH each thinking the assignment would be done by the other
[
https://issues.apache.org/jira/browse/HBASE-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-13330:
Fix Version/s: (was: 1.2.1)
1.2.0
> Region left unassigned due to AM & SSH each thinking the assignment would be
> done by the other
> --
>
> Key: HBASE-13330
> URL: https://issues.apache.org/jira/browse/HBASE-13330
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
>Affects Versions: 1.0.0
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Fix For: 1.2.0, 1.3.0, 1.0.3, 0.98.16, 1.1.4
>
> Attachments: 13330-branch-1.txt, 13330-v2-branch-1.txt,
> 13330-v3-branch-1.txt
>
>
> Here is what I found during analysis of an issue. Raising this jira and a fix
> will follow.
> The TL;DR of this is that the AssignmentManager thinks the
> ServerShutdownHandler would assign the region and the ServerShutdownHandler
> thinks that the AssignmentManager would assign the region. The region
> (0d6cf37c18c54c6f4744750c6a7be837) ultimately never gets assigned. Below is
> an analysis from the logs that captures the flow of events.
> 1. The AssignmentManager had initially assigned this region to
> dnj1-bcpc-r3n8.example.com,60020,1425598187703
> 2. When the master restarted it did a scan of the meta to learn about the
> regions in the cluster. It found this region being assigned to
> dnj1-bcpc-r3n8.example.com,60020,1425598187703 from the meta record.
> 3. However, this server (dnj1-bcpc-r3n8.example.com,60020,1425598187703) was
> not alive anymore. So, the AssignmentManager queued up a
> ServerShutdownHandling task for this (that asynchronously executes):
> {noformat}
> 2015-03-06 14:09:31,355 DEBUG org.apache.hadoop.hbase.master.ServerManager:
> Added=dnj1-bcpc-r3n8.example.com,60020,1425598187703 to dead servers,
> submitted shutdown handler to be executed meta=false
> {noformat}
> 4. The AssignmentManager proceeded to read the RIT nodes from ZK. It found
> this region as well:
> {noformat}
> 2015-03-06 14:09:31,527 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Processing
> 0d6cf37c18c54c6f4744750c6a7be837
> in state: RS_ZK_REGION_FAILED_OPEN
> {noformat}
> 5. The region was moved to CLOSED state:
> {noformat}
> 2015-03-06 14:09:31,527 WARN org.apache.hadoop.hbase.master.RegionStates:
> 0d6cf37c18c54c6f4744750c6a7be837 moved to CLOSED on
> dnj1-bcpc-r3n2.example.com,60020,1425603618259, expected
> dnj1-bcpc-r3n8.example.com,60020,1425598187703
> {noformat}
> Note the reference to dnj1-bcpc-r3n2.example.com,60020,1425603618259. This
> means that the region was assigned to
> dnj1-bcpc-r3n2.example.com,60020,1425603618259 but that regionserver couldn't
> open the region for some reason, and it changed the state to
> RS_ZK_REGION_FAILED_OPEN in RIT znode on ZK.
> 6. After that the AssignmentManager tried to assign it again. However, the
> assignment didn't happen because the ServerShutdownHandling task queued
> earlier didn't yet execute:
> {noformat}
> 2015-03-06 14:09:31,527 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Skip assigning
> phMonthlyVersion,\x89\x80\x00\x00,1423149098980.0d6cf37c18c54c6f4744750c6a7be837.,
> it's host dnj1-bcpc-r3n8.example.com,60020,1425598187703 is dead but not
> processed yet
> {noformat}
> 7. Eventually the ServerShutdownHandling task executed.
> {noformat}
> 2015-03-06 14:09:35,188 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs
> for dnj1-bcpc-r3n8.example.com,60020,1425598187703 before assignment.
> 2015-03-06 14:09:35,209 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 19
> region(s) that dnj1-bcpc-r3n8.example.com,60020,1425598187703 was
> carrying (and 0 regions(s) that were opening on this server)
> 2015-03-06 14:09:35,211 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished
> processing of shutdown of dnj1-bcpc-r3n8.example.com,60020,1425598187703
> {noformat}
> 8. However, the ServerShutdownHandling task skipped the region in question.
> This was because this region was in RIT, and the ServerShutdownHandling task
> thinks that the AssignmentManager would assign it as part of handling the RIT
> nodes:
> {noformat}
> 2015-03-06 14:09:35,210 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Skip assigning
> region in transition on other server{0d6cf37c18c54c6f4744750c6a7be837
> state=CLOSED, ts=1425668971527,
> server=dnj1-bcpc-r3n2.example.com,60020,1425603618259}
> {noformat}
> 9. At some point in the future, when the server
> dnj1-bcpc-r3n2.example.com,60020,1425603618259 dies, the
> ServerShutdownHandling for it gets queued up (from the log
> hbase-hbase-master-dnj1-bcpc-r
[jira] [Updated] (HBASE-14930) check_compatibility.sh needs smarter exit codes
[ https://issues.apache.org/jira/browse/HBASE-14930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak updated HBASE-14930: Attachment: HBASE-14930_master_v1.patch Patch that changes the exit codes we use during the setup before running the tool from "1" to "2." This lets us continue to use the script's exit code overall as the exit code of the Java API Compliance Checker tool. > check_compatibility.sh needs smarter exit codes > --- > > Key: HBASE-14930 > URL: https://issues.apache.org/jira/browse/HBASE-14930 > Project: HBase > Issue Type: Bug >Reporter: Dima Spivak >Assignee: Dima Spivak > Attachments: HBASE-14930_master_v1.patch > > > The check_compatibility.sh tool in dev_support uses the Java API Compliance > Checker to do static analysis of source/binary incompatibilties between two > HBase branches. One problem, though, is that the script has a few instances > where it may return an exit code of 1 (e.g. if Maven steps fail), but this is > the same exit code that the Java ACC tool itself uses to denote that the tool > succeeded, but found incompatibilities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14930) check_compatibility.sh needs smarter exit codes
[ https://issues.apache.org/jira/browse/HBASE-14930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak updated HBASE-14930: Status: Patch Available (was: Open) > check_compatibility.sh needs smarter exit codes > --- > > Key: HBASE-14930 > URL: https://issues.apache.org/jira/browse/HBASE-14930 > Project: HBase > Issue Type: Bug >Reporter: Dima Spivak >Assignee: Dima Spivak > Attachments: HBASE-14930_master_v1.patch > > > The check_compatibility.sh tool in dev_support uses the Java API Compliance > Checker to do static analysis of source/binary incompatibilties between two > HBase branches. One problem, though, is that the script has a few instances > where it may return an exit code of 1 (e.g. if Maven steps fail), but this is > the same exit code that the Java ACC tool itself uses to denote that the tool > succeeded, but found incompatibilities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14580) Make the HBaseMiniCluster compliant with Kerberos
[
https://issues.apache.org/jira/browse/HBASE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14580:
Fix Version/s: (was: 1.2.1)
1.2.0
> Make the HBaseMiniCluster compliant with Kerberos
> -
>
> Key: HBASE-14580
> URL: https://issues.apache.org/jira/browse/HBASE-14580
> Project: HBase
> Issue Type: Improvement
> Components: security, test
>Affects Versions: 2.0.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: hbase-14580.v2.patch, hbase-14580.v2.patch,
> patch-14580.v1.patch
>
>
> Whne using MiniKDC and the minicluster in a unit test, there is a conflict
> causeed by HBaseTestingUtility:
> {code}
> public static User getDifferentUser(final Configuration c,
> final String differentiatingSuffix)
> throws IOException {
>// snip
> String username = User.getCurrent().getName() +
> differentiatingSuffix; < problem here
> User user = User.createUserForTesting(c, username,
> new String[]{"supergroup"});
> return user;
> }
> {code}
> This creates users like securedUser/localh...@example.com.hfs.0, and this
> does not work.
> My fix is to return the current user when Kerberos is set. I don't think that
> there is another option (any other opinion?). However this user is not in a
> group so we have logs like 'WARN [IPC Server handler 9 on 61366]
> security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521))
> - No groups available for user securedUser' I'm not sure of its impact.
> [~apurtell], what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-14580) Make the HBaseMiniCluster compliant with Kerberos
[
https://issues.apache.org/jira/browse/HBASE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-14580:
-
> Make the HBaseMiniCluster compliant with Kerberos
> -
>
> Key: HBASE-14580
> URL: https://issues.apache.org/jira/browse/HBASE-14580
> Project: HBase
> Issue Type: Improvement
> Components: security, test
>Affects Versions: 2.0.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: hbase-14580.v2.patch, hbase-14580.v2.patch,
> patch-14580.v1.patch
>
>
> Whne using MiniKDC and the minicluster in a unit test, there is a conflict
> causeed by HBaseTestingUtility:
> {code}
> public static User getDifferentUser(final Configuration c,
> final String differentiatingSuffix)
> throws IOException {
>// snip
> String username = User.getCurrent().getName() +
> differentiatingSuffix; < problem here
> User user = User.createUserForTesting(c, username,
> new String[]{"supergroup"});
> return user;
> }
> {code}
> This creates users like securedUser/localh...@example.com.hfs.0, and this
> does not work.
> My fix is to return the current user when Kerberos is set. I don't think that
> there is another option (any other opinion?). However this user is not in a
> group so we have logs like 'WARN [IPC Server handler 9 on 61366]
> security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521))
> - No groups available for user securedUser' I'm not sure of its impact.
> [~apurtell], what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14089) Remove unnecessary draw of system entropy from RecoverableZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14089: Fix Version/s: (was: 1.2.1) 1.2.0 > Remove unnecessary draw of system entropy from RecoverableZooKeeper > --- > > Key: HBASE-14089 > URL: https://issues.apache.org/jira/browse/HBASE-14089 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0 > > Attachments: HBASE-14089.patch > > > I had a look at instances where we use SecureRandom, which could block if > insufficient entropy, in the 0.98 and master branch code. (Random in contrast > is a PRNG seeded by System#nanoTime, it doesn't draw from system entropy.) > Most uses are in encryption related code, our native encryption and SSL, but > we do also use SecureRandom for salting znode metadata in > RecoverableZooKeeper#appendMetadata, which is called whenever we do setData. > Conceivably we could block unexpectedly when constructing data to write out > to a znode if entropy gets too low until more is available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14580) Make the HBaseMiniCluster compliant with Kerberos
[
https://issues.apache.org/jira/browse/HBASE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-14580.
-
Resolution: Fixed
> Make the HBaseMiniCluster compliant with Kerberos
> -
>
> Key: HBASE-14580
> URL: https://issues.apache.org/jira/browse/HBASE-14580
> Project: HBase
> Issue Type: Improvement
> Components: security, test
>Affects Versions: 2.0.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: hbase-14580.v2.patch, hbase-14580.v2.patch,
> patch-14580.v1.patch
>
>
> Whne using MiniKDC and the minicluster in a unit test, there is a conflict
> causeed by HBaseTestingUtility:
> {code}
> public static User getDifferentUser(final Configuration c,
> final String differentiatingSuffix)
> throws IOException {
>// snip
> String username = User.getCurrent().getName() +
> differentiatingSuffix; < problem here
> User user = User.createUserForTesting(c, username,
> new String[]{"supergroup"});
> return user;
> }
> {code}
> This creates users like securedUser/localh...@example.com.hfs.0, and this
> does not work.
> My fix is to return the current user when Kerberos is set. I don't think that
> there is another option (any other opinion?). However this user is not in a
> group so we have logs like 'WARN [IPC Server handler 9 on 61366]
> security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521))
> - No groups available for user securedUser' I'm not sure of its impact.
> [~apurtell], what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14021) Quota table has a wrong description on the UI
[ https://issues.apache.org/jira/browse/HBASE-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14021: Fix Version/s: (was: 1.2.1) 1.2.0 > Quota table has a wrong description on the UI > - > > Key: HBASE-14021 > URL: https://issues.apache.org/jira/browse/HBASE-14021 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 1.1.0 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 1.3.0 > > Attachments: HBASE-14021.patch, HBASE-14021.patch, > HBASE-14021branch-1.1.patch, error.png, fix.png > > > !error.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14758) Add UT case for unchecked error/exception thrown in AsyncProcess#sendMultiAction
[ https://issues.apache.org/jira/browse/HBASE-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14758: Fix Version/s: (was: 1.2.1) 1.2.0 > Add UT case for unchecked error/exception thrown in > AsyncProcess#sendMultiAction > > > Key: HBASE-14758 > URL: https://issues.apache.org/jira/browse/HBASE-14758 > Project: HBase > Issue Type: Test >Affects Versions: 1.1.2, 0.98.15 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.17, 1.0.4 > > Attachments: HBASE-14758.branch-0.98.patch, HBASE-14758.patch > > > This is an addendum for HBASE-14359, open a new JIRA to trigger HadoopQA run -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14539) Slight improvement of StoreScanner.optimize
[ https://issues.apache.org/jira/browse/HBASE-14539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14539: Fix Version/s: (was: 1.2.1) 1.2.0 > Slight improvement of StoreScanner.optimize > --- > > Key: HBASE-14539 > URL: https://issues.apache.org/jira/browse/HBASE-14539 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3 > > Attachments: 14539-0.98.txt > > > While looking at the code I noticed that StoreScanner.optimize does not some > unnecessary work. This is a very tight loop and even just looking up a > reference can throw off the CPUs cache lines. This does safe a few percent of > performance (not a lot, though). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14734) TestGenerateDelegationToken fails with BindAddress clash
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042111#comment-15042111 ] stack commented on HBASE-14734: --- Again https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/lastCompletedBuild/jdk=latest1.7,label=Hadoop/testReport/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > TestGenerateDelegationToken fails with BindAddress clash > > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack > > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14891) Add log for uncaught exception in RegionServerMetricsWrapperRunnable
[
https://issues.apache.org/jira/browse/HBASE-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14891:
Fix Version/s: (was: 1.2.1)
1.2.0
> Add log for uncaught exception in RegionServerMetricsWrapperRunnable
>
>
> Key: HBASE-14891
> URL: https://issues.apache.org/jira/browse/HBASE-14891
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14891.branch-1.2.patch,
> HBASE-14891.branch-1.patch, HBASE-14891.patch, HBASE-14891.v2.patch,
> HBASE-14891.v2.patch
>
>
> We are using ScheduledExecutorService#scheduleWithFixedDelay to periodically
> reporting metrics. From javadoc of this method
> {quote}
> If any execution of the task encounters an exception, subsequent executions
> are suppressed.
> {quote}
> We could see if any uncaught exception like NPE thrown in the task thread,
> there won't be any error thrown but the task would be silently terminated.
> In our real case we were making some modification to the metrics and causing
> NPE carelessly, then we found metrics disappeared but cannot get a clue from
> RS log.
> Suggest to catch Throwable in
> MetricsRegionServerWrapperImpl$RegionServerMetricsWrapperRunnable, add some
> log there and suppress the exception to give it another try at next scheduled
> run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14903) Table Or Region?
[ https://issues.apache.org/jira/browse/HBASE-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042122#comment-15042122 ] Andrew Purtell commented on HBASE-14903: Prepare a patch for the site documentation and attach it here. > Table Or Region? > > > Key: HBASE-14903 > URL: https://issues.apache.org/jira/browse/HBASE-14903 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: 胡托 >Priority: Blocker > > I've been reading on Latest Reference Guide and try to translated into > Chinese! > I think this sentence "When a table is in the process of splitting," > should be "When a Region is in the process of splitting," on chapter 【62.2. > hbase:meta】。 > By the way,is this document the > latest?【http://hbase.apache.org/book.html#arch.overview】I will translate it！ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14354) Minor improvements for usage of the mlock agent
[ https://issues.apache.org/jira/browse/HBASE-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14354: Fix Version/s: (was: 1.2.1) 1.2.0 > Minor improvements for usage of the mlock agent > --- > > Key: HBASE-14354 > URL: https://issues.apache.org/jira/browse/HBASE-14354 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.1.3 > > Attachments: > 0001-HBASE-14354-Minor-improvements-for-usage-of-the-mloc.patch, > HBASE-14354.v1.patch > > > 1. MLOCK_AGENT points to the wrong path in hbase-config.sh > When the mlock agent is build, the binary is installed under > $HBASE_HOME/lib/native and not under $HBASE_HOME/native > 2. By default we pass $HBASE_REGIONSERVER_UID to the agent options which > causes the mlock agent to attempt to do a setuid in order to mlock the memory > of the RS process. We should only pass that user if specified in the > environment, not by default. (the agent currently handles that gracefully) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14928) Start row should be set for query through HBase REST gateway involving globbing option
[ https://issues.apache.org/jira/browse/HBASE-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14928: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review, Jerry. > Start row should be set for query through HBase REST gateway involving > globbing option > -- > > Key: HBASE-14928 > URL: https://issues.apache.org/jira/browse/HBASE-14928 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17 > > Attachments: 14928-v1.txt > > > As Ben Sutton reported in the thread, Slow response on HBase REST api using > globbing option, query through the Rest API with a globbing option i.e. > http://:/table/key\* executes extremely slowly. > Jerry He pointed out that PrefixFilter is used for query involving globbing > option. > This issue is to fix this bug by setting start row for such queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Status: Open (was: Patch Available) > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch, HBASE-14901.5.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Status: Patch Available (was: Open) > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch, HBASE-14901.5.patch, HBASE-14901.6.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14901) There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-14901: -- Attachment: HBASE-14901.6.patch Merge in HStore from change yesterday... > There is duplicated code to create/manage encryption keys > - > > Key: HBASE-14901 > URL: https://issues.apache.org/jira/browse/HBASE-14901 > Project: HBase > Issue Type: Bug > Components: encryption >Affects Versions: 2.0.0 >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14901.1.patch, HBASE-14901.2.patch, > HBASE-14901.3.patch, HBASE-14901.5.patch, HBASE-14901.6.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14916) Add checkstyle_report.py to other branches
[ https://issues.apache.org/jira/browse/HBASE-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14916: - Attachment: HBASE-14916-branch-1-v2.patch uploading exact same patch as 'v2' to trigger build. > Add checkstyle_report.py to other branches > -- > > Key: HBASE-14916 > URL: https://issues.apache.org/jira/browse/HBASE-14916 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14916-branch-1-v2.patch, HBASE-14916-branch-1.patch > > > Given test-patch.sh is always run from master, and that it now uses > checkstyle_report.py, we should pull back the script to other branches too. > Otherwise we see error like: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/jenkins.build/dev-support/test-patch.sh: > line 662: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/dev-support/checkstyle_report.py: > No such file or directory > [reference|https://builds.apache.org/job/PreCommit-HBASE-Build/16734//consoleFull] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14917) Log in console if individual tests in test-patch.sh fail or pass.
[ https://issues.apache.org/jira/browse/HBASE-14917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14917: - Attachment: HBASE-14917-v2.patch v2: same patch, new name to trigger the build > Log in console if individual tests in test-patch.sh fail or pass. > - > > Key: HBASE-14917 > URL: https://issues.apache.org/jira/browse/HBASE-14917 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-14917-v2.patch, HBASE-14917.patch > > > Got 2 runs like > https://issues.apache.org/jira/browse/HBASE-14865?focusedCommentId=15037056&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15037056 > where can't figure out what went wrong in patch testing. > Logging results from individual tests to console as discussed > [here|https://mail-archives.apache.org/mod_mbox/hbase-dev/201512.mbox/%3CCAAjhxrrL4-qty562%3DcMyBJ2xyhGqHi3MFAgf9ygrzQf1%2BZmHtw%40mail.gmail.com%3E] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042228#comment-15042228
]
Zhe Zhang commented on HBASE-14790:
---
Thanks for the catch. If needed I guess we can make it public. Or at least have
a public API to return the acked length.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14926) Hung ThriftServer; no timeout on read from client; if client crashes, worker thread gets stuck reading
[
https://issues.apache.org/jira/browse/HBASE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14926:
--
Attachment: 14926v2.txt
Ok. Played more trying to capture thriftserver in described state. I saw the
issue in thread dump but proved elusive (wasn't sure if it my timeout that
fixed it or not). The change is pretty basic so should be safe enough...
setting a timeout on server socket... which we want anyways. Going to commit
on back of @apurtell +1. v2 adds logging of the timeout to thrift2 version too
and adds a sentence or two more to examples on how to get going with thrift
server.
> Hung ThriftServer; no timeout on read from client; if client crashes, worker
> thread gets stuck reading
> --
>
> Key: HBASE-14926
> URL: https://issues.apache.org/jira/browse/HBASE-14926
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 1.0.3, 0.98.16
>Reporter: stack
>Assignee: stack
> Attachments: 14926.patch, 14926v2.txt
>
>
> Thrift server is hung. All worker threads are doing this:
> {code}
> "thrift-worker-0" daemon prio=10 tid=0x7f0bb95c2800 nid=0xf6a7 runnable
> [0x7f0b956e]
>java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00066d859490> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
> at
> org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
> at
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> They never recover.
> I don't have client side logs.
> We've been here before: HBASE-4967 "connected client thrift sockets should
> have a server side read timeout" but this patch only got applied to fb branch
> (and thrift has changed since then).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042285#comment-15042285 ] Enis Soztutar commented on HBASE-6721: -- Thanks [~toffer] for the updated patch. All of the review items have been addressed, and hence I'm +1 for committing this. [~apurtell], [~eclark] or others have any more concerns, feedback? Given that this is an optional coprocessor based thing, I think this can directly go to master. > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch, > hbase-6721-v18.patch, immediateAssignments Sequence Diagram.svg, > randomAssignment Sequence Diagram.svg, retainAssignment Sequence Diagram.svg, > roundRobinAssignment Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042297#comment-15042297 ] Enis Soztutar commented on HBASE-14925: --- I like the idea for a way to iterate over rows. But an CVS like output out of the box for all regions or all regions of a table, which is parsable by other scripts would be very useful. > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: huaxiang sun > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14926) Hung ThriftServer; no timeout on read from client; if client crashes, worker thread gets stuck reading
[
https://issues.apache.org/jira/browse/HBASE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14926:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
1.2.0
2.0.0
Release Note: Adds a timeout to server read from clients. Adds new configs
hbase.thrift.server.socket.read.timeout for setting read timeout on server
socket in milliseconds. Default is 6;
Status: Resolved (was: Patch Available)
Pushed to 1.2+ Thrift starts to vary if I go back more. Let me know if want it
on 0.98 and I'll hack it up.
> Hung ThriftServer; no timeout on read from client; if client crashes, worker
> thread gets stuck reading
> --
>
> Key: HBASE-14926
> URL: https://issues.apache.org/jira/browse/HBASE-14926
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 1.0.3, 0.98.16
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14926.patch, 14926v2.txt
>
>
> Thrift server is hung. All worker threads are doing this:
> {code}
> "thrift-worker-0" daemon prio=10 tid=0x7f0bb95c2800 nid=0xf6a7 runnable
> [0x7f0b956e]
>java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00066d859490> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
> at
> org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
> at
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> They never recover.
> I don't have client side logs.
> We've been here before: HBASE-4967 "connected client thrift sockets should
> have a server side read timeout" but this patch only got applied to fb branch
> (and thrift has changed since then).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042319#comment-15042319
]
Ted Yu commented on HBASE-6721:
---
Running TestShell with latest patch, I got:
{code}
1) Error:
test_Test_Basic_Group_Commands(Hbase::GroupShellTest):
NativeException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException:
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered
master coprocessor service found for name hbase.pb.GroupAdminService
at
org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
...
2) Failure:
test_Test_bogus_arguments(Hbase::GroupShellTest)
[./src/test/ruby/shell/group_shell_test.rb:85:in `test_Test_bogus_arguments'
org/jruby/RubyProc.java:270:in `call'
org/jruby/RubyKernel.java:2105:in `send'
org/jruby/RubyArray.java:1620:in `each'
org/jruby/RubyArray.java:1620:in `each']:
exception expected but was
Class:
Message: <"org.apache.hadoop.hbase.exceptions.UnknownProtocolException:
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered
master coprocessor service found for name hbase.pb.GroupAdminService\n\tat
org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682)\n\tat
org.apache.hadoop.hbase.
protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964)\n\tat
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184)\n\tat org.
apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)\n\tat
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)\n\tat
org.apache.hadoop.hbase.ipc. RpcExecutor$1.run(RpcExecutor.java:108)\n\tat
java.lang.Thread.run(Thread.java:745)\n">
{code}
Francis:
Do the tests pass for you ?
> RegionServer Group based Assignment
> ---
>
> Key: HBASE-6721
> URL: https://issues.apache.org/jira/browse/HBASE-6721
> Project: HBase
> Issue Type: New Feature
>Reporter: Francis Liu
>Assignee: Francis Liu
> Labels: hbase-6721
> Attachments: 6721-master-webUI.patch, HBASE-6721
> GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf,
> HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf,
> HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch,
> HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch,
> HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch,
> HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch,
> HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch,
> HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch,
> HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch,
> HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch,
> HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch,
> HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg,
> hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch,
> hbase-6721-v18.patch, immediateAssignments Sequence Diagram.svg,
> randomAssignment Sequence Diagram.svg, retainAssignment Sequence Diagram.svg,
> roundRobinAssignment Sequence Diagram.svg
>
>
> In multi-tenant deployments of HBase, it is likely that a RegionServer will
> be serving out regions from a number of different tables owned by various
> client applications. Being able to group a subset of running RegionServers
> and assign specific tables to it, provides a client application a level of
> isolation and resource allocation.
> The proposal essentially is to have an AssignmentManager which is aware of
> RegionServer groups and assigns tables to region servers based on groupings.
> Load balancing will occur on a per group basis as well.
> This is essentially a simplification of the approach taken in HBASE-4120. See
> attached document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042321#comment-15042321 ] Ted Yu commented on HBASE-13153: [~anoop.hbase] [~jerryhe]: Any more comment(s) ? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14866) VerifyReplication should use peer configuration in peer connection
[
https://issues.apache.org/jira/browse/HBASE-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-14866:
--
Attachment: hbase-14866-v5.patch
Attaching a new patch that moves the ZK related configuration methods to
ZKConfig, which also moves from hbase-client to hbase-common, in order to make
it available to HBaseConfiguration.createClusterConf().
TestZKConfig moves from hbase-server (don't know why it was there) to
hbase-common, to keep it together with ZKConfig.
I left the {{applyClusterKeyToConf()}} method in HBaseConfiguration, so that it
can be private. This should avoid any future independent use of it.
> VerifyReplication should use peer configuration in peer connection
> --
>
> Key: HBASE-14866
> URL: https://issues.apache.org/jira/browse/HBASE-14866
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14866.patch, HBASE-14866_v1.patch,
> hbase-14866-v4.patch, hbase-14866-v5.patch, hbase-14866_v2.patch,
> hbase-14866_v3.patch
>
>
> VerifyReplication uses the replication peer's configuration to construct the
> ZooKeeper quorum address for the peer connection. However, other
> configuration properties in the peer's configuration are dropped. It should
> merge all configuration properties from the {{ReplicationPeerConfig}} when
> creating the peer connection and obtaining a credentials for the peer cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14795) Enhance the spark-hbase scan operations
[ https://issues.apache.org/jira/browse/HBASE-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042336#comment-15042336 ] Ted Yu commented on HBASE-14795: In the groups field for reviewers, please fill in hbase so that people would receive notifications of reviews. > Enhance the spark-hbase scan operations > --- > > Key: HBASE-14795 > URL: https://issues.apache.org/jira/browse/HBASE-14795 > Project: HBase > Issue Type: Improvement >Reporter: Ted Malaska >Assignee: Zhan Zhang >Priority: Minor > Attachments: > 0001-HBASE-14795-Enhance-the-spark-hbase-scan-operations.patch > > > This is a sub-jira of HBASE-14789. This jira is to focus on the replacement > of TableInputFormat for a more custom scan implementation that will make the > following use case more effective. > Use case: > In the case you have multiple scan ranges on a single table with in a single > query. TableInputFormat will scan the the outer range of the scan start and > end range where this implementation can be more pointed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14769: - Attachment: HBASE-14769-master-v9.patch v6 as v9 to trigger build. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-14769-master-v2.patch, > HBASE-14769-master-v3.patch, HBASE-14769-master-v4.patch, > HBASE-14769-master-v5.patch, HBASE-14769-master-v6.patch, > HBASE-14769-master-v7.patch, HBASE-14769-master-v8.patch, > HBASE-14769-master-v9.patch, HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
