[jira] [Updated] (HBASE-8199) Eliminate exception for ExportSnapshot against the null table snapshot (with no data in)
[ https://issues.apache.org/jira/browse/HBASE-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-8199: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk, 0.95 and 0.94 Thanks Julian for the patch, and Jon for the review. > Eliminate exception for ExportSnapshot against the null table snapshot (with > no data in) > > > Key: HBASE-8199 > URL: https://issues.apache.org/jira/browse/HBASE-8199 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.95.0, 0.94.6, 0.94.7 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.95.0, 0.94.6 > > Attachments: HBASE-8199-0.94.patch, HBASE-8199-trunk.patch, > HBASE-8199-with-test.patch > > > In the test environment, when snapshoting the null table (with no data), > snapshot artifact can be generated successfully with snapshot meta data, > while there is no CF data under region dir. Then when exporting this null > snapshot via ExportSnapshot, "java.io.IOException: No input paths specified > in job" occurred, which affects the normal application flow. > But seems snapshot for null table also makes sense just with the meta > information (.snapshot/), although ".archive/" does not show up. Later > snapshot restore seems work from this "dummy" snapshot artifact with no CF > data. > Just eliminate exception for ExportSnapshot against the null table snapshot > to make application work through. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8199) Eliminate exception for ExportSnapshot against the null table snapshot (with no data in)
[ https://issues.apache.org/jira/browse/HBASE-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-8199: --- Fix Version/s: (was: 0.94.6) 0.94.7 > Eliminate exception for ExportSnapshot against the null table snapshot (with > no data in) > > > Key: HBASE-8199 > URL: https://issues.apache.org/jira/browse/HBASE-8199 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.95.0, 0.94.6, 0.94.7 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.95.0, 0.94.7 > > Attachments: HBASE-8199-0.94.patch, HBASE-8199-trunk.patch, > HBASE-8199-with-test.patch > > > In the test environment, when snapshoting the null table (with no data), > snapshot artifact can be generated successfully with snapshot meta data, > while there is no CF data under region dir. Then when exporting this null > snapshot via ExportSnapshot, "java.io.IOException: No input paths specified > in job" occurred, which affects the normal application flow. > But seems snapshot for null table also makes sense just with the meta > information (.snapshot/), although ".archive/" does not show up. Later > snapshot restore seems work from this "dummy" snapshot artifact with no CF > data. > Just eliminate exception for ExportSnapshot against the null table snapshot > to make application work through. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7987) Snapshot Manifest file instead of multiple empty files
[
https://issues.apache.org/jira/browse/HBASE-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615000#comment-13615000
]
Matteo Bertozzi commented on HBASE-7987:
{quote}There seems to be some overlap... Can we reuse some code?{quote}
Yeah they are identical, the only difference is that one iterates over a list
of Store and the other over disk... It's not easy share the code between the
two since is just a for loop and a add to the protobuf. (We have the same
duplication in the old code for disabled table and enabled table).
> Snapshot Manifest file instead of multiple empty files
> --
>
> Key: HBASE-7987
> URL: https://issues.apache.org/jira/browse/HBASE-7987
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Matteo Bertozzi
> Attachments: HBASE-7987.sketch, HBASE-7987-v2.sketch
>
>
> Currently taking a snapshot means creating one empty file for each file in
> the source table directory, plus copying the .regioninfo file for each
> region, the table descriptor file and a snapshotInfo file.
> during the restore or snapshot verification we traverse the filesystem
> (fs.listStatus()) to find the snapshot files, and we open the .regioninfo
> files to get the information.
> to avoid hammering the NameNode and having lots of empty files, we can use a
> manifest file that contains the list of files and information that we need.
> To keep the RS parallelism that we have, each RS can write its own manifest.
> {code}
> message SnapshotDescriptor {
> required string name;
> optional string table;
> optional int64 creationTime;
> optional Type type;
> optional int32 version;
> }
> message SnapshotRegionManifest {
> optional int32 version;
> required RegionInfo regionInfo;
> repeated FamilyFiles familyFiles;
> message StoreFile {
> required string name;
> optional Reference reference;
> }
> message FamilyFiles {
> required bytes familyName;
> repeated StoreFile storeFiles;
> }
> }
> {code}
> {code}
> /hbase/.snapshot/
> /hbase/.snapshot//snapshotInfo
> /hbase/.snapshot//
> /hbase/.snapshot///tableInfo
> /hbase/.snapshot///regionManifest(.n)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8024) Make Store flush algorithm pluggable
[
https://issues.apache.org/jira/browse/HBASE-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615013#comment-13615013
]
Maryann Xue commented on HBASE-8024:
Modification of HStore#internalFlushCache() for our LOB use case:
{code}
private Path internalFlushCacheToBlobStore(final SortedSet set,
final long logCacheFlushId,
TimeRangeTracker snapshotTimeRangeTracker,
AtomicLong flushedSize,
MonitoredTask status)
throws IOException {
StoreFile.Writer writer;
// Find the smallest read point across all the Scanners.
long smallestReadPoint = region.getSmallestReadPoint();
long flushed = 0;
Path referenceFilePath = null;
Path blobFilePath = null;
// Don't flush if there are no entries.
if (set.size() == 0) {
return null;
}
Scan scan = new Scan();
scan.setMaxVersions(scanInfo.getMaxVersions());
// Use a store scanner to find which rows to flush.
// Note that we need to retain deletes, hence
// treat this as a minor compaction.
InternalScanner scanner = new StoreScanner(this, scan, Collections
.singletonList(new CollectionBackedScanner(set, this.comparator)),
ScanType.MINOR_COMPACT, this.region.getSmallestReadPoint(),
HConstants.OLDEST_TIMESTAMP);
BlobStore blobStore =
BlobStoreManager.getInstance().getBlobStore(getTableName(),
family.getNameAsString());
if (null == blobStore) {
blobStore =
BlobStoreManager.getInstance().createBlobStore(getTableName(), family);
}
StoreFile.Writer blobWriter = null;
try {
// TODO: We can fail in the below block before we complete adding this
// flush to list of store files. Add cleanup of anything put on
filesystem
// if we fail.
synchronized (flushLock) {
status.setStatus("Flushing " + this + ": creating writer");
int referenceKeyValueCount = set.size();
int blobKeyValueCount = 0;
// A. Write the map out to the disk
writer = createWriterInTmp(referenceKeyValueCount);
writer.setTimeRangeTracker(snapshotTimeRangeTracker);
referenceFilePath = writer.getPath();
Iterator iter = set.iterator();
while(null != iter && iter.hasNext()) {
if (iter.next().getType() == KeyValue.Type.Put.getCode()) {
blobKeyValueCount++;
}
}
blobWriter = blobStore.createWriterInTmp(blobKeyValueCount,
this.compression,
region.getRegionInfo());
blobFilePath = blobWriter.getPath();
String targetPathName = dateFormatter.format(new Date());
Path targetPath = new Path(blobStore.getHomePath(), targetPathName);
String relativePath = targetPathName + Path.SEPARATOR +
blobFilePath.getName();
// Append the BLOB_STORE_VERSION before the relative path name
byte[] referenceValue = Bytes.add(
new byte[] { BlobStoreConstants.BLOB_STORE_VERSION },
Bytes.toBytes(relativePath));
try {
List kvs = new ArrayList();
boolean hasMore;
do {
hasMore = scanner.next(kvs);
if (!kvs.isEmpty()) {
for (KeyValue kv : kvs) {
// If we know that this KV is going to be included always, then
let us
// set its memstoreTS to 0. This will help us save space when
writing to disk.
if (kv.getMemstoreTS() <= smallestReadPoint) {
// let us not change the original KV. It could be in the
memstore
// changing its memstoreTS could affect other
threads/scanners.
kv = kv.shallowCopy();
kv.setMemstoreTS(0);
}
if (kv.getType() == KeyValue.Type.Reference.getCode()) {
writer.append(kv);
}
else {
// append the original keyValue in the blob file.
blobWriter.append(kv);
// append reference KeyValue.
// The key is same, the value is the blobfile's filename
KeyValue reference = new KeyValue(kv.getBuffer(),
kv.getRowOffset(),
kv.getRowLength(),
kv.getBuffer(),
kv.getFamilyOffset(),
kv.getFamilyLength(),
kv.getBuffer(),
kv.getQualifierOffset(),
kv.getQualifierLength(),
kv.getTimestamp(),
KeyValue.Type.Reference,
referenceValue,
0,
referenceValue.length);
writer.append(reference);
[jira] [Commented] (HBASE-8024) Make Store flush algorithm pluggable
[ https://issues.apache.org/jira/browse/HBASE-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615015#comment-13615015 ] Maryann Xue commented on HBASE-8024: In the LOB use case: add an independent LOB writer for real LOB data, and replace the original value of the KeyValue with LOB file path. > Make Store flush algorithm pluggable > > > Key: HBASE-8024 > URL: https://issues.apache.org/jira/browse/HBASE-8024 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.95.0, 0.96.0, 0.94.5 >Reporter: Maryann Xue >Assignee: Maryann Xue > Attachments: HBASE-8024-trunk.patch, HBASE-8024.v2.patch > > > The idea is to make "StoreFlusher" an interface instead of an implementation > class, and have the original StoreFlusher as the default store flush impl. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8064) hbase connection could not reuse
[ https://issues.apache.org/jira/browse/HBASE-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615075#comment-13615075 ] Yuan Kang commented on HBASE-8064: -- I found in hbase trunk there is no getHRegionConnection() method yet,it have been replaced by getProtocol().this bug may be fixed by someone else. in the trunk code(At revision 1461481) line 1512 "protocols.put(protocol, server);" the map key is replaced by protocol. > hbase connection could not reuse > > > Key: HBASE-8064 > URL: https://issues.apache.org/jira/browse/HBASE-8064 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.94.0 > Environment: hadoop-1.0.2 hbase-0.94.0 >Reporter: Yuan Kang >Assignee: Yuan Kang > Labels: patch > Attachments: HConnectionManager-connection-could-not-reuse.patch > > > when hconnection is used by one matchine,the connection return to the pool. > if anather matchine reget the connection,it can be resued. > but in the code the caching map don't be managered correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8195) hbase.table.archive.directory reference should be removed from hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-8195: --- Status: Open (was: Patch Available) > hbase.table.archive.directory reference should be removed from > hbase-default.xml > > > Key: HBASE-8195 > URL: https://issues.apache.org/jira/browse/HBASE-8195 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-8195-v0-trunk.patch, HBASE-8195-v0-trunk.patch > > > hbase.table.archive.directory is not used anymore. Therefor, it should be > removed from hbase-default.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8195) hbase.table.archive.directory reference should be removed from hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-8195: --- Attachment: HBASE-8195-v0-trunk.patch > hbase.table.archive.directory reference should be removed from > hbase-default.xml > > > Key: HBASE-8195 > URL: https://issues.apache.org/jira/browse/HBASE-8195 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-8195-v0-trunk.patch, HBASE-8195-v0-trunk.patch > > > hbase.table.archive.directory is not used anymore. Therefor, it should be > removed from hbase-default.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8195) hbase.table.archive.directory reference should be removed from hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-8195: --- Status: Patch Available (was: Open) Re-attached. > hbase.table.archive.directory reference should be removed from > hbase-default.xml > > > Key: HBASE-8195 > URL: https://issues.apache.org/jira/browse/HBASE-8195 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-8195-v0-trunk.patch, HBASE-8195-v0-trunk.patch > > > hbase.table.archive.directory is not used anymore. Therefor, it should be > removed from hbase-default.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615146#comment-13615146 ] Nicolas Liochon commented on HBASE-7878: It's a potential data corruption... On the 'no' side, I'm currently doing some mttr tests, in this seems to be on the critical path now. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8195) hbase.table.archive.directory reference should be removed from hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615174#comment-13615174
]
Hadoop QA commented on HBASE-8195:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12575689/HBASE-8195-v0-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5027//console
This message is automatically generated.
> hbase.table.archive.directory reference should be removed from
> hbase-default.xml
>
>
> Key: HBASE-8195
> URL: https://issues.apache.org/jira/browse/HBASE-8195
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
> Attachments: HBASE-8195-v0-trunk.patch, HBASE-8195-v0-trunk.patch
>
>
> hbase.table.archive.directory is not used anymore. Therefor, it should be
> removed from hbase-default.xml
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615185#comment-13615185 ] Ted Yu commented on HBASE-7878: --- I understand that in case false is returned from recoverLease, we would wait longer. One remedy I can think of is to bundle lease recovery for several files together so that the extra wait can be amortized. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8147) Add Integration Test for "The HCat Scenario"
[ https://issues.apache.org/jira/browse/HBASE-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615212#comment-13615212 ] Hudson commented on HBASE-8147: --- Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/]) HBASE-8147. Adds an integration test for HBASE-8140 (issue observed when commitJob was launching another job). (Revision 1461304) Result = FAILURE ddas : Files : * /hbase/branches/0.95/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestImportTsv.java > Add Integration Test for "The HCat Scenario" > > > Key: HBASE-8147 > URL: https://issues.apache.org/jira/browse/HBASE-8147 > Project: HBase > Issue Type: Sub-task > Components: mapreduce, test >Affects Versions: 0.95.0, 0.98.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: > 0001-HBASE-8147-Add-test-jobs-launched-based-on-other-job.patch > > > HBASE-8140 needs an integration test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8172) Fix ./bin/*.rb scripts or remove them
[
https://issues.apache.org/jira/browse/HBASE-8172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615213#comment-13615213
]
Hudson commented on HBASE-8172:
---
Integrated in hbase-0.95-on-hadoop2 #45 (See
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/])
HBASE-8172 Fix ./bin/*.rb scripts or remove them (Revision 1461164)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.95/bin/get-active-master.rb
* /hbase/branches/0.95/bin/region_mover.rb
* /hbase/branches/0.95/bin/region_status.rb
> Fix ./bin/*.rb scripts or remove them
> -
>
> Key: HBASE-8172
> URL: https://issues.apache.org/jira/browse/HBASE-8172
> Project: HBase
> Issue Type: Task
> Components: scripts
>Reporter: stack
>Assignee: rajeshbabu
>Priority: Blocker
> Fix For: 0.95.0
>
> Attachments: HBASE-8172.patch
>
>
> I started to review the bin content to see what had rotted and what had not.
> The first script I tried doesn't say how to run it and when I did it did this:
> {code}
> durruti:hbase-0.97-SNAPSHOT stack$ ./bin/get-active-master.rb
> env: hbase-jruby: No such file or directory
> durruti:hbase-0.97-SNAPSHOT stack$ ./bin/hbase org.jruby.Main
> bin/get-active-master.rb
> NativeException: java.lang.IllegalArgumentException: Not a host:port pair:
> PBUF
> 172.21.3.100??
> __ensure__ at bin/get-active-master.rb:37
> (root) at bin/get-active-master.rb:34
> durruti:hbase-0.97-SNAPSHOT stack$ vi bin/
> {code}
> Check and fix all *.rb scripts or remove.
> Blocker on 0.95.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615214#comment-13615214 ] Hudson commented on HBASE-8163: --- Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/]) HBASE-8163 MemStoreChunkPool: An improvement for JAVA GC when using MSLAB (Revision 1461399) Result = FAILURE zjushch : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.95.0, 0.98.0 > > Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, > hbase-8163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch, > hbase-8163v5.patch, hbase-8163v6.patch > > > *Usage:* > Disable pool(default): configure "hbase.hregion.memstore.chunkpool.maxsize" > as 0 > Enable pool: configure "hbase.hregion.memstore.chunkpool.maxsize" as a > percentage of global memstore size (between 0.0 and 1.0, recommend to set it > as the gap between min and max sizes of memstore, e.g. 0.5) > *Background*： > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*： > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8140) TableMapReduceUtils#addDependencyJar fails when nested inside another MR job
[ https://issues.apache.org/jira/browse/HBASE-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615215#comment-13615215 ] Hudson commented on HBASE-8140: --- Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/]) HBASE-8147. Adds an integration test for HBASE-8140 (issue observed when commitJob was launching another job). (Revision 1461304) Result = FAILURE ddas : Files : * /hbase/branches/0.95/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestImportTsv.java > TableMapReduceUtils#addDependencyJar fails when nested inside another MR job > > > Key: HBASE-8140 > URL: https://issues.apache.org/jira/browse/HBASE-8140 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: 0001-HBASE-8140-addendum-add-test-category.patch, > 8140-port-jarfinder-0.94.patch, 8140-port-jarfinder-trunk.patch > > > TableMapReduceUtils#addDependencyJar is used when configuring a mapreduce job > to make sure dependencies of the job are shipped to the cluster. The code > depends on finding an actual jar file containing the necessary classes. This > is not always the case, for instance, when run at the end of another > mapreduce job. In that case, dependency jars have already been shipped to the > cluster and expanded in the parent job's run folder. Those dependencies are > there, just not available as jars. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8005) DemoClient.cpp can not be built correctly
[ https://issues.apache.org/jira/browse/HBASE-8005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615216#comment-13615216 ] Hudson commented on HBASE-8005: --- Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/]) HBASE-8005 DemoClient.cpp can not be built correctly (Revision 1461178) Result = FAILURE stack : Files : * /hbase/branches/0.95/hbase-examples/src/main/cpp/Makefile > DemoClient.cpp can not be built correctly > --- > > Key: HBASE-8005 > URL: https://issues.apache.org/jira/browse/HBASE-8005 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 0.94.5 >Reporter: Ling Kun > Fix For: 0.95.0 > > Attachments: 0001-solve-a-DemoClient.cpp-build-problem.patch, > HBASE-8005-v0-0.94.patch, HBASE-8005-v0.patch, makefile.patch > > > when compile the DemoClient.cpp within the latest HBase release package, it > seems that this file is out of date(plenty of APIs need a new std::map Text> & attributes parameter ). And I have also tried to replace the > DemoClient.cpp with the latest trunk version. However, after some > modification there are still link problems when compile the code( undefined > reference to `apache::thrift::transport::TSocket::TSocket(std::string, > int)', I have passed the -lthrift parameter to gcc). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8199) Eliminate exception for ExportSnapshot against the null table snapshot (with no data in)
[ https://issues.apache.org/jira/browse/HBASE-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615217#comment-13615217 ] Hudson commented on HBASE-8199: --- Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/]) HBASE-8199 Eliminate exception for ExportSnapshot against the null table snapshot (Julian Zhou) (Revision 1461344) Result = FAILURE mbertozzi : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java > Eliminate exception for ExportSnapshot against the null table snapshot (with > no data in) > > > Key: HBASE-8199 > URL: https://issues.apache.org/jira/browse/HBASE-8199 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.95.0, 0.94.6, 0.94.7 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.95.0, 0.94.7 > > Attachments: HBASE-8199-0.94.patch, HBASE-8199-trunk.patch, > HBASE-8199-with-test.patch > > > In the test environment, when snapshoting the null table (with no data), > snapshot artifact can be generated successfully with snapshot meta data, > while there is no CF data under region dir. Then when exporting this null > snapshot via ExportSnapshot, "java.io.IOException: No input paths specified > in job" occurred, which affects the normal application flow. > But seems snapshot for null table also makes sense just with the meta > information (.snapshot/), although ".archive/" does not show up. Later > snapshot restore seems work from this "dummy" snapshot artifact with no CF > data. > Just eliminate exception for ExportSnapshot against the null table snapshot > to make application work through. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7908) Add jstack logs in hbase-common.sh when killing the server.
[ https://issues.apache.org/jira/browse/HBASE-7908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615218#comment-13615218 ] Hudson commented on HBASE-7908: --- Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/]) HBASE-7908 Add jstack logs in hbase-common.sh when killing the server (Revision 1461166) Result = FAILURE stack : Files : * /hbase/branches/0.95/bin/hbase-common.sh > Add jstack logs in hbase-common.sh when killing the server. > --- > > Key: HBASE-7908 > URL: https://issues.apache.org/jira/browse/HBASE-7908 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.95.0 > > Attachments: HBASE-7908-v0-trunk.patch, HBASE-7908-v1-trunk.patch > > > It might be a good idea to keep the jstack logs when killing -9 never ending > servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7597) TestHBaseFsck#testRegionShouldNotBeDeployed seems to be flaky
[ https://issues.apache.org/jira/browse/HBASE-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615219#comment-13615219 ] Hudson commented on HBASE-7597: --- Integrated in hbase-0.95-on-hadoop2 #45 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/45/]) HBASE-7597 TestHBaseFsck#testRegionShouldNotBeDeployed seems to be flaky, v2 (Revision 1461229) Result = FAILURE jxiang : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > TestHBaseFsck#testRegionShouldNotBeDeployed seems to be flaky > - > > Key: HBASE-7597 > URL: https://issues.apache.org/jira/browse/HBASE-7597 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jimmy Xiang > Fix For: 0.95.0, 0.98.0 > > Attachments: trunk-7597.patch, trunk-7597_v2.patch > > > I ran the entire test suite many times and always failed on, at least, > testRegionShouldNotBeDeployed. > Results below. I will attached more result when current tests are done. > Failed tests: > testDeleteExpiredStoreFiles(org.apache.hadoop.hbase.regionserver.TestStore): > expected:<2> but was:<4> > > testAcquireTaskAtStartup(org.apache.hadoop.hbase.regionserver.TestSplitLogWorker): > Waiting timed out after [1 000] msec > testRegionShouldNotBeDeployed(org.apache.hadoop.hbase.util.TestHBaseFsck): > expected:<[SHOULD_NOT_BE_DEPLOYED]> but was:<[]> > > testPermissionsWatcher(org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8172) Fix ./bin/*.rb scripts or remove them
[
https://issues.apache.org/jira/browse/HBASE-8172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615223#comment-13615223
]
Hudson commented on HBASE-8172:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/])
HBASE-8172 Fix ./bin/*.rb scripts or remove them (Revision 1461163)
Result = FAILURE
stack :
Files :
* /hbase/trunk/bin/get-active-master.rb
* /hbase/trunk/bin/region_mover.rb
* /hbase/trunk/bin/region_status.rb
> Fix ./bin/*.rb scripts or remove them
> -
>
> Key: HBASE-8172
> URL: https://issues.apache.org/jira/browse/HBASE-8172
> Project: HBase
> Issue Type: Task
> Components: scripts
>Reporter: stack
>Assignee: rajeshbabu
>Priority: Blocker
> Fix For: 0.95.0
>
> Attachments: HBASE-8172.patch
>
>
> I started to review the bin content to see what had rotted and what had not.
> The first script I tried doesn't say how to run it and when I did it did this:
> {code}
> durruti:hbase-0.97-SNAPSHOT stack$ ./bin/get-active-master.rb
> env: hbase-jruby: No such file or directory
> durruti:hbase-0.97-SNAPSHOT stack$ ./bin/hbase org.jruby.Main
> bin/get-active-master.rb
> NativeException: java.lang.IllegalArgumentException: Not a host:port pair:
> PBUF
> 172.21.3.100??
> __ensure__ at bin/get-active-master.rb:37
> (root) at bin/get-active-master.rb:34
> durruti:hbase-0.97-SNAPSHOT stack$ vi bin/
> {code}
> Check and fix all *.rb scripts or remove.
> Blocker on 0.95.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8147) Add Integration Test for "The HCat Scenario"
[ https://issues.apache.org/jira/browse/HBASE-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615222#comment-13615222 ] Hudson commented on HBASE-8147: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/]) HBASE-8147. Adds an integration test for HBASE-8140 (issue observed when commitJob was launching another job). (Revision 1461303) Result = FAILURE ddas : Files : * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestImportTsv.java > Add Integration Test for "The HCat Scenario" > > > Key: HBASE-8147 > URL: https://issues.apache.org/jira/browse/HBASE-8147 > Project: HBase > Issue Type: Sub-task > Components: mapreduce, test >Affects Versions: 0.95.0, 0.98.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: > 0001-HBASE-8147-Add-test-jobs-launched-based-on-other-job.patch > > > HBASE-8140 needs an integration test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8190) Add documentation of region merges to the book
[ https://issues.apache.org/jira/browse/HBASE-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615224#comment-13615224 ] Hudson commented on HBASE-8190: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/]) HBASE-8190 Add documentation of region merges to the book (Revision 1461396) Result = FAILURE zjushch : Files : * /hbase/trunk/src/docbkx/book.xml > Add documentation of region merges to the book > -- > > Key: HBASE-8190 > URL: https://issues.apache.org/jira/browse/HBASE-8190 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.98.0 > > Attachments: hbase-8190v1.patch, hbase-8190v2.patch, > hbase-8190v3.patch > > > Add documentation of region merges to the book -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615225#comment-13615225 ] Hudson commented on HBASE-8163: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/]) HBASE-8163 MemStoreChunkPool: An improvement for JAVA GC when using MSLAB (Revision 1461398) Result = FAILURE zjushch : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java > MemStoreChunkPool: An improvement for JAVA GC when using MSLAB > -- > > Key: HBASE-8163 > URL: https://issues.apache.org/jira/browse/HBASE-8163 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.95.0, 0.98.0 > > Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, > hbase-8163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch, > hbase-8163v5.patch, hbase-8163v6.patch > > > *Usage:* > Disable pool(default): configure "hbase.hregion.memstore.chunkpool.maxsize" > as 0 > Enable pool: configure "hbase.hregion.memstore.chunkpool.maxsize" as a > percentage of global memstore size (between 0.0 and 1.0, recommend to set it > as the gap between min and max sizes of memstore, e.g. 0.5) > *Background*： > When we use mslab,we will copy the keyvalue together in a structure called > *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. > *Problem*： > When one chunk is full, we will create a new chunk, and then the old chunk > will be reclaimed by JVM if no reference to it. > Mostly the chunk object would be promoted when doing Young GC, cause > increasing the cost of YGC > When a Chunk object has no reference? It should meet two following condition: > 1.Memstore which this chunk belongs to is flushed > 2.No scanner is opening on the memstore which this chunk belongs to > *Solution:* > 1.Create a chunk pool to manage the no-reference chunks, instead of being > reclaimed by JVM > 2.When a Chunk has no reference, put it back to the pool > 3.The pool has a max capacity, it will skip the chunks when achieve the max > size > 4.When we need new Chunk to store KeyValue, get it from the pool if exists, > else create new one by pool, so we could be able to reuse the old chunks > *Test results:* > Environment: > hbase-version:0.94 > -Xms4G -Xmx4G -Xmn2G > Row size=50 bytes, Value size=1024 bytes > 50 concurrent theads per client, insert 10,000,000 rows > Before: > Avg write request per second:12953 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 747 36.503 48 2.492 38.995 > After: > Avg write request per second:14025 > After testing, final result of jstat -gcutil : > YGC YGCT FGC FGCT GCT > 711 20.344 4 0.284 20.628 > *Improvement: YGC 40+%; WPS 5%+* > review board : > https://reviews.apache.org/r/10056/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8140) TableMapReduceUtils#addDependencyJar fails when nested inside another MR job
[ https://issues.apache.org/jira/browse/HBASE-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615226#comment-13615226 ] Hudson commented on HBASE-8140: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/]) HBASE-8147. Adds an integration test for HBASE-8140 (issue observed when commitJob was launching another job). (Revision 1461303) Result = FAILURE ddas : Files : * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestImportTsv.java > TableMapReduceUtils#addDependencyJar fails when nested inside another MR job > > > Key: HBASE-8140 > URL: https://issues.apache.org/jira/browse/HBASE-8140 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: 0001-HBASE-8140-addendum-add-test-category.patch, > 8140-port-jarfinder-0.94.patch, 8140-port-jarfinder-trunk.patch > > > TableMapReduceUtils#addDependencyJar is used when configuring a mapreduce job > to make sure dependencies of the job are shipped to the cluster. The code > depends on finding an actual jar file containing the necessary classes. This > is not always the case, for instance, when run at the end of another > mapreduce job. In that case, dependency jars have already been shipped to the > cluster and expanded in the parent job's run folder. Those dependencies are > there, just not available as jars. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8005) DemoClient.cpp can not be built correctly
[ https://issues.apache.org/jira/browse/HBASE-8005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615227#comment-13615227 ] Hudson commented on HBASE-8005: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/]) HBASE-8005 DemoClient.cpp can not be built correctly (Revision 1461177) Result = FAILURE stack : Files : * /hbase/trunk/hbase-examples/src/main/cpp/Makefile > DemoClient.cpp can not be built correctly > --- > > Key: HBASE-8005 > URL: https://issues.apache.org/jira/browse/HBASE-8005 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 0.94.5 >Reporter: Ling Kun > Fix For: 0.95.0 > > Attachments: 0001-solve-a-DemoClient.cpp-build-problem.patch, > HBASE-8005-v0-0.94.patch, HBASE-8005-v0.patch, makefile.patch > > > when compile the DemoClient.cpp within the latest HBase release package, it > seems that this file is out of date(plenty of APIs need a new std::map Text> & attributes parameter ). And I have also tried to replace the > DemoClient.cpp with the latest trunk version. However, after some > modification there are still link problems when compile the code( undefined > reference to `apache::thrift::transport::TSocket::TSocket(std::string, > int)', I have passed the -lthrift parameter to gcc). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8199) Eliminate exception for ExportSnapshot against the null table snapshot (with no data in)
[ https://issues.apache.org/jira/browse/HBASE-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615228#comment-13615228 ] Hudson commented on HBASE-8199: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/]) HBASE-8199 Eliminate exception for ExportSnapshot against the null table snapshot (Julian Zhou) (Revision 1461340) Result = FAILURE mbertozzi : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java > Eliminate exception for ExportSnapshot against the null table snapshot (with > no data in) > > > Key: HBASE-8199 > URL: https://issues.apache.org/jira/browse/HBASE-8199 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 0.95.0, 0.94.6, 0.94.7 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.95.0, 0.94.7 > > Attachments: HBASE-8199-0.94.patch, HBASE-8199-trunk.patch, > HBASE-8199-with-test.patch > > > In the test environment, when snapshoting the null table (with no data), > snapshot artifact can be generated successfully with snapshot meta data, > while there is no CF data under region dir. Then when exporting this null > snapshot via ExportSnapshot, "java.io.IOException: No input paths specified > in job" occurred, which affects the normal application flow. > But seems snapshot for null table also makes sense just with the meta > information (.snapshot/), although ".archive/" does not show up. Later > snapshot restore seems work from this "dummy" snapshot artifact with no CF > data. > Just eliminate exception for ExportSnapshot against the null table snapshot > to make application work through. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7908) Add jstack logs in hbase-common.sh when killing the server.
[ https://issues.apache.org/jira/browse/HBASE-7908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615229#comment-13615229 ] Hudson commented on HBASE-7908: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/]) HBASE-7908 Add jstack logs in hbase-common.sh when killing the server (Revision 1461167) Result = FAILURE stack : Files : * /hbase/trunk/bin/hbase-common.sh > Add jstack logs in hbase-common.sh when killing the server. > --- > > Key: HBASE-7908 > URL: https://issues.apache.org/jira/browse/HBASE-7908 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.95.0 > > Attachments: HBASE-7908-v0-trunk.patch, HBASE-7908-v1-trunk.patch > > > It might be a good idea to keep the jstack logs when killing -9 never ending > servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7597) TestHBaseFsck#testRegionShouldNotBeDeployed seems to be flaky
[ https://issues.apache.org/jira/browse/HBASE-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615230#comment-13615230 ] Hudson commented on HBASE-7597: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #466 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/466/]) HBASE-7597 TestHBaseFsck#testRegionShouldNotBeDeployed seems to be flaky, v2 (Revision 1461228) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > TestHBaseFsck#testRegionShouldNotBeDeployed seems to be flaky > - > > Key: HBASE-7597 > URL: https://issues.apache.org/jira/browse/HBASE-7597 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jimmy Xiang > Fix For: 0.95.0, 0.98.0 > > Attachments: trunk-7597.patch, trunk-7597_v2.patch > > > I ran the entire test suite many times and always failed on, at least, > testRegionShouldNotBeDeployed. > Results below. I will attached more result when current tests are done. > Failed tests: > testDeleteExpiredStoreFiles(org.apache.hadoop.hbase.regionserver.TestStore): > expected:<2> but was:<4> > > testAcquireTaskAtStartup(org.apache.hadoop.hbase.regionserver.TestSplitLogWorker): > Waiting timed out after [1 000] msec > testRegionShouldNotBeDeployed(org.apache.hadoop.hbase.util.TestHBaseFsck): > expected:<[SHOULD_NOT_BE_DEPLOYED]> but was:<[]> > > testPermissionsWatcher(org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8195) hbase.table.archive.directory reference should be removed from hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8195: -- Resolution: Fixed Fix Version/s: 0.98.0 0.95.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Integrated to 0.95 and trunk. > hbase.table.archive.directory reference should be removed from > hbase-default.xml > > > Key: HBASE-8195 > URL: https://issues.apache.org/jira/browse/HBASE-8195 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: HBASE-8195-v0-trunk.patch, HBASE-8195-v0-trunk.patch > > > hbase.table.archive.directory is not used anymore. Therefor, it should be > removed from hbase-default.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[
https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615287#comment-13615287
]
Ted Yu commented on HBASE-7878:
---
In HLogSplitter#splitLog():
{code}
for (FileStatus log : logfiles) {
...
in = getReader(fs, log, conf, skipErrors);
{code}
Lease is recovered in getReader().
We can issue lease recovery requests for all of logfiles - outside the above
for loop.
When it comes to getReader(), the wait time would be much shorter.
> recoverFileLease does not check return value of recoverLease
>
>
> Key: HBASE-7878
> URL: https://issues.apache.org/jira/browse/HBASE-7878
> Project: HBase
> Issue Type: Bug
> Components: util
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Eric Newton
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.95.0, 0.98.0
>
> Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2,
> 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2,
> 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt,
> 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch,
> 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt,
> 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt,
> 7878-trunk-v9.txt, 7878-trunk-v9.txt
>
>
> I think this is a problem, so I'm opening a ticket so an HBase person takes a
> look.
> Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease
> recovery for Accumulo after HBase's lease recovery. During testing, we
> experienced data loss. I found it is necessary to wait until recoverLease
> returns true to know that the file has been truly closed. In FSHDFSUtils,
> the return result of recoverLease is not checked. In the unit tests created
> to check lease recovery in HBASE-2645, the return result of recoverLease is
> always checked.
> I think FSHDFSUtils should be modified to check the return result, and wait
> until it returns true.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7904: -- Attachment: 7904-v9.txt > Make mapreduce jobs pass based on 2.0.4-alpha > - > > Key: HBASE-7904 > URL: https://issues.apache.org/jira/browse/HBASE-7904 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0 > > Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, > 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, > 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, > 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, > hbase-7904-v3.txt > > > 2.0.3-alpha has been released. > We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7904: -- Attachment: (was: 7904-v9.txt) > Make mapreduce jobs pass based on 2.0.4-alpha > - > > Key: HBASE-7904 > URL: https://issues.apache.org/jira/browse/HBASE-7904 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0 > > Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, > 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, > 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, > 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, > hbase-7904-v3.txt > > > 2.0.3-alpha has been released. > We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8176) Backport HBASE-5335 "Dynamic Schema Configurations"
[ https://issues.apache.org/jira/browse/HBASE-8176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615325#comment-13615325 ] Andrew Purtell commented on HBASE-8176: --- bq. On the other side, I do find it valuable for those who want to keep 0.94 in production for a long time, considering the big changes in 0.96 and risks of upgrading. There is value in having CompoundConfiguration in 0.94 because it allows for finer grained compaction policy. The patch looks reasonable. At first glance I have mostly some concerns about the shell changes. - It looks like they are backwards compatible. Is that true? - Could you add to the site documentation or the shell's help text somewhere an example of how to change compaction policy using CONFIG? What does this look like? How easy is it to type it in? > Backport HBASE-5335 "Dynamic Schema Configurations" > --- > > Key: HBASE-8176 > URL: https://issues.apache.org/jira/browse/HBASE-8176 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: clockfly >Assignee: clockfly >Priority: Minor > Fix For: 0.94.7 > > Attachments: hbase-8176.patch > > > With HBASE-5335, we can support per-table configuration and per-family > configurations. > We can use it to customize the compaction on table/family basis, customize > the flush, and etc.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8192) wrong logic in HRegion.bulkLoadHFiles(List)
[
https://issues.apache.org/jira/browse/HBASE-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615335#comment-13615335
]
Ted Yu commented on HBASE-8192:
---
This is first patch from Chenghao.
Can we add test case in a follow-on JIRA ?
> wrong logic in HRegion.bulkLoadHFiles(List)
> ---
>
> Key: HBASE-8192
> URL: https://issues.apache.org/jira/browse/HBASE-8192
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.1, 0.94.2, 0.94.3, 0.94.4, 0.94.5
>Reporter: Chenghao Jiang
> Labels: patch
> Fix For: 0.94.7
>
> Attachments: 8192.txt
>
>
> the wrong logic is here:
> when a ColumnFamily does not exist, it gets a null store object, then
> ioes.add(ioe); failures.add(p)
> but the code below, if (failures.size() != 0), it prints a warn log and
> return false, so it will never go into the code if (ioes.size() != 0) below,
> and IOException will not be thrown, then the client will keep retry forever.
> there is the same situation when doing store.assertBulkLoadHFileOk, if any
> WrongRegionException is caught and failures.add(p), then all the other
> IOException thrown by assertBulkLoadHFileOk will be ignored.
> so i think if (failures.size() != 0) {} should be dealt with after if
> (ioes.size() !=0) {}
> {code}
> for (Pair p : familyPaths) {
> byte[] familyName = p.getFirst();
> String path = p.getSecond();
> Store store = getStore(familyName);
> if (store == null) {
> IOException ioe = new DoNotRetryIOException(
> "No such column family " + Bytes.toStringBinary(familyName));
> ioes.add(ioe);
> failures.add(p);
> } else {
> try {
> store.assertBulkLoadHFileOk(new Path(path));
> } catch (WrongRegionException wre) {
> // recoverable (file doesn't fit in region)
> failures.add(p);
> } catch (IOException ioe) {
> // unrecoverable (hdfs problem)
> ioes.add(ioe);
> }
> }
> }
> // validation failed, bail out before doing anything permanent.
> if (failures.size() != 0) {
> StringBuilder list = new StringBuilder();
> for (Pair p : failures) {
> list.append("\n").append(Bytes.toString(p.getFirst())).append(" : ")
> .append(p.getSecond());
> }
> // problem when validating
> LOG.warn("There was a recoverable bulk load failure likely due to a" +
> " split. These (family, HFile) pairs were not loaded: " + list);
> return false;
> }
> // validation failed because of some sort of IO problem.
> if (ioes.size() != 0) {
> LOG.error("There were IO errors when checking if bulk load is ok. " +
> "throwing exception!");
> throw MultipleIOException.createIOException(ioes);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615341#comment-13615341 ] Nicolas Liochon commented on HBASE-7878: bq. I understand that in case false is returned from recoverLease, we would wait longer. Yes, we were not waiting before so we were wrong but fast. It seems that the recovery takes 5s bq. One remedy I can think of is to bundle lease recovery for several files together so that the extra wait can be amortized. It seems to be a good idea. I continue to investigate (there are other issues as well) but this one seems to be a clear quick win. Because even if the recovery is immediate, we now wait 1s (the first call returns false, so we wait 1s and then retry). Why is the leaseRecovery done by the regionserver vs. the master, btw? Moreover, I would also expect to have just a few WAL file opened on the hdfs side (the one for .meta., the current one, may be the previous one if we have just rolled?). We should call the lease recovery of these ones first may be? > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615357#comment-13615357 ] Ted Yu commented on HBASE-7878: --- bq. Why is the leaseRecovery done by the regionserver vs. the master, btw? Can you clarify this a bit ? There're two paths: The first is HMaster#finishInitialization() calling MasterFileSystem#splitLogAfterStartup() - optimization suggested above applies in this code path Another is through SplitLogWorker. bq. the first call returns false, so we wait 1s and then retry I guess we can shorten the 1 second wait in the false return case. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615364#comment-13615364 ] Nicolas Liochon commented on HBASE-7878: bq. Another is through SplitLogWorker. So when we have the distributed log on (the default) it's done by the region server. Would it make sense to have it done by the master even before distributing the tasks to the region servers: this way the NN would do the recovery while we're distributing the tasks. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615370#comment-13615370
]
Hadoop QA commented on HBASE-7904:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12575709/7904-v9.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5028//console
This message is automatically generated.
> Make mapreduce jobs pass based on 2.0.4-alpha
> -
>
> Key: HBASE-7904
> URL: https://issues.apache.org/jira/browse/HBASE-7904
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.95.0
>
> Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt,
> 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt,
> 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt,
> 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt,
> hbase-7904-v3.txt
>
>
> 2.0.3-alpha has been released.
> We should upgrade the dependency.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8195) hbase.table.archive.directory reference should be removed from hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615393#comment-13615393 ] Hudson commented on HBASE-8195: --- Integrated in HBase-TRUNK #3997 (See [https://builds.apache.org/job/HBase-TRUNK/3997/]) HBASE-8195 hbase.table.archive.directory reference should be removed from hbase-default.xml (Jean-Marc) (Revision 1461576) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml > hbase.table.archive.directory reference should be removed from > hbase-default.xml > > > Key: HBASE-8195 > URL: https://issues.apache.org/jira/browse/HBASE-8195 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: HBASE-8195-v0-trunk.patch, HBASE-8195-v0-trunk.patch > > > hbase.table.archive.directory is not used anymore. Therefor, it should be > removed from hbase-default.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions
[ https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5583: -- Attachment: HBASE-5583_new_4_WIP.patch This is still WIP in progress. But overall in better shape. Will add more tests with more split keys. May be some refactorings are needed still. > Master restart on create table with splitkeys does not recreate table with > all the splitkey regions > --- > > Key: HBASE-5583 > URL: https://issues.apache.org/jira/browse/HBASE-5583 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.95.0 > > Attachments: HBASE-5583_new_1.patch, HBASE-5583_new_2.patch, > HBASE-5583_new_4_WIP.patch > > > -> Create table using splitkeys > -> MAster goes down before all regions are added to meta > -> On master restart the table is again enabled but with less number of > regions than specified in splitkeys > Anyway client will get an exception if i had called sync create table. But > table exists or not check will say table exists. > Is this scenario to be handled by client only or can we have some mechanism > on the master side for this? Pls suggest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8195) hbase.table.archive.directory reference should be removed from hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615419#comment-13615419 ] Hudson commented on HBASE-8195: --- Integrated in hbase-0.95 #110 (See [https://builds.apache.org/job/hbase-0.95/110/]) HBASE-8195 hbase.table.archive.directory reference should be removed from hbase-default.xml (Jean-Marc) (Revision 1461578) Result = FAILURE tedyu : Files : * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml > hbase.table.archive.directory reference should be removed from > hbase-default.xml > > > Key: HBASE-8195 > URL: https://issues.apache.org/jira/browse/HBASE-8195 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: HBASE-8195-v0-trunk.patch, HBASE-8195-v0-trunk.patch > > > hbase.table.archive.directory is not used anymore. Therefor, it should be > removed from hbase-default.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[
https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615458#comment-13615458
]
Ted Yu commented on HBASE-7878:
---
Interesting idea.
In SplitLogManager#splitLogDistributed(), we can issue lease recovery request
after obtaining log file list:
{code}
FileStatus[] logfiles = getFileList(logDirs, filter);
{code}
One intricacy I can think of is that such lease recovery request is issued from
master. SplitLogWorker may encounter the following exception:
{code}
Caused by: org.apache.hadoop.hdfs.protocol.RecoveryInProgressException: Failed
to close file /user/jenkins/hbase/TestHLog/hlogdir/hlog.1364388129638. Lease
recovery is in progress. Try again later.
{code}
Note: RecoveryInProgressException isn't in hadoop 1.0
> recoverFileLease does not check return value of recoverLease
>
>
> Key: HBASE-7878
> URL: https://issues.apache.org/jira/browse/HBASE-7878
> Project: HBase
> Issue Type: Bug
> Components: util
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Eric Newton
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.95.0, 0.98.0
>
> Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2,
> 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2,
> 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt,
> 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch,
> 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt,
> 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt,
> 7878-trunk-v9.txt, 7878-trunk-v9.txt
>
>
> I think this is a problem, so I'm opening a ticket so an HBase person takes a
> look.
> Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease
> recovery for Accumulo after HBase's lease recovery. During testing, we
> experienced data loss. I found it is necessary to wait until recoverLease
> returns true to know that the file has been truly closed. In FSHDFSUtils,
> the return result of recoverLease is not checked. In the unit tests created
> to check lease recovery in HBASE-2645, the return result of recoverLease is
> always checked.
> I think FSHDFSUtils should be modified to check the return result, and wait
> until it returns true.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8144) Limit number of attempts to assign a region
[ https://issues.apache.org/jira/browse/HBASE-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8144: --- Status: Open (was: Patch Available) > Limit number of attempts to assign a region > --- > > Key: HBASE-8144 > URL: https://issues.apache.org/jira/browse/HBASE-8144 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: trunk-8144.patch, trunk-8144_v2.patch, > trunk-8144_v3.1.patch, trunk-8144_v3.patch > > > In sending a region open request to a region server, we make sure we try at > most some configured times. However, once the request is accepted by the > region server, the region could go through this transition forever: > failed_open (in ZK) => closed => opening => failed_open (in ZK), assuming no > RPC/network issue. > It will be good to break the loop and limit the number of tries and move the > region to failed_open state (will be introduced in HBASE-8137) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8144) Limit number of attempts to assign a region
[ https://issues.apache.org/jira/browse/HBASE-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8144: --- Attachment: trunk-8144_v3.2.patch Minor change: increased the timeout for testMoveRegion. > Limit number of attempts to assign a region > --- > > Key: HBASE-8144 > URL: https://issues.apache.org/jira/browse/HBASE-8144 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: trunk-8144.patch, trunk-8144_v2.patch, > trunk-8144_v3.1.patch, trunk-8144_v3.2.patch, trunk-8144_v3.patch > > > In sending a region open request to a region server, we make sure we try at > most some configured times. However, once the request is accepted by the > region server, the region could go through this transition forever: > failed_open (in ZK) => closed => opening => failed_open (in ZK), assuming no > RPC/network issue. > It will be good to break the loop and limit the number of tries and move the > region to failed_open state (will be introduced in HBASE-8137) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8144) Limit number of attempts to assign a region
[ https://issues.apache.org/jira/browse/HBASE-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8144: --- Status: Patch Available (was: Open) > Limit number of attempts to assign a region > --- > > Key: HBASE-8144 > URL: https://issues.apache.org/jira/browse/HBASE-8144 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: trunk-8144.patch, trunk-8144_v2.patch, > trunk-8144_v3.1.patch, trunk-8144_v3.2.patch, trunk-8144_v3.patch > > > In sending a region open request to a region server, we make sure we try at > most some configured times. However, once the request is accepted by the > region server, the region could go through this transition forever: > failed_open (in ZK) => closed => opening => failed_open (in ZK), assuming no > RPC/network issue. > It will be good to break the loop and limit the number of tries and move the > region to failed_open state (will be introduced in HBASE-8137) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7835) Implementation of the log splitting without creating intermediate files
[ https://issues.apache.org/jira/browse/HBASE-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615476#comment-13615476 ] Ted Yu commented on HBASE-7835: --- @Jeffrey: Can you rebase your patch ? Thanks > Implementation of the log splitting without creating intermediate files > > > Key: HBASE-7835 > URL: https://issues.apache.org/jira/browse/HBASE-7835 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.95.0 > > Attachments: hbase-7006_12.patch, hbase-7006_1.patch, > hbase-7006_2.patch, hbase-7006_3.patch, hbase-7006_3.patch, > hbase-7006_6.patch, hbase-7006_8.patch > > > The sub task is to check in a separate patch for major implementation for the > proposal. > More checkins will include a new "replay" command and more metrics support. > Thanks, > -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7415) [snapshots] Add task information to snapshot operation
[ https://issues.apache.org/jira/browse/HBASE-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7415: --- Attachment: HBASE-7415-v3.patch attached v3, is just a rebase against current trunk > [snapshots] Add task information to snapshot operation > -- > > Key: HBASE-7415 > URL: https://issues.apache.org/jira/browse/HBASE-7415 > Project: HBase > Issue Type: New Feature > Components: Client, master, regionserver, snapshots, Zookeeper >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.95.0 > > Attachments: hbase-7415-v0.patch, hbase-7415-v1.patch, > HBASE-7415-v1-rebase.patch, HBASE-7415-v2.patch, HBASE-7415-v3.patch, > HBase_Snapshot_Task_UI.png > > > Snapshot operations should have some sort of progresss information available > via the WebUI so admins can track progress of operations. This should go a > long way to enable 'good' admins to not hose their clusters by running > concurrent snapshot operations (e.g. rename while a clone is in progress). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (HBASE-7835) Implementation of the log splitting without creating intermediate files
Jeffrey, Thanks for answering. That explains it. On Wed, Mar 27, 2013 at 9:47 AM, Ted Yu (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/HBASE-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615476#comment-13615476] > > Ted Yu commented on HBASE-7835: > --- > > @Jeffrey: > Can you rebase your patch ? > > Thanks > > > Implementation of the log splitting without creating intermediate files > > > > > > Key: HBASE-7835 > > URL: https://issues.apache.org/jira/browse/HBASE-7835 > > Project: HBase > > Issue Type: Sub-task > > Components: regionserver > >Reporter: Jeffrey Zhong > >Assignee: Jeffrey Zhong > > Fix For: 0.95.0 > > > > Attachments: hbase-7006_12.patch, hbase-7006_1.patch, > hbase-7006_2.patch, hbase-7006_3.patch, hbase-7006_3.patch, > hbase-7006_6.patch, hbase-7006_8.patch > > > > > > The sub task is to check in a separate patch for major implementation > for the proposal. > > More checkins will include a new "replay" command and more metrics > support. > > Thanks, > > -Jeffrey > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira >
[jira] [Commented] (HBASE-7835) Implementation of the log splitting without creating intermediate files
[ https://issues.apache.org/jira/browse/HBASE-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615495#comment-13615495 ] Himanshu Vashishtha commented on HBASE-7835: Jeffrey, Thanks for answering. That explains it. > Implementation of the log splitting without creating intermediate files > > > Key: HBASE-7835 > URL: https://issues.apache.org/jira/browse/HBASE-7835 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.95.0 > > Attachments: hbase-7006_12.patch, hbase-7006_1.patch, > hbase-7006_2.patch, hbase-7006_3.patch, hbase-7006_3.patch, > hbase-7006_6.patch, hbase-7006_8.patch > > > The sub task is to check in a separate patch for major implementation for the > proposal. > More checkins will include a new "replay" command and more metrics support. > Thanks, > -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615497#comment-13615497 ] Ted Yu commented on HBASE-7878: --- Talking with Jeffrey, SplitLogManager can provide hint to SplitLogWorker about the average number of WAL files to be picked up by each worker. SplitLogWorker would try to pick up WAL files according to this hint so that SplitLogWorker can issue lease recovery request up front. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615505#comment-13615505 ] Ted Yu commented on HBASE-8164: --- Will integrate patch v4 today if there is no more review comment. > TestTableLockManager fails intermittently in trunk builds > - > > Key: HBASE-8164 > URL: https://issues.apache.org/jira/browse/HBASE-8164 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.0 > > Attachments: 8164-v2.txt, 8164-v3.txt, 8164-v4.txt > > > In build #3979: > testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test > timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-5945: --- Priority: Major (was: Blocker) Downgrading priority since I don't think this is a relevant issue anymore. Please discuss if you feel otherwise. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.95.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-5945: --- Status: Open (was: Patch Available) This patch is really stale. Canceling. > Reduce buffer copies in IPC server response path > > > Key: HBASE-5945 > URL: https://issues.apache.org/jira/browse/HBASE-5945 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 0.96.0 >Reporter: Todd Lipcon >Assignee: stack > Fix For: 0.95.0 > > Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, > 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, > hbase-5495.txt > > > The new PB code is sloppy with buffers and makes several needless copies. > This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8192) wrong logic in HRegion.bulkLoadHFiles(List)
[
https://issues.apache.org/jira/browse/HBASE-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615533#comment-13615533
]
Jonathan Hsieh commented on HBASE-8192:
---
Its good to pick up good habits on the first patch. Since this is a logic bug
so I'd feel a little more strongly about unit testing.
[~cythelia] let us know if you plan on adding it and want help or if I should
add the test.
> wrong logic in HRegion.bulkLoadHFiles(List)
> ---
>
> Key: HBASE-8192
> URL: https://issues.apache.org/jira/browse/HBASE-8192
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.1, 0.94.2, 0.94.3, 0.94.4, 0.94.5
>Reporter: Chenghao Jiang
> Labels: patch
> Fix For: 0.94.7
>
> Attachments: 8192.txt
>
>
> the wrong logic is here:
> when a ColumnFamily does not exist, it gets a null store object, then
> ioes.add(ioe); failures.add(p)
> but the code below, if (failures.size() != 0), it prints a warn log and
> return false, so it will never go into the code if (ioes.size() != 0) below,
> and IOException will not be thrown, then the client will keep retry forever.
> there is the same situation when doing store.assertBulkLoadHFileOk, if any
> WrongRegionException is caught and failures.add(p), then all the other
> IOException thrown by assertBulkLoadHFileOk will be ignored.
> so i think if (failures.size() != 0) {} should be dealt with after if
> (ioes.size() !=0) {}
> {code}
> for (Pair p : familyPaths) {
> byte[] familyName = p.getFirst();
> String path = p.getSecond();
> Store store = getStore(familyName);
> if (store == null) {
> IOException ioe = new DoNotRetryIOException(
> "No such column family " + Bytes.toStringBinary(familyName));
> ioes.add(ioe);
> failures.add(p);
> } else {
> try {
> store.assertBulkLoadHFileOk(new Path(path));
> } catch (WrongRegionException wre) {
> // recoverable (file doesn't fit in region)
> failures.add(p);
> } catch (IOException ioe) {
> // unrecoverable (hdfs problem)
> ioes.add(ioe);
> }
> }
> }
> // validation failed, bail out before doing anything permanent.
> if (failures.size() != 0) {
> StringBuilder list = new StringBuilder();
> for (Pair p : failures) {
> list.append("\n").append(Bytes.toString(p.getFirst())).append(" : ")
> .append(p.getSecond());
> }
> // problem when validating
> LOG.warn("There was a recoverable bulk load failure likely due to a" +
> " split. These (family, HFile) pairs were not loaded: " + list);
> return false;
> }
> // validation failed because of some sort of IO problem.
> if (ioes.size() != 0) {
> LOG.error("There were IO errors when checking if bulk load is ok. " +
> "throwing exception!");
> throw MultipleIOException.createIOException(ioes);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8192) wrong logic in HRegion.bulkLoadHFiles(List)
[
https://issues.apache.org/jira/browse/HBASE-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615533#comment-13615533
]
Jonathan Hsieh edited comment on HBASE-8192 at 3/27/13 5:35 PM:
Its good to pick up good habits on the first patch. Since this is a logic bug,
I feel a little more strongly about unit testing.
[~cythelia] let us know if you plan on adding it and want help or if I should
add the test.
was (Author: jmhsieh):
Its good to pick up good habits on the first patch. Since this is a logic
bug so I'd feel a little more strongly about unit testing.
[~cythelia] let us know if you plan on adding it and want help or if I should
add the test.
> wrong logic in HRegion.bulkLoadHFiles(List)
> ---
>
> Key: HBASE-8192
> URL: https://issues.apache.org/jira/browse/HBASE-8192
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.1, 0.94.2, 0.94.3, 0.94.4, 0.94.5
>Reporter: Chenghao Jiang
> Labels: patch
> Fix For: 0.94.7
>
> Attachments: 8192.txt
>
>
> the wrong logic is here:
> when a ColumnFamily does not exist, it gets a null store object, then
> ioes.add(ioe); failures.add(p)
> but the code below, if (failures.size() != 0), it prints a warn log and
> return false, so it will never go into the code if (ioes.size() != 0) below,
> and IOException will not be thrown, then the client will keep retry forever.
> there is the same situation when doing store.assertBulkLoadHFileOk, if any
> WrongRegionException is caught and failures.add(p), then all the other
> IOException thrown by assertBulkLoadHFileOk will be ignored.
> so i think if (failures.size() != 0) {} should be dealt with after if
> (ioes.size() !=0) {}
> {code}
> for (Pair p : familyPaths) {
> byte[] familyName = p.getFirst();
> String path = p.getSecond();
> Store store = getStore(familyName);
> if (store == null) {
> IOException ioe = new DoNotRetryIOException(
> "No such column family " + Bytes.toStringBinary(familyName));
> ioes.add(ioe);
> failures.add(p);
> } else {
> try {
> store.assertBulkLoadHFileOk(new Path(path));
> } catch (WrongRegionException wre) {
> // recoverable (file doesn't fit in region)
> failures.add(p);
> } catch (IOException ioe) {
> // unrecoverable (hdfs problem)
> ioes.add(ioe);
> }
> }
> }
> // validation failed, bail out before doing anything permanent.
> if (failures.size() != 0) {
> StringBuilder list = new StringBuilder();
> for (Pair p : failures) {
> list.append("\n").append(Bytes.toString(p.getFirst())).append(" : ")
> .append(p.getSecond());
> }
> // problem when validating
> LOG.warn("There was a recoverable bulk load failure likely due to a" +
> " split. These (family, HFile) pairs were not loaded: " + list);
> return false;
> }
> // validation failed because of some sort of IO problem.
> if (ioes.size() != 0) {
> LOG.error("There were IO errors when checking if bulk load is ok. " +
> "throwing exception!");
> throw MultipleIOException.createIOException(ioes);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8202) MultiTableOutputFormat should support hbase.mapred.output.quorum
Ted Yu created HBASE-8202:
-
Summary: MultiTableOutputFormat should support
hbase.mapred.output.quorum
Key: HBASE-8202
URL: https://issues.apache.org/jira/browse/HBASE-8202
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
This was brought up by David Koch in thread 'hbase.mapred.output.quorum ignored
in Mapper job with HDFS source and HBase sink' where he wanted to import a file
on HDFS from one cluster A (source) into HBase
tables on a different cluster B (destination) using a Mapper job with an
HBase sink.
Here is my analysis:
MultiTableOutputFormat doesn't extend TableOutputFormat:
{code}
public class MultiTableOutputFormat extends
OutputFormat {
{code}
Relevant configuration w.r.t. output quorum is setup in
TableOutputFormat#setConf():
{code}
public void setConf(Configuration otherConf) {
this.conf = HBaseConfiguration.create(otherConf);
String tableName = this.conf.get(OUTPUT_TABLE);
if(tableName == null || tableName.length() <= 0) {
throw new IllegalArgumentException("Must specify table name");
}
String address = this.conf.get(QUORUM_ADDRESS);
int zkClientPort = conf.getInt(QUORUM_PORT, 0);
String serverClass = this.conf.get(REGION_SERVER_CLASS);
String serverImpl = this.conf.get(REGION_SERVER_IMPL);
try {
if (address != null) {
ZKUtil.applyClusterKeyToConf(this.conf, address);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8197) Cannot distinguish between HBase default config value and client overridden config value
[
https://issues.apache.org/jira/browse/HBASE-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615543#comment-13615543
]
Jonathan Hsieh commented on HBASE-8197:
---
Is this sufficient? (notice the "Loaded from ..." comment).
{code}
[jon@a1222 ~]$ hbase org.apache.hadoop.hbase.HBaseConfiguration
hbase.online.schema.update.enablefalse
hbase.master.port57000
...
{code}
> Cannot distinguish between HBase default config value and client overridden
> config value
>
>
> Key: HBASE-8197
> URL: https://issues.apache.org/jira/browse/HBASE-8197
> Project: HBase
> Issue Type: Bug
> Components: Usability
>Affects Versions: 0.94.6
>Reporter: James Taylor
>Priority: Minor
>
> By HBase setting default values in hbase-default.xml, it makes it difficult
> for a platform to know if a value has been overridden by the client. For
> example, Phoenix has a default value it would like to use for
> hbase.client.scanner.caching. I'd like to be able to call config.setIfUnset
> option to use our default value if a client hasn't already provided their
> own. Since HBase sets this in hbase-default.xml to 1, I can't differentiate
> between the client have a value in their hbase-sites.xml as 1 versus the
> default value we're getting from HBase.
> Wouldn't it be better if HBase had constants defined in a Java interface that
> would be used by HBase as the default value if none other is provided, like
> this:
> config.getInt("hbase.client.scanner.caching",
> HBaseDefault.CLIENT_SCANNER_CACHING);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615547#comment-13615547 ] Nicolas Liochon commented on HBASE-7878: I don't see why it would be better than if the call is done by the master? In any case, if we have less worker than WAL files, we should start by the 'oldest' WAL files: these ones are likely to be closed, so the lease recovery should be immediate (for these ones, may be recoverLease will return true at the first call?). > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8197) Cannot distinguish between HBase default config value and client overridden config value
[
https://issues.apache.org/jira/browse/HBASE-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615549#comment-13615549
]
Jean-Marc Spaggiari commented on HBASE-8197:
That will be a nice add!
> Cannot distinguish between HBase default config value and client overridden
> config value
>
>
> Key: HBASE-8197
> URL: https://issues.apache.org/jira/browse/HBASE-8197
> Project: HBase
> Issue Type: Bug
> Components: Usability
>Affects Versions: 0.94.6
>Reporter: James Taylor
>Priority: Minor
>
> By HBase setting default values in hbase-default.xml, it makes it difficult
> for a platform to know if a value has been overridden by the client. For
> example, Phoenix has a default value it would like to use for
> hbase.client.scanner.caching. I'd like to be able to call config.setIfUnset
> option to use our default value if a client hasn't already provided their
> own. Since HBase sets this in hbase-default.xml to 1, I can't differentiate
> between the client have a value in their hbase-sites.xml as 1 versus the
> default value we're getting from HBase.
> Wouldn't it be better if HBase had constants defined in a Java interface that
> would be used by HBase as the default value if none other is provided, like
> this:
> config.getInt("hbase.client.scanner.caching",
> HBaseDefault.CLIENT_SCANNER_CACHING);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8197) Cannot distinguish between HBase default config value and client overridden config value
[
https://issues.apache.org/jira/browse/HBASE-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615552#comment-13615552
]
Jonathan Hsieh commented on HBASE-8197:
---
If you have a custom hbase-site.xml for your client in your classpath, it
should show those values and the fact it came from there when you run the
HBaseConfiguration command.
Doesn't look like it is in the ref guide -- we should add a little section
about it, filing a follow on.
> Cannot distinguish between HBase default config value and client overridden
> config value
>
>
> Key: HBASE-8197
> URL: https://issues.apache.org/jira/browse/HBASE-8197
> Project: HBase
> Issue Type: Bug
> Components: Usability
>Affects Versions: 0.94.6
>Reporter: James Taylor
>Priority: Minor
>
> By HBase setting default values in hbase-default.xml, it makes it difficult
> for a platform to know if a value has been overridden by the client. For
> example, Phoenix has a default value it would like to use for
> hbase.client.scanner.caching. I'd like to be able to call config.setIfUnset
> option to use our default value if a client hasn't already provided their
> own. Since HBase sets this in hbase-default.xml to 1, I can't differentiate
> between the client have a value in their hbase-sites.xml as 1 versus the
> default value we're getting from HBase.
> Wouldn't it be better if HBase had constants defined in a Java interface that
> would be used by HBase as the default value if none other is provided, like
> this:
> config.getInt("hbase.client.scanner.caching",
> HBaseDefault.CLIENT_SCANNER_CACHING);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7415) [snapshots] Add task information to snapshot operation
[
https://issues.apache.org/jira/browse/HBASE-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615553#comment-13615553
]
Hadoop QA commented on HBASE-7415:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12575727/HBASE-7415-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5030//console
This message is automatically generated.
> [snapshots] Add task information to snapshot operation
> --
>
> Key: HBASE-7415
> URL: https://issues.apache.org/jira/browse/HBASE-7415
> Project: HBase
> Issue Type: New Feature
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.95.0
>
> Attachments: hbase-7415-v0.patch, hbase-7415-v1.patch,
> HBASE-7415-v1-rebase.patch, HBASE-7415-v2.patch, HBASE-7415-v3.patch,
> HBase_Snapshot_Task_UI.png
>
>
> Snapshot operations should have some sort of progresss information available
> via the WebUI so admins can track progress of operations. This should go a
> long way to enable 'good' admins to not hose their clusters by running
> concurrent snapshot operations (e.g. rename while a clone is in progress).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615559#comment-13615559 ] Ted Yu commented on HBASE-7878: --- bq. for these ones, may be recoverLease will return true at the first call? I think so. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8144) Limit number of attempts to assign a region
[
https://issues.apache.org/jira/browse/HBASE-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615561#comment-13615561
]
Hadoop QA commented on HBASE-8144:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12575726/trunk-8144_v3.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5029//console
This message is automatically generated.
> Limit number of attempts to assign a region
> ---
>
> Key: HBASE-8144
> URL: https://issues.apache.org/jira/browse/HBASE-8144
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.95.0, 0.98.0
>
> Attachments: trunk-8144.patch, trunk-8144_v2.patch,
> trunk-8144_v3.1.patch, trunk-8144_v3.2.patch, trunk-8144_v3.patch
>
>
> In sending a region open request to a region server, we make sure we try at
> most some configured times. However, once the request is accepted by the
> region server, the region could go through this transition forever:
> failed_open (in ZK) => closed => opening => failed_open (in ZK), assuming no
> RPC/network issue.
> It will be good to break the loop and limit the number of tries and move the
> region to failed_open state (will be introduced in HBASE-8137)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8202) MultiTableOutputFormat should support hbase.mapred.output.quorum
[
https://issues.apache.org/jira/browse/HBASE-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-8202:
--
Component/s: mapreduce
> MultiTableOutputFormat should support hbase.mapred.output.quorum
>
>
> Key: HBASE-8202
> URL: https://issues.apache.org/jira/browse/HBASE-8202
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Ted Yu
>
> This was brought up by David Koch in thread 'hbase.mapred.output.quorum
> ignored in Mapper job with HDFS source and HBase sink' where he wanted to
> import a file on HDFS from one cluster A (source) into HBase
> tables on a different cluster B (destination) using a Mapper job with an
> HBase sink.
> Here is my analysis:
> MultiTableOutputFormat doesn't extend TableOutputFormat:
> {code}
> public class MultiTableOutputFormat extends
> OutputFormat {
> {code}
> Relevant configuration w.r.t. output quorum is setup in
> TableOutputFormat#setConf():
> {code}
> public void setConf(Configuration otherConf) {
> this.conf = HBaseConfiguration.create(otherConf);
> String tableName = this.conf.get(OUTPUT_TABLE);
> if(tableName == null || tableName.length() <= 0) {
> throw new IllegalArgumentException("Must specify table name");
> }
> String address = this.conf.get(QUORUM_ADDRESS);
> int zkClientPort = conf.getInt(QUORUM_PORT, 0);
> String serverClass = this.conf.get(REGION_SERVER_CLASS);
> String serverImpl = this.conf.get(REGION_SERVER_IMPL);
> try {
> if (address != null) {
> ZKUtil.applyClusterKeyToConf(this.conf, address);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7893) hbase.hregion.majorcompaction not taking effect.
[ https://issues.apache.org/jira/browse/HBASE-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615579#comment-13615579 ] Sergey Shelukhin commented on HBASE-7893: - +1 to close... > hbase.hregion.majorcompaction not taking effect. > > > Key: HBASE-7893 > URL: https://issues.apache.org/jira/browse/HBASE-7893 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 0.92.2 > Environment: Linux 2.6.21.7-2.fc8xen-ec2-v1.0 #1 SMP Tue > Sep 1 10:25:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux > Fedora release 8 (Werewolf) >Reporter: Pranav Sharma > Attachments: hbase-hadoop-master-ip-10-118-138-149.log, > hbase-hadoop-regionserver-ip-10-118-187-126.log > > > For disabling auto major compaction, I configured > hbase.hregion.majorcompaction = 0 in the config file and restarted the > cluster. In spite of this, major compaction continues to run everyday. Here > is the config I set: > > hbase.hregion.majorcompaction > 0 > > What other way can I disable auto major compaction? I expected this config to > work. What am I doing wrong here? > Please advice. Thanks a lot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8152) Avoid creating empty reference file when splitkey is outside the range of a store file
[ https://issues.apache.org/jira/browse/HBASE-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615581#comment-13615581 ] Sergey Shelukhin commented on HBASE-8152: - Can you please provide a trunk patch? > Avoid creating empty reference file when splitkey is outside the range of a > store file > -- > > Key: HBASE-8152 > URL: https://issues.apache.org/jira/browse/HBASE-8152 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, HFile >Affects Versions: 0.94.5 >Reporter: clockfly >Assignee: clockfly >Priority: Trivial > Fix For: 0.94.7 > > Attachments: hbase-8152.0.94patch.v2, hbase-8152.patch0.94 > > > When splitting a store file, if the split key is before the first key, or > greater than the last key, then only one reference file should be created. > Currently, two reference file will be created. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8152) Avoid creating empty reference file when splitkey is outside the range of a store file
[
https://issues.apache.org/jira/browse/HBASE-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615594#comment-13615594
]
Sergey Shelukhin commented on HBASE-8152:
-
{quote}
- byte[] SPLITKEY = new byte[] { (LAST_CHAR-FIRST_CHAR)/2, FIRST_CHAR};
+ byte[] SPLITKEY = new byte[] { (LAST_CHAR + FIRST_CHAR)/2, FIRST_CHAR};
{quote}
How did this work before?
{quote}
- assertTrue(count == 0);
+
+}
+else {
{quote}
Nit: whitespace, linebreaks.
bq. + byte[] splitKey = KeyValue.createLastOnRow(splitRow).getKey();
Nit^2: no need to create array copy (getKey), could just create KV like here,
and then use getBuffer/getKeyOffset/... in compares.
Both of the compares are exclusive. Split key can only be in one file, right?
So one of them should be inclusive unless I'm missing something.
> Avoid creating empty reference file when splitkey is outside the range of a
> store file
> --
>
> Key: HBASE-8152
> URL: https://issues.apache.org/jira/browse/HBASE-8152
> Project: HBase
> Issue Type: Improvement
> Components: Filesystem Integration, HFile
>Affects Versions: 0.94.5
>Reporter: clockfly
>Assignee: clockfly
>Priority: Trivial
> Fix For: 0.94.7
>
> Attachments: hbase-8152.0.94patch.v2, hbase-8152.patch0.94
>
>
> When splitting a store file, if the split key is before the first key, or
> greater than the last key, then only one reference file should be created.
> Currently, two reference file will be created.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8202) MultiTableOutputFormat should support writing to another HBase cluster
[
https://issues.apache.org/jira/browse/HBASE-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-8202:
--
Summary: MultiTableOutputFormat should support writing to another HBase
cluster (was: MultiTableOutputFormat should support hbase.mapred.output.quorum)
> MultiTableOutputFormat should support writing to another HBase cluster
> --
>
> Key: HBASE-8202
> URL: https://issues.apache.org/jira/browse/HBASE-8202
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Ted Yu
>
> This was brought up by David Koch in thread 'hbase.mapred.output.quorum
> ignored in Mapper job with HDFS source and HBase sink' where he wanted to
> import a file on HDFS from one cluster A (source) into HBase
> tables on a different cluster B (destination) using a Mapper job with an
> HBase sink.
> Here is my analysis:
> MultiTableOutputFormat doesn't extend TableOutputFormat:
> {code}
> public class MultiTableOutputFormat extends
> OutputFormat {
> {code}
> Relevant configuration w.r.t. output quorum is setup in
> TableOutputFormat#setConf():
> {code}
> public void setConf(Configuration otherConf) {
> this.conf = HBaseConfiguration.create(otherConf);
> String tableName = this.conf.get(OUTPUT_TABLE);
> if(tableName == null || tableName.length() <= 0) {
> throw new IllegalArgumentException("Must specify table name");
> }
> String address = this.conf.get(QUORUM_ADDRESS);
> int zkClientPort = conf.getInt(QUORUM_PORT, 0);
> String serverClass = this.conf.get(REGION_SERVER_CLASS);
> String serverImpl = this.conf.get(REGION_SERVER_IMPL);
> try {
> if (address != null) {
> ZKUtil.applyClusterKeyToConf(this.conf, address);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8200) Integration Test classes are not part of the default HBase classpath. We should fix this or note workaround in documentation
[ https://issues.apache.org/jira/browse/HBASE-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615596#comment-13615596 ] Sergey Shelukhin commented on HBASE-8200: - Should it work from source checkout? I thought the prescribed way for that is to use mvn. > Integration Test classes are not part of the default HBase classpath. We > should fix this or note workaround in documentation > > > Key: HBASE-8200 > URL: https://issues.apache.org/jira/browse/HBASE-8200 > Project: HBase > Issue Type: Bug > Components: test > Environment: ubuntu >Reporter: Aleksandr Shulman > > Expected behavior: > Following the instructions here: > http://hbase.apache.org/book/hbase.tests.html#integration.tests > After pulling down the latest bits from trunk and running, > > mvn clean compile test-compile package install assembly:assembly -DskipTests > I should be able to run integration tests with the following command: > > bin/hbase --config /path/to/configs > > org.apache.hadoop.hbase.IntegrationTestsDriver > According to the documentation, this should kick off the tests. > Observed behavior: > Upon executing > > bin/hbase --config /path/to/configs > > org.apache.hadoop.hbase.IntegrationTestsDriver > I see the following: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/IntegrationTestsDriver > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hbase.IntegrationTestsDriver > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > I can fix it by adding the following line to the hbase-env.sh file > corresponding to the configs: > export HBASE_CLASSPATH=/path/to/project/hbase/hbase-it/target/test-classes/ > This produces the correct output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8197) Cannot distinguish between HBase default config value and client overridden config value
[
https://issues.apache.org/jira/browse/HBASE-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615599#comment-13615599
]
Jonathan Hsieh commented on HBASE-8197:
---
[~jmspaggi] from the reaction, it sounds like this may be something only in
hadoop2 (which I run/test on top of). If you run this on hbase+hadoop1 do you
get the "loaded from" info?
> Cannot distinguish between HBase default config value and client overridden
> config value
>
>
> Key: HBASE-8197
> URL: https://issues.apache.org/jira/browse/HBASE-8197
> Project: HBase
> Issue Type: Bug
> Components: Usability
>Affects Versions: 0.94.6
>Reporter: James Taylor
>Priority: Minor
>
> By HBase setting default values in hbase-default.xml, it makes it difficult
> for a platform to know if a value has been overridden by the client. For
> example, Phoenix has a default value it would like to use for
> hbase.client.scanner.caching. I'd like to be able to call config.setIfUnset
> option to use our default value if a client hasn't already provided their
> own. Since HBase sets this in hbase-default.xml to 1, I can't differentiate
> between the client have a value in their hbase-sites.xml as 1 versus the
> default value we're getting from HBase.
> Wouldn't it be better if HBase had constants defined in a Java interface that
> would be used by HBase as the default value if none other is provided, like
> this:
> config.getInt("hbase.client.scanner.caching",
> HBaseDefault.CLIENT_SCANNER_CACHING);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8197) Cannot distinguish between HBase default config value and client overridden config value
[
https://issues.apache.org/jira/browse/HBASE-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615599#comment-13615599
]
Jonathan Hsieh edited comment on HBASE-8197 at 3/27/13 6:28 PM:
This is in my hbase testing clusters -- hbase on top of hadoop2. [~jmspaggi]
from the reaction, it sounds like this may be something only in hadoop2 (which
I run/test on top of). If you run this on hbase+hadoop1 do you get the "loaded
from" info?
was (Author: jmhsieh):
[~jmspaggi] from the reaction, it sounds like this may be something only in
hadoop2 (which I run/test on top of). If you run this on hbase+hadoop1 do you
get the "loaded from" info?
> Cannot distinguish between HBase default config value and client overridden
> config value
>
>
> Key: HBASE-8197
> URL: https://issues.apache.org/jira/browse/HBASE-8197
> Project: HBase
> Issue Type: Bug
> Components: Usability
>Affects Versions: 0.94.6
>Reporter: James Taylor
>Priority: Minor
>
> By HBase setting default values in hbase-default.xml, it makes it difficult
> for a platform to know if a value has been overridden by the client. For
> example, Phoenix has a default value it would like to use for
> hbase.client.scanner.caching. I'd like to be able to call config.setIfUnset
> option to use our default value if a client hasn't already provided their
> own. Since HBase sets this in hbase-default.xml to 1, I can't differentiate
> between the client have a value in their hbase-sites.xml as 1 versus the
> default value we're getting from HBase.
> Wouldn't it be better if HBase had constants defined in a Java interface that
> would be used by HBase as the default value if none other is provided, like
> this:
> config.getInt("hbase.client.scanner.caching",
> HBaseDefault.CLIENT_SCANNER_CACHING);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8197) Cannot distinguish between HBase default config value and client overridden config value
[
https://issues.apache.org/jira/browse/HBASE-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615599#comment-13615599
]
Jonathan Hsieh edited comment on HBASE-8197 at 3/27/13 6:29 PM:
This is already in my hbase testing clusters -- a hbase 0.96.5rc on top of
hadoop2. [~jmspaggi] from the reaction, it sounds like this may be something
only in hadoop2 (which I run/test on top of). If you run this on hbase+hadoop1
do you get the "loaded from" info?
was (Author: jmhsieh):
This is in my hbase testing clusters -- hbase on top of hadoop2.
[~jmspaggi] from the reaction, it sounds like this may be something only in
hadoop2 (which I run/test on top of). If you run this on hbase+hadoop1 do you
get the "loaded from" info?
> Cannot distinguish between HBase default config value and client overridden
> config value
>
>
> Key: HBASE-8197
> URL: https://issues.apache.org/jira/browse/HBASE-8197
> Project: HBase
> Issue Type: Bug
> Components: Usability
>Affects Versions: 0.94.6
>Reporter: James Taylor
>Priority: Minor
>
> By HBase setting default values in hbase-default.xml, it makes it difficult
> for a platform to know if a value has been overridden by the client. For
> example, Phoenix has a default value it would like to use for
> hbase.client.scanner.caching. I'd like to be able to call config.setIfUnset
> option to use our default value if a client hasn't already provided their
> own. Since HBase sets this in hbase-default.xml to 1, I can't differentiate
> between the client have a value in their hbase-sites.xml as 1 versus the
> default value we're getting from HBase.
> Wouldn't it be better if HBase had constants defined in a Java interface that
> would be used by HBase as the default value if none other is provided, like
> this:
> config.getInt("hbase.client.scanner.caching",
> HBaseDefault.CLIENT_SCANNER_CACHING);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8161) setting blocking file count on table level doesn't work
[ https://issues.apache.org/jira/browse/HBASE-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8161: Resolution: Fixed Release Note: in trunk and 95 Status: Resolved (was: Patch Available) > setting blocking file count on table level doesn't work > --- > > Key: HBASE-8161 > URL: https://issues.apache.org/jira/browse/HBASE-8161 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-8161-v0-hmm.patch, HBASE-8161-v0.patch > > > MemStoreFlusher always uses server-wide config... it should ask the store > correct question instead -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7948) client doesn't need to refresh meta while the region is opening
[ https://issues.apache.org/jira/browse/HBASE-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7948: Resolution: Fixed Status: Resolved (was: Patch Available) > client doesn't need to refresh meta while the region is opening > --- > > Key: HBASE-7948 > URL: https://issues.apache.org/jira/browse/HBASE-7948 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7948-v0.patch, HBASE-7948-v1.patch, > HBASE-7948-v1.patch, HBASE-7948-v2.patch, HBASE-7948-v3.patch, > HBASE-7948-v4.patch, HBASE-7948-v5-.patch, HBASE-7948-v5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8152) Avoid creating empty reference file when splitkey is outside the range of a store file
[ https://issues.apache.org/jira/browse/HBASE-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615609#comment-13615609 ] Ted Yu commented on HBASE-8152: --- Nice review, Sergey. I wonder if priority for this JIRA should be at Trivia. > Avoid creating empty reference file when splitkey is outside the range of a > store file > -- > > Key: HBASE-8152 > URL: https://issues.apache.org/jira/browse/HBASE-8152 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, HFile >Affects Versions: 0.94.5 >Reporter: clockfly >Assignee: clockfly >Priority: Trivial > Fix For: 0.94.7 > > Attachments: hbase-8152.0.94patch.v2, hbase-8152.patch0.94 > > > When splitting a store file, if the split key is before the first key, or > greater than the last key, then only one reference file should be created. > Currently, two reference file will be created. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7875) introduce a compaction switch in HBase Shell
[
https://issues.apache.org/jira/browse/HBASE-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615611#comment-13615611
]
Sergey Shelukhin commented on HBASE-7875:
-
+0 on feature - if it makes sense as global config setting it could be added as
a config setting and it should work for both table and column family implicitly
(will have to be asked of the store, not region, though).
+1 on patch
> introduce a compaction switch in HBase Shell
>
>
> Key: HBASE-7875
> URL: https://issues.apache.org/jira/browse/HBASE-7875
> Project: HBase
> Issue Type: New Feature
> Components: shell
>Affects Versions: 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Trivial
> Attachments: HBASE-7875.txt
>
>
> A trivial patch to add an per-table compaction switch. it would be helpful
> for OPS or manual testing in real cluster, etc.
> Usage:
> hbase(main):016:0> alter 'YCSBTest', {METHOD => 'table_att',
> COMPACTION_ENABLE => 'FALSE'}
> Updating all regions with the new schema...
> 64/64 regions updated.
> Done.
> 0 row(s) in 1.1290 seconds
> hbase(main):017:0> describe 'YCSBTest'
> DESCRIPTION
> ENABLED
>
> {NAME => 'YCSBTest', COMPACTION_ENABLE => 'false', SPLIT_POLICY =>
> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPoli true
>
> cy', FAMILIES => [{NAME => 'test', DATA_BLOCK_ENCODING => 'NONE',
> BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'S
>
> NAPPY', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0',
> KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DI
>
> SK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
>
>
> 1 row(s) in 0.0210 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8152) Avoid creating empty reference file when splitkey is outside the range of a store file
[ https://issues.apache.org/jira/browse/HBASE-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615609#comment-13615609 ] Ted Yu edited comment on HBASE-8152 at 3/27/13 6:35 PM: Nice review, Sergey. I wonder if priority for this JIRA should be at Trivial. was (Author: yuzhih...@gmail.com): Nice review, Sergey. I wonder if priority for this JIRA should be at Trivia. > Avoid creating empty reference file when splitkey is outside the range of a > store file > -- > > Key: HBASE-8152 > URL: https://issues.apache.org/jira/browse/HBASE-8152 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, HFile >Affects Versions: 0.94.5 >Reporter: clockfly >Assignee: clockfly >Priority: Trivial > Fix For: 0.94.7 > > Attachments: hbase-8152.0.94patch.v2, hbase-8152.patch0.94 > > > When splitting a store file, if the split key is before the first key, or > greater than the last key, then only one reference file should be created. > Currently, two reference file will be created. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8203) master ui should display region servers with host, port plus startcode
rajeshbabu created HBASE-8203:
-
Summary: master ui should display region servers with host, port
plus startcode
Key: HBASE-8203
URL: https://issues.apache.org/jira/browse/HBASE-8203
Project: HBase
Issue Type: Bug
Components: UI
Affects Versions: 0.95.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Priority: Minor
Fix For: 0.98.0
in master UI we are displaying only hostname of region servers.
some shell commands need servername(host,port,startcode format) as argument
{code}
A 'server_name' is the host, port plus startcode of a regionserver. For
example: host187.example.com,60020,1289493121758 (find servername in
master ui or when you do detailed status in shell)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8203) master ui should display region servers with host, port plus startcode
[
https://issues.apache.org/jira/browse/HBASE-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8203:
--
Attachment: HBASE-8203.patch
trivial patch.
> master ui should display region servers with host, port plus startcode
> --
>
> Key: HBASE-8203
> URL: https://issues.apache.org/jira/browse/HBASE-8203
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 0.95.0
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: HBASE-8203.patch
>
>
> in master UI we are displaying only hostname of region servers.
> some shell commands need servername(host,port,startcode format) as argument
> {code}
> A 'server_name' is the host, port plus startcode of a regionserver. For
> example: host187.example.com,60020,1289493121758 (find servername in
> master ui or when you do detailed status in shell)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8197) Cannot distinguish between HBase default config value and client overridden config value
[
https://issues.apache.org/jira/browse/HBASE-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615617#comment-13615617
]
Jean-Marc Spaggiari commented on HBASE-8197:
Here is what I'm getting from 0.94.6 + Hadoop 1.0.3
{code}
hbase@node3:~/hbase-0.94.6$ bin/hbase org.apache.hadoop.hbase.HBaseConfiguration
hbase.online.schema.update.enablefalse
fs.s3n.implorg.apache.hadoop.fs.s3native.NativeS3FileSystem
hbase.master.port6
hbase.hstore.compaction.max10
{code}
> Cannot distinguish between HBase default config value and client overridden
> config value
>
>
> Key: HBASE-8197
> URL: https://issues.apache.org/jira/browse/HBASE-8197
> Project: HBase
> Issue Type: Bug
> Components: Usability
>Affects Versions: 0.94.6
>Reporter: James Taylor
>Priority: Minor
>
> By HBase setting default values in hbase-default.xml, it makes it difficult
> for a platform to know if a value has been overridden by the client. For
> example, Phoenix has a default value it would like to use for
> hbase.client.scanner.caching. I'd like to be able to call config.setIfUnset
> option to use our default value if a client hasn't already provided their
> own. Since HBase sets this in hbase-default.xml to 1, I can't differentiate
> between the client have a value in their hbase-sites.xml as 1 versus the
> default value we're getting from HBase.
> Wouldn't it be better if HBase had constants defined in a Java interface that
> would be used by HBase as the default value if none other is provided, like
> this:
> config.getInt("hbase.client.scanner.caching",
> HBaseDefault.CLIENT_SCANNER_CACHING);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8200) Integration Test classes are not part of the default HBase classpath. We should fix this or note workaround in documentation
[ https://issues.apache.org/jira/browse/HBASE-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615620#comment-13615620 ] Enis Soztutar commented on HBASE-8200: -- bq. Should it work from source checkout? Yes, ideally. Maven is one way of running things, but it should be runnable from the command line as well. > Integration Test classes are not part of the default HBase classpath. We > should fix this or note workaround in documentation > > > Key: HBASE-8200 > URL: https://issues.apache.org/jira/browse/HBASE-8200 > Project: HBase > Issue Type: Bug > Components: test > Environment: ubuntu >Reporter: Aleksandr Shulman > > Expected behavior: > Following the instructions here: > http://hbase.apache.org/book/hbase.tests.html#integration.tests > After pulling down the latest bits from trunk and running, > > mvn clean compile test-compile package install assembly:assembly -DskipTests > I should be able to run integration tests with the following command: > > bin/hbase --config /path/to/configs > > org.apache.hadoop.hbase.IntegrationTestsDriver > According to the documentation, this should kick off the tests. > Observed behavior: > Upon executing > > bin/hbase --config /path/to/configs > > org.apache.hadoop.hbase.IntegrationTestsDriver > I see the following: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/IntegrationTestsDriver > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hbase.IntegrationTestsDriver > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > I can fix it by adding the following line to the hbase-env.sh file > corresponding to the configs: > export HBASE_CLASSPATH=/path/to/project/hbase/hbase-it/target/test-classes/ > This produces the correct output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7680) implement compaction policy for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-7680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615629#comment-13615629 ] Jimmy Xiang commented on HBASE-7680: StripeCompactor has methods not implemented. Did your performance test use the attached patch? > implement compaction policy for stripe compactions > -- > > Key: HBASE-7680 > URL: https://issues.apache.org/jira/browse/HBASE-7680 > Project: HBase > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7680-v0.patch, > HBASE-7680-v0-with-7679-and-7935.patch, HBASE-7680-v1.patch, > HBASE-7680-v1-with-7679.patch, HBASE-7680-v2.patch, > HBASE-7680-v2-with-7679-and-8034.patch, HBASE-7680-v3.patch, > HBASE-7680-v3-with-7679.patch, HBASE-7680-v4.patch, > HBASE-7680-v4-with-7679.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7680) implement compaction policy for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-7680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615632#comment-13615632 ] Sergey Shelukhin commented on HBASE-7680: - Compactor is in HBASE-7967, tests in HBASE-8000. I changed policy logic as described in the doc. Thanks for feedback on /r/! I'll set some tests in motion and get to it soon :) > implement compaction policy for stripe compactions > -- > > Key: HBASE-7680 > URL: https://issues.apache.org/jira/browse/HBASE-7680 > Project: HBase > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7680-v0.patch, > HBASE-7680-v0-with-7679-and-7935.patch, HBASE-7680-v1.patch, > HBASE-7680-v1-with-7679.patch, HBASE-7680-v2.patch, > HBASE-7680-v2-with-7679-and-8034.patch, HBASE-7680-v3.patch, > HBASE-7680-v3-with-7679.patch, HBASE-7680-v4.patch, > HBASE-7680-v4-with-7679.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8164: -- Fix Version/s: 0.95.0 Hadoop Flags: Reviewed Integrated to 0.95 and trunk. Thanks for the reviews, Stack and Enis. Will wait for a few builds before resolving. > TestTableLockManager fails intermittently in trunk builds > - > > Key: HBASE-8164 > URL: https://issues.apache.org/jira/browse/HBASE-8164 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.95.0, 0.98.0 > > Attachments: 8164-v2.txt, 8164-v3.txt, 8164-v4.txt > > > In build #3979: > testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test > timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8204) Logging improvements and try to recover lease even when append is not activated.
Nicolas Liochon created HBASE-8204: -- Summary: Logging improvements and try to recover lease even when append is not activated. Key: HBASE-8204 URL: https://issues.apache.org/jira/browse/HBASE-8204 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor See patch :-). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8204) Logging improvements and try to recover lease even when append is not activated.
[ https://issues.apache.org/jira/browse/HBASE-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-8204: --- Attachment: HBASE-8204.v1.patch > Logging improvements and try to recover lease even when append is not > activated. > > > Key: HBASE-8204 > URL: https://issues.apache.org/jira/browse/HBASE-8204 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Attachments: HBASE-8204.v1.patch > > > See patch :-). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8204) Logging improvements and try to recover lease even when append is not activated.
[ https://issues.apache.org/jira/browse/HBASE-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-8204: --- Status: Patch Available (was: Open) > Logging improvements and try to recover lease even when append is not > activated. > > > Key: HBASE-8204 > URL: https://issues.apache.org/jira/browse/HBASE-8204 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Attachments: HBASE-8204.v1.patch > > > See patch :-). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7878) recoverFileLease does not check return value of recoverLease
[ https://issues.apache.org/jira/browse/HBASE-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615651#comment-13615651 ] Eric Newton commented on HBASE-7878: Perhaps SequenceFileLogReader can be removed? Accumulo didn't use a sequence file because of the unreliable file size. Maybe we are getting correct file sizes now that we're all calling recoverLease() properly? It would be a nice simplification if it is no longer necessary. I've opened ACCUMULO-1216 to investigate the corresponding change in accumulo. > recoverFileLease does not check return value of recoverLease > > > Key: HBASE-7878 > URL: https://issues.apache.org/jira/browse/HBASE-7878 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.95.0, 0.94.6 >Reporter: Eric Newton >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0 > > Attachments: 7878.94, 7878-94.addendum, 7878-94.addendum2, > 7878-addendum.txt, 7878-trunk.addendum, 7878-trunk.addendum2, > 7878-trunk-v10.txt, 7878-trunk-v11-test.txt, 7878-trunk-v12.txt, > 7878-trunk-v13.txt, 7878-trunk-v14.txt, 7878-trunk-v15.patch, > 7878-trunk-v16.txt, 7878-trunk-v2.txt, 7878-trunk-v3.txt, 7878-trunk-v4.txt, > 7878-trunk-v5.txt, 7878-trunk-v6.txt, 7878-trunk-v7.txt, 7878-trunk-v8.txt, > 7878-trunk-v9.txt, 7878-trunk-v9.txt > > > I think this is a problem, so I'm opening a ticket so an HBase person takes a > look. > Apache Accumulo has moved its write-ahead log to HDFS. I modeled the lease > recovery for Accumulo after HBase's lease recovery. During testing, we > experienced data loss. I found it is necessary to wait until recoverLease > returns true to know that the file has been truly closed. In FSHDFSUtils, > the return result of recoverLease is not checked. In the unit tests created > to check lease recovery in HBASE-2645, the return result of recoverLease is > always checked. > I think FSHDFSUtils should be modified to check the return result, and wait > until it returns true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8204) Logging improvements and try to recover lease even when append is not activated.
[
https://issues.apache.org/jira/browse/HBASE-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615664#comment-13615664
]
Ted Yu commented on HBASE-8204:
---
Looks like Hadoop QA is down again.
{code}
+ if (!isAppendSupported(conf)) {
+LOG.warn("Running on HDFS without append. Can't recover the lease
with append.");
+break;
+ }
{code}
Should we return in the above case ? Otherwise we incur the 1 second sleep.
> Logging improvements and try to recover lease even when append is not
> activated.
>
>
> Key: HBASE-8204
> URL: https://issues.apache.org/jira/browse/HBASE-8204
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Attachments: HBASE-8204.v1.patch
>
>
> See patch :-).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8204) Logging improvements and try to recover lease even when append is not activated.
[ https://issues.apache.org/jira/browse/HBASE-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615668#comment-13615668 ] Ted Yu commented on HBASE-8204: --- In the above case, when ex != null I think throwing ex out of recoverFileLease() may be more informative to the caller. > Logging improvements and try to recover lease even when append is not > activated. > > > Key: HBASE-8204 > URL: https://issues.apache.org/jira/browse/HBASE-8204 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.96.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Attachments: HBASE-8204.v1.patch > > > See patch :-). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8203) master ui should display region servers with host, port plus startcode
[
https://issues.apache.org/jira/browse/HBASE-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615671#comment-13615671
]
Sergey Shelukhin commented on HBASE-8203:
-
+1
> master ui should display region servers with host, port plus startcode
> --
>
> Key: HBASE-8203
> URL: https://issues.apache.org/jira/browse/HBASE-8203
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 0.95.0
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: HBASE-8203.patch
>
>
> in master UI we are displaying only hostname of region servers.
> some shell commands need servername(host,port,startcode format) as argument
> {code}
> A 'server_name' is the host, port plus startcode of a regionserver. For
> example: host187.example.com,60020,1289493121758 (find servername in
> master ui or when you do detailed status in shell)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7615) Add metrics for snapshots
[ https://issues.apache.org/jira/browse/HBASE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7615: --- Attachment: HBASE-7615-v1.patch Added snapshot, restore, clone time metrics > Add metrics for snapshots > - > > Key: HBASE-7615 > URL: https://issues.apache.org/jira/browse/HBASE-7615 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Matteo Bertozzi > Attachments: HBASE-7615-v0.patch, HBASE-7615-v0-ui-corrupted.png, > HBASE-7615-v0-ui.png, HBASE-7615-v1.patch > > > Metrics should be added for snapshot. > From Matteo: Output that we have in SnapshotInfo should be covered: > "%d HFiles (%d in archive), total size %s (%.2f%% %s shared with the source > table)" > Jesse mentioned snaphot counts, average time to completion. > I think we should have counts for successful / failed snapshots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615699#comment-13615699
]
Sergey Shelukhin commented on HBASE-7970:
-
Logic in readBlockData in HFileBlock can use these interchangably...
>From other places two separate streams would be trivial to remove.
Given that we expect checksum failures to happen rarely, should we keep one of
these in block reader, and open the other one on demand if we do get checksum
failure?
> Improve file descriptor usage: currently, there are two file descriptors per
> storefile
> --
>
> Key: HBASE-7970
> URL: https://issues.apache.org/jira/browse/HBASE-7970
> Project: HBase
> Issue Type: Sub-task
>Reporter: Himanshu Vashishtha
>
> This is because there are two open calls in the HFile: one with checksum and
> another for without checksum support in v2:
> see the method in HFile:createReaderWithEncoding()
> {code}
> FSDataInputStream fsdis = fs.open(path);
> FSDataInputStream fsdisNoFsChecksum = fsdis;
> // If the fs is not an instance of HFileSystem, then create an
> // instance of HFileSystem that wraps over the specified fs.
> // In this case, we will not be able to avoid checksumming inside
> // the filesystem.
> if (!(fs instanceof HFileSystem)) {
> hfs = new HFileSystem(fs);
> } else {
> hfs = (HFileSystem)fs;
> // open a stream to read data without checksum verification in
> // the filesystem
> fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615739#comment-13615739
]
Ted Yu commented on HBASE-7970:
---
In HFileBlock#FSReaderV2() ctor:
{code}
this.useHBaseChecksumConfigured = useHBaseChecksum;
{code}
In HFile#createReaderWithEncoding():
{code}
if (!(fs instanceof HFileSystem)) {
hfs = new HFileSystem(fs);
} else {
hfs = (HFileSystem)fs;
// open a stream to read data without checksum verification in
// the filesystem
fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
}
return pickReaderVersion(path, fsdis, fsdisNoFsChecksum,
{code}
FSReaderV2 ctor is called through pickReaderVersion().
Looks like we can check configuration prior to calling pickReaderVersion() so
that fsdisNoFsChecksum doesn't have to be opened if HBase checksum is inactive.
> Improve file descriptor usage: currently, there are two file descriptors per
> storefile
> --
>
> Key: HBASE-7970
> URL: https://issues.apache.org/jira/browse/HBASE-7970
> Project: HBase
> Issue Type: Sub-task
>Reporter: Himanshu Vashishtha
>
> This is because there are two open calls in the HFile: one with checksum and
> another for without checksum support in v2:
> see the method in HFile:createReaderWithEncoding()
> {code}
> FSDataInputStream fsdis = fs.open(path);
> FSDataInputStream fsdisNoFsChecksum = fsdis;
> // If the fs is not an instance of HFileSystem, then create an
> // instance of HFileSystem that wraps over the specified fs.
> // In this case, we will not be able to avoid checksumming inside
> // the filesystem.
> if (!(fs instanceof HFileSystem)) {
> hfs = new HFileSystem(fs);
> } else {
> hfs = (HFileSystem)fs;
> // open a stream to read data without checksum verification in
> // the filesystem
> fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7615) Add metrics for snapshots
[
https://issues.apache.org/jira/browse/HBASE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615777#comment-13615777
]
Ted Yu commented on HBASE-7615:
---
Went through the patch which looks good.
{code}
+new CloneSnapshotHandler(master, snapshot, hTableDescriptor,
metricsMaster).prepare();
{code}
nit: both metricsMaster and this.metricsMaster are used. It would be nice to be
consistent.
Hadoop QA is down again - may need to run test suite manually or later.
> Add metrics for snapshots
> -
>
> Key: HBASE-7615
> URL: https://issues.apache.org/jira/browse/HBASE-7615
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
> Attachments: HBASE-7615-v0.patch, HBASE-7615-v0-ui-corrupted.png,
> HBASE-7615-v0-ui.png, HBASE-7615-v1.patch
>
>
> Metrics should be added for snapshot.
> From Matteo: Output that we have in SnapshotInfo should be covered:
> "%d HFiles (%d in archive), total size %s (%.2f%% %s shared with the source
> table)"
> Jesse mentioned snaphot counts, average time to completion.
> I think we should have counts for successful / failed snapshots.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7679) implement store file management for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-7679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7679: Attachment: HBASE-7679-v12.patch Updating with r feedback > implement store file management for stripe compactions > -- > > Key: HBASE-7679 > URL: https://issues.apache.org/jira/browse/HBASE-7679 > Project: HBase > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7667-and-7603-v0-incomplete.patch, > HBASE-7667-and-7603-v0-incomplete.patch, HBASE-7667-and-7603-v1.patch, > HBASE-7667-and-7603-v1.patch, HBASE-7667-v1.patch, HBASE-7667-v1.patch, > HBASE-7667-v2.patch, HBASE-7667-v2.patch, HBASE-7667-v3.patch, > HBASE-7679-v10.patch, HBASE-7679-v11.patch, HBASE-7679-v12.patch, > HBASE-7679-v4.patch, HBASE-7679-v5.patch, HBASE-7679-v6.patch, > HBASE-7679-v7-.patch, HBASE-7679-v7.patch, HBASE-7679-v8.patch, > HBASE-7679-v9.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7871) HBase can be stuck when closing regions concurrently
[
https://issues.apache.org/jira/browse/HBASE-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615800#comment-13615800
]
Ted Yu commented on HBASE-7871:
---
[~eclark]:
Your opinion on this issue would be valuable.
> HBase can be stuck when closing regions concurrently
> -
>
> Key: HBASE-7871
> URL: https://issues.apache.org/jira/browse/HBASE-7871
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.96.0
>Reporter: Nicolas Liochon
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.95.0, 0.98.0
>
> Attachments: 7871.patch, 7871-v2.patch, s1.txt, TestStartStop.java
>
>
> The attached test fails ~1% of the the time on 0.96. It seems it does not
> fail on 0.94.5. It's simple: a table creation and some puts.
> I attach the stack. Logs says nothing it seems.
> The suspicious part is:
> {noformat}
> "RS_CLOSE_REGION-localhost,57575,1361197489166-2" prio=10
> tid=0x7fb0c8775800 nid=0x61ac runnable [0x7fb09f272000]
>java.lang.Thread.State: RUNNABLE
> at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
> at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
> at java.util.TreeMap.remove(TreeMap.java:585)
> at java.util.TreeSet.remove(TreeSet.java:259)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:55)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:86)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:40)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1063)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:969)
> - locked <0x0006944e2558> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:146)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:203)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8205) HBCK support for table locks
Enis Soztutar created HBASE-8205: Summary: HBCK support for table locks Key: HBASE-8205 URL: https://issues.apache.org/jira/browse/HBASE-8205 Project: HBase Issue Type: Improvement Components: hbck, master, regionserver Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.95.0, 0.98.0 Table locks have been introduced in HBASE-7305, HBASE-7546, and others (see the design doc at HBASE-7305). This issue adds support in HBCK to report and fix possible conditions about table locks. Namely, if due to some bug, the table lock remains not-released, then HBCK should be able to report it, and remove the lock, so that normal table operations will continue. Also see the comments in HBASE-7977. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8205) HBCK support for table locks
[ https://issues.apache.org/jira/browse/HBASE-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8205: - Attachment: hbase-8205_v1.patch Attaching a patch: - Adds configuration "hbase.table.lock.expire.ms", 10min by default. - HBCK, by default checks for "expired" table locks. Only errors out if some table lock is expired. - Adds -fixTableLocks arg to hbck. When set, it deletes the locks, so normal operation can continue. > HBCK support for table locks > > > Key: HBASE-8205 > URL: https://issues.apache.org/jira/browse/HBASE-8205 > Project: HBase > Issue Type: Improvement > Components: hbck, master, regionserver >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.95.0, 0.98.0 > > Attachments: hbase-8205_v1.patch > > > Table locks have been introduced in HBASE-7305, HBASE-7546, and others (see > the design doc at HBASE-7305). > This issue adds support in HBCK to report and fix possible conditions about > table locks. Namely, if due to some bug, the table lock remains not-released, > then HBCK should be able to report it, and remove the lock, so that normal > table operations will continue. > Also see the comments in HBASE-7977. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
