[jira] [Commented] (HBASE-10594) Speed up TestRestoreSnapshotFromClient a bit
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909682#comment-13909682 ] Matteo Bertozzi commented on HBASE-10594: - +1 > Speed up TestRestoreSnapshotFromClient a bit > > > Key: HBASE-10594 > URL: https://issues.apache.org/jira/browse/HBASE-10594 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 10594-0.94.txt > > > Looking through the longest running test in 0.94 I noticed that > TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes > (264s on my local box). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10584) Inconsistency between tableExists and listTables in implementation
[
https://issues.apache.org/jira/browse/HBASE-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909669#comment-13909669
]
Feng Honghua commented on HBASE-10584:
--
The failed case is due to another bug in HMaster/FSTableDescriptors for which
HBASE-10595 is created. This case can pass after that bug is fixed.
> Inconsistency between tableExists and listTables in implementation
> --
>
> Key: HBASE-10584
> URL: https://issues.apache.org/jira/browse/HBASE-10584
> Project: HBase
> Issue Type: Bug
> Components: Client, master
>Reporter: Feng Honghua
>Assignee: Feng Honghua
> Attachments: HBASE-10584-trunk_v1.patch, HBASE-10584-trunk_v1.patch
>
>
> # HBaseAdmin.tableExists is implemented by scanning meta table
> # HBaseAdmin.listTables(and HBaseAdmin.getTableDescriptor) is implemented by
> talking with HMaster which responses by querying the FSTableDescriptors, and
> FSTableDescriptors return all tables by scanning all the table descriptor
> files in FS(cache also plays here, so most of time it can be satisfied by
> cache)...
> Actually HBaseAdmin requests HMaster to check if a table exists internally
> when implementing deleteTable(see below), then why does it use a
> different(scanning meta table) way to implementing tableExists() for outside
> user to use for the same purpose?
> {code}
> tableExists = false;
> GetTableDescriptorsResponse htds;
> MasterKeepAliveConnection master = connection.getKeepAliveMasterService();
> try {
> GetTableDescriptorsRequest req =
> RequestConverter.buildGetTableDescriptorsRequest(tableName);
> htds = master.getTableDescriptors(null, req);
> } catch (ServiceException se) {
> throw ProtobufUtil.getRemoteException(se);
> } finally {
> master.close();
> }
> tableExists = !htds.getTableSchemaList().isEmpty();
> {code}
> (Above verifying that table descriptor file is deleted can guarantee all
> items of this table are deleted from meta table...)
> Since creating table descriptor files and inserting item to meta table occur
> in different time without atomic semantic, this inconsistency in
> implementation can lead to confusing behaviors when create-table or
> delete-table fails midway, (before according cleanup is done) table
> descriptor file may exists while no item exists in meta table (for
> create-table where table descriptor file is created before inserting item to
> meta table), this leads to listTables including that table, while tableExists
> says no. Similar inconsistency if delete-table fails mid-way...
> Confusing behavior can happen during the process even though eventually it
> succeed:
> # During table creation, when a user calls listTables and then calls
> tableExists for this table after the table descriptor is created but before
> item is inserted to meta table. He will find the listTables includes a table
> but tableExists return false for that same table, this behavior is confusing
> and should only acceptable during the table is being deleted...
> # Similar behavior occurs during table deletion.
> Seems the benefit of implementing tableExists this way is we can avoid
> talking with HMaster, considering we talk with HMaster for listTables and
> getTableDescriptor, such benefit can't offset the drawback from inconsistency.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HBASE-10595) HBaseAdmin.getTableDescriptor can wrongly get the previous table's TableDescriptor even after the table dir in hdfs is removed
Feng Honghua created HBASE-10595: Summary: HBaseAdmin.getTableDescriptor can wrongly get the previous table's TableDescriptor even after the table dir in hdfs is removed Key: HBASE-10595 URL: https://issues.apache.org/jira/browse/HBASE-10595 Project: HBase Issue Type: Bug Components: master, util Reporter: Feng Honghua Assignee: Feng Honghua When a table dir (in hdfs) is removed(by outside), HMaster will still return the cached TableDescriptor to client for getTableDescriptor request. On the contrary, HBaseAdmin.listTables() is handled correctly in current implementation, for a table whose table dir in hdfs is removed by outside, getTableDescriptor can still retrieve back a valid (old) table descriptor, while listTables says it doesn't exist, this is inconsistent The reason for this bug is because HMaster (via FSTableDescriptors) doesn't check if the table dir exists for getTableDescriptor() request, (while it lists all existing table dirs(not firstly respects cache) and returns accordingly for listTables() request) When a table is deleted via deleteTable, the cache will be cleared after the table dir and tableInfo file is removed, listTables/getTableDescriptor inconsistency should be transient(though still exists, when table dir is removed while cache is not cleared) and harder to expose -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10594) Speed up TestRestoreSnapshotFromClient a bit
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-10594: -- Attachment: 10594-0.94.txt Trivial change. The observation is that snapshot1 is only used in one of the tests, but yet it is re-created for every single test. Shave 10% of the runtime (i.e. 1 minute on the jenkinds boxes). Not important, but might not hurt either. > Speed up TestRestoreSnapshotFromClient a bit > > > Key: HBASE-10594 > URL: https://issues.apache.org/jira/browse/HBASE-10594 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 10594-0.94.txt > > > Looking through the longest running test in 0.94 I noticed that > TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes > (264s on my local box). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HBASE-10594) Speed up TestRestoreSnapshotFromClient a bit
Lars Hofhansl created HBASE-10594: - Summary: Speed up TestRestoreSnapshotFromClient a bit Key: HBASE-10594 URL: https://issues.apache.org/jira/browse/HBASE-10594 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Minor Looking through the longest running test in 0.94 I noticed that TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes (264s on my local box). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-8803) region_mover.rb should move multiple regions at a time
[ https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909646#comment-13909646 ] Lars Hofhansl commented on HBASE-8803: -- Let's get it in. > region_mover.rb should move multiple regions at a time > -- > > Key: HBASE-8803 > URL: https://issues.apache.org/jira/browse/HBASE-8803 > Project: HBase > Issue Type: Bug > Components: Usability >Affects Versions: 0.98.0, 0.94.8, 0.95.1 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Fix For: 0.99.0 > > Attachments: 8803v5.txt, HBASE-8803-v0-trunk.patch, > HBASE-8803-v1-0.94.patch, HBASE-8803-v1-trunk.patch, > HBASE-8803-v2-0.94.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v3-0.94.patch, > HBASE-8803-v4-0.94.patch, HBASE-8803-v4-trunk.patch, > HBASE-8803-v5-0.94.patch, HBASE-8803-v6-0.94.patch, HBASE-8803-v6-trunk.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > When there is many regions in a cluster, rolling_restart can take hours > because region_mover is moving the regions one by one. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10592) Refactor PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909644#comment-13909644 ] Jean-Marc Spaggiari commented on HBASE-10592: - Will test that tomorrow for sure. Currently actively testing 0.94.17 so I'm delayed a bit... > Refactor PerformanceEvaluation tool > --- > > Key: HBASE-10592 > URL: https://issues.apache.org/jira/browse/HBASE-10592 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.96.2, 0.98.1, 0.99.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10592.00-0.96.patch, HBASE-10592.00-0.98.patch, > HBASE-10592.00.patch > > > PerfEval is kind of a mess. It's painful to add new features because the test > options are itemized and passed as parameters to internal methods. > Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode > is a chore because of it. > This patch refactors the tool. Options are now passed around to methods and > such as a POJO instead of one-by-one. Get rid of accessors that don't help > anyone. On the mapreduce side, serialization is now handled using json > (jackson is a dependency anyway) instead of the hand-rolled regex we used > before. Also do away with custom InputSplit and FileFormat, instead using > Text and NLineInputFormat. On the local mode side, combine 1 client and N > clients into the same implementation. That implementation now uses an > ExecutorService, so we can later decouple number of client workers from > number of client tasks. Finally, drop a bunch of confusing local state, > instead use the new TestOptions POJO as a parameter to static methods. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10592) Refactor PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909641#comment-13909641 ] stack commented on HBASE-10592: --- +1 on making a testing tool easier to customize. +1 on 0.96. > Refactor PerformanceEvaluation tool > --- > > Key: HBASE-10592 > URL: https://issues.apache.org/jira/browse/HBASE-10592 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.96.2, 0.98.1, 0.99.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10592.00-0.96.patch, HBASE-10592.00-0.98.patch, > HBASE-10592.00.patch > > > PerfEval is kind of a mess. It's painful to add new features because the test > options are itemized and passed as parameters to internal methods. > Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode > is a chore because of it. > This patch refactors the tool. Options are now passed around to methods and > such as a POJO instead of one-by-one. Get rid of accessors that don't help > anyone. On the mapreduce side, serialization is now handled using json > (jackson is a dependency anyway) instead of the hand-rolled regex we used > before. Also do away with custom InputSplit and FileFormat, instead using > Text and NLineInputFormat. On the local mode side, combine 1 client and N > clients into the same implementation. That implementation now uses an > ExecutorService, so we can later decouple number of client workers from > number of client tasks. Finally, drop a bunch of confusing local state, > instead use the new TestOptions POJO as a parameter to static methods. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10593) FileInputStream in JenkinsHash#main() is never closed
[
https://issues.apache.org/jira/browse/HBASE-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909639#comment-13909639
]
stack commented on HBASE-10593:
---
Why not work on removing this unused class instead of 'fixing' it?
> FileInputStream in JenkinsHash#main() is never closed
> -
>
> Key: HBASE-10593
> URL: https://issues.apache.org/jira/browse/HBASE-10593
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Trivial
>
> {code}
> FileInputStream in = new FileInputStream(args[0]);
> {code}
> The above FileInputStream is not closed.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-8803) region_mover.rb should move multiple regions at a time
[ https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909616#comment-13909616 ] Jean-Marc Spaggiari commented on HBASE-8803: Really want to get this committed folks :( I have to re-apply it locally each time there is a new release to test. Any chance? > region_mover.rb should move multiple regions at a time > -- > > Key: HBASE-8803 > URL: https://issues.apache.org/jira/browse/HBASE-8803 > Project: HBase > Issue Type: Bug > Components: Usability >Affects Versions: 0.98.0, 0.94.8, 0.95.1 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Fix For: 0.99.0 > > Attachments: 8803v5.txt, HBASE-8803-v0-trunk.patch, > HBASE-8803-v1-0.94.patch, HBASE-8803-v1-trunk.patch, > HBASE-8803-v2-0.94.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v3-0.94.patch, > HBASE-8803-v4-0.94.patch, HBASE-8803-v4-trunk.patch, > HBASE-8803-v5-0.94.patch, HBASE-8803-v6-0.94.patch, HBASE-8803-v6-trunk.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > When there is many regions in a cluster, rolling_restart can take hours > because region_mover is moving the regions one by one. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10584) Inconsistency between tableExists and listTables in implementation
[
https://issues.apache.org/jira/browse/HBASE-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909615#comment-13909615
]
Feng Honghua commented on HBASE-10584:
--
I'm looking at the failed case.
> Inconsistency between tableExists and listTables in implementation
> --
>
> Key: HBASE-10584
> URL: https://issues.apache.org/jira/browse/HBASE-10584
> Project: HBase
> Issue Type: Bug
> Components: Client, master
>Reporter: Feng Honghua
>Assignee: Feng Honghua
> Attachments: HBASE-10584-trunk_v1.patch, HBASE-10584-trunk_v1.patch
>
>
> # HBaseAdmin.tableExists is implemented by scanning meta table
> # HBaseAdmin.listTables(and HBaseAdmin.getTableDescriptor) is implemented by
> talking with HMaster which responses by querying the FSTableDescriptors, and
> FSTableDescriptors return all tables by scanning all the table descriptor
> files in FS(cache also plays here, so most of time it can be satisfied by
> cache)...
> Actually HBaseAdmin requests HMaster to check if a table exists internally
> when implementing deleteTable(see below), then why does it use a
> different(scanning meta table) way to implementing tableExists() for outside
> user to use for the same purpose?
> {code}
> tableExists = false;
> GetTableDescriptorsResponse htds;
> MasterKeepAliveConnection master = connection.getKeepAliveMasterService();
> try {
> GetTableDescriptorsRequest req =
> RequestConverter.buildGetTableDescriptorsRequest(tableName);
> htds = master.getTableDescriptors(null, req);
> } catch (ServiceException se) {
> throw ProtobufUtil.getRemoteException(se);
> } finally {
> master.close();
> }
> tableExists = !htds.getTableSchemaList().isEmpty();
> {code}
> (Above verifying that table descriptor file is deleted can guarantee all
> items of this table are deleted from meta table...)
> Since creating table descriptor files and inserting item to meta table occur
> in different time without atomic semantic, this inconsistency in
> implementation can lead to confusing behaviors when create-table or
> delete-table fails midway, (before according cleanup is done) table
> descriptor file may exists while no item exists in meta table (for
> create-table where table descriptor file is created before inserting item to
> meta table), this leads to listTables including that table, while tableExists
> says no. Similar inconsistency if delete-table fails mid-way...
> Confusing behavior can happen during the process even though eventually it
> succeed:
> # During table creation, when a user calls listTables and then calls
> tableExists for this table after the table descriptor is created but before
> item is inserted to meta table. He will find the listTables includes a table
> but tableExists return false for that same table, this behavior is confusing
> and should only acceptable during the table is being deleted...
> # Similar behavior occurs during table deletion.
> Seems the benefit of implementing tableExists this way is we can avoid
> talking with HMaster, considering we talk with HMaster for listTables and
> getTableDescriptor, such benefit can't offset the drawback from inconsistency.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10586) hadoop2-compat IPC metric registred twice
[
https://issues.apache.org/jira/browse/HBASE-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909609#comment-13909609
]
Hudson commented on HBASE-10586:
FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #97 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/97/])
HBASE-10586 hadoop2-compat IPC metric registred twice (mbertozzi: rev 1570737)
*
/hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java
> hadoop2-compat IPC metric registred twice
> -
>
> Key: HBASE-10586
> URL: https://issues.apache.org/jira/browse/HBASE-10586
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 0.98.0, 0.96.1.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: HBASE-10586-v0.patch
>
>
> There is an extra snapshot/addrecord line in hadoop2-compat
> MetricsHBaseServerSourceImpl resulting in IPC metrics with a ".1"
> the extra line is not present in the hadoop1-compat "mirror"
> {code}
> "numCallsInGeneralQueue.1" : 0,
> "numCallsInReplicationQueue.1" : 0,
> "numCallsInPriorityQueue.1" : 0,
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10589) Reduce unnecessary TestRowProcessorEndpoint resource usage
[ https://issues.apache.org/jira/browse/HBASE-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909608#comment-13909608 ] Hudson commented on HBASE-10589: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #97 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/97/]) HBASE-10589 Reduce unnecessary TestRowProcessorEndpoint resource usage (apurtell: rev 1570763) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.java > Reduce unnecessary TestRowProcessorEndpoint resource usage > -- > > Key: HBASE-10589 > URL: https://issues.apache.org/jira/browse/HBASE-10589 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10589.patch > > > We don't need 1000 concurrent threads when 100 will do. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10580) IntegrationTestingUtility#restoreCluster leak resource when running in a mini cluster mode
[ https://issues.apache.org/jira/browse/HBASE-10580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909611#comment-13909611 ] Hudson commented on HBASE-10580: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #97 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/97/]) HBASE-10580: IntegrationTestingUtility#restoreCluster leak resource when running in a mini cluster mode (jeffreyz: rev 1570765) * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestingUtility.java > IntegrationTestingUtility#restoreCluster leak resource when running in a mini > cluster mode > -- > > Key: HBASE-10580 > URL: https://issues.apache.org/jira/browse/HBASE-10580 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.96.1, 0.99.0 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: hbase-10580.patch > > > When the utility isn't run in distributed cluster mode, the restore only > shutdown MiniHBaseCluster not MiniDFSCluster & MiniZKCluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10547) TestFixedLengthWrapper#testReadWrite occasionally fails with the IBM JDK
[
https://issues.apache.org/jira/browse/HBASE-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909610#comment-13909610
]
Hudson commented on HBASE-10547:
FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #97 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/97/])
HBASE-10547 TestFixedLengthWrapper#testReadWrite occasionally fails with the
IBM JDK (apurtell: rev 1570744)
*
/hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/types/TestFixedLengthWrapper.java
> TestFixedLengthWrapper#testReadWrite occasionally fails with the IBM JDK
>
>
> Key: HBASE-10547
> URL: https://issues.apache.org/jira/browse/HBASE-10547
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.96.1.1
> Environment: IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64
> Compressed References 20131114_175264 (JIT enabled, AOT enabled)
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: 10547.patch
>
>
> Here's the trace.
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 29.288 sec
> <<< FAILURE!
> testReadWrite(org.apache.hadoop.hbase.types.TestFixedLengthWrapper) Time
> elapsed: 0.025 sec <<< FAILURE!
> arrays first differed at element [8]; expected:<-40> but was:<0>
> at
> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
> at org.junit.Assert.internalArrayEquals(Assert.java:473)
> at org.junit.Assert.assertArrayEquals(Assert.java:294)
> at org.junit.Assert.assertArrayEquals(Assert.java:305)
> at
> org.apache.hadoop.hbase.types.TestFixedLengthWrapper.testReadWrite(TestFixedLengthWrapper.java:60)
> {noformat}
> This is with 0.98.0.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HBASE-10593) FileInputStream in JenkinsHash#main() is never closed
Ted Yu created HBASE-10593:
--
Summary: FileInputStream in JenkinsHash#main() is never closed
Key: HBASE-10593
URL: https://issues.apache.org/jira/browse/HBASE-10593
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Trivial
{code}
FileInputStream in = new FileInputStream(args[0]);
{code}
The above FileInputStream is not closed.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-9914) Port fix for HBASE-9836 'Intermittent TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking failure' to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909588#comment-13909588 ] Ted Yu commented on HBASE-9914: --- That would be great, [~takeshi.miao]. > Port fix for HBASE-9836 'Intermittent > TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking > failure' to 0.94 > - > > Key: HBASE-9914 > URL: https://issues.apache.org/jira/browse/HBASE-9914 > Project: HBase > Issue Type: Test >Reporter: Ted Yu > Labels: noob > > According to this thread: http://search-hadoop.com/m/3CzC31BQsDd , > TestRegionObserverScannerOpenHook#testRegionObserverCompactionTimeStacking > sometimes failed. > This issue is to port the fix from HBASE-9836 to 0.94 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10584) Inconsistency between tableExists and listTables in implementation
[
https://issues.apache.org/jira/browse/HBASE-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909585#comment-13909585
]
Hadoop QA commented on HBASE-10584:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630502/HBASE-10584-trunk_v1.patch
against trunk revision .
ATTACHMENT ID: 12630502
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestHBaseFsck
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8775//console
This message is automatically generated.
> Inconsistency between tableExists and listTables in implementation
> --
>
> Key: HBASE-10584
> URL: https://issues.apache.org/jira/browse/HBASE-10584
> Project: HBase
> Issue Type: Bug
> Components: Client, master
>Reporter: Feng Honghua
>Assignee: Feng Honghua
> Attachments: HBASE-10584-trunk_v1.patch, HBASE-10584-trunk_v1.patch
>
>
> # HBaseAdmin.tableExists is implemented by scanning meta table
> # HBaseAdmin.listTables(and HBaseAdmin.getTableDescriptor) is implemented by
> talking with HMaster which responses by querying the FSTableDescriptors, and
> FSTableDescriptors return all tables by scanning all the table descriptor
> files in FS(cache also plays here, so most of time it can be satisfied by
> cache)...
> Actually HBaseAdmin requests HMaster to check if a table exists internally
> when implementing deleteTable(see below), then why does it use a
> different(scanning meta table) way to implementing tableExists() for outside
> user to use for the same purpose?
> {code}
> tableExists = false;
> GetTableDescriptorsResponse htds;
> MasterKeepAliveConnection master = connection.getKeepAliveMasterService();
> try {
> GetTableDescriptorsRequest req =
> RequestConverter.buildGetTableDescriptorsRequest(tableName);
> htds = master.getTableDescriptors(null, req);
> } catch (ServiceException se) {
> throw ProtobufUtil.getRemoteException(se);
> } finally {
> master.close();
> }
> tableExists = !htds.getTableSchemaLi
[jira] [Commented] (HBASE-10592) Refactor PerformanceEvaluation tool
[
https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909583#comment-13909583
]
Hadoop QA commented on HBASE-10592:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630510/HBASE-10592.00-0.96.patch
against trunk revision .
ATTACHMENT ID: 12630510
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8777//console
This message is automatically generated.
> Refactor PerformanceEvaluation tool
> ---
>
> Key: HBASE-10592
> URL: https://issues.apache.org/jira/browse/HBASE-10592
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.96.2, 0.98.1, 0.99.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 0.99.0
>
> Attachments: HBASE-10592.00-0.96.patch, HBASE-10592.00-0.98.patch,
> HBASE-10592.00.patch
>
>
> PerfEval is kind of a mess. It's painful to add new features because the test
> options are itemized and passed as parameters to internal methods.
> Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode
> is a chore because of it.
> This patch refactors the tool. Options are now passed around to methods and
> such as a POJO instead of one-by-one. Get rid of accessors that don't help
> anyone. On the mapreduce side, serialization is now handled using json
> (jackson is a dependency anyway) instead of the hand-rolled regex we used
> before. Also do away with custom InputSplit and FileFormat, instead using
> Text and NLineInputFormat. On the local mode side, combine 1 client and N
> clients into the same implementation. That implementation now uses an
> ExecutorService, so we can later decouple number of client workers from
> number of client tasks. Finally, drop a bunch of confusing local state,
> instead use the new TestOptions POJO as a parameter to static methods.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909584#comment-13909584
]
Nick Dimiduk commented on HBASE-9953:
-
Attached patch is post-refactor.
> PerformanceEvaluation: Decouple data size from client concurrency
> -
>
> Key: HBASE-9953
> URL: https://issues.apache.org/jira/browse/HBASE-9953
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-9953.00.patch
>
>
> PerfEval tool provides a {{--rows=R}} for specifying the number of records to
> work with and requires the user provide a value of N, used as the concurrency
> level. From what I can tell, every concurrent process will interact with R
> rows. In order to perform an apples-to-apples test, the user must
> re-calculate the value R for every new value of N.
> Instead, I propose accepting a {{--size=S}} for the amount of data to
> interact with and let PerfEval divide that amongst the N clients on the
> user's behalf.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HBASE-9953) PerformanceEvaluation: Decouple data size from client concurrency
[
https://issues.apache.org/jira/browse/HBASE-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-9953:
Attachment: HBASE-9953.00.patch
> PerformanceEvaluation: Decouple data size from client concurrency
> -
>
> Key: HBASE-9953
> URL: https://issues.apache.org/jira/browse/HBASE-9953
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-9953.00.patch
>
>
> PerfEval tool provides a {{--rows=R}} for specifying the number of records to
> work with and requires the user provide a value of N, used as the concurrency
> level. From what I can tell, every concurrent process will interact with R
> rows. In order to perform an apples-to-apples test, the user must
> re-calculate the value R for every new value of N.
> Instead, I propose accepting a {{--size=S}} for the amount of data to
> interact with and let PerfEval divide that amongst the N clients on the
> user's behalf.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HBASE-10592) Refactor PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10592: - Affects Version/s: 0.99.0 0.98.1 0.96.2 > Refactor PerformanceEvaluation tool > --- > > Key: HBASE-10592 > URL: https://issues.apache.org/jira/browse/HBASE-10592 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.96.2, 0.98.1, 0.99.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10592.00-0.96.patch, HBASE-10592.00-0.98.patch, > HBASE-10592.00.patch > > > PerfEval is kind of a mess. It's painful to add new features because the test > options are itemized and passed as parameters to internal methods. > Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode > is a chore because of it. > This patch refactors the tool. Options are now passed around to methods and > such as a POJO instead of one-by-one. Get rid of accessors that don't help > anyone. On the mapreduce side, serialization is now handled using json > (jackson is a dependency anyway) instead of the hand-rolled regex we used > before. Also do away with custom InputSplit and FileFormat, instead using > Text and NLineInputFormat. On the local mode side, combine 1 client and N > clients into the same implementation. That implementation now uses an > ExecutorService, so we can later decouple number of client workers from > number of client tasks. Finally, drop a bunch of confusing local state, > instead use the new TestOptions POJO as a parameter to static methods. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10592) Refactor PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10592: - Attachment: HBASE-10592.00-0.96.patch And here's a patch for 0.96. Changes touching TestHFileOutputFormat was... surprising. Should unwind the dependence in another issue. > Refactor PerformanceEvaluation tool > --- > > Key: HBASE-10592 > URL: https://issues.apache.org/jira/browse/HBASE-10592 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.96.2, 0.98.1, 0.99.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10592.00-0.96.patch, HBASE-10592.00-0.98.patch, > HBASE-10592.00.patch > > > PerfEval is kind of a mess. It's painful to add new features because the test > options are itemized and passed as parameters to internal methods. > Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode > is a chore because of it. > This patch refactors the tool. Options are now passed around to methods and > such as a POJO instead of one-by-one. Get rid of accessors that don't help > anyone. On the mapreduce side, serialization is now handled using json > (jackson is a dependency anyway) instead of the hand-rolled regex we used > before. Also do away with custom InputSplit and FileFormat, instead using > Text and NLineInputFormat. On the local mode side, combine 1 client and N > clients into the same implementation. That implementation now uses an > ExecutorService, so we can later decouple number of client workers from > number of client tasks. Finally, drop a bunch of confusing local state, > instead use the new TestOptions POJO as a parameter to static methods. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HBASE-10405) Intermittent TestDistributedLogSplitting#testLogReplayForDisablingTable failure
[
https://issues.apache.org/jira/browse/HBASE-10405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-10405.
Resolution: Cannot Reproduce
> Intermittent TestDistributedLogSplitting#testLogReplayForDisablingTable
> failure
> ---
>
> Key: HBASE-10405
> URL: https://issues.apache.org/jira/browse/HBASE-10405
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>
> Here was the assertion failure:
> {code} java.lang.AssertionError: expected:<1000> but was:<0> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:743) at
> org.junit.Assert.assertEquals(Assert.java:118) at
> org.junit.Assert.assertEquals(Assert.java:555) at
> org.junit.Assert.assertEquals(Assert.java:542) at
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:838)
>
> {code}
> This was due to the loop starting around line 823 not picking up any edits
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10592) Refactor PerformanceEvaluation tool
[
https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909562#comment-13909562
]
Hadoop QA commented on HBASE-10592:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630504/HBASE-10592.00-0.98.patch
against trunk revision .
ATTACHMENT ID: 12630504
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8776//console
This message is automatically generated.
> Refactor PerformanceEvaluation tool
> ---
>
> Key: HBASE-10592
> URL: https://issues.apache.org/jira/browse/HBASE-10592
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 0.99.0
>
> Attachments: HBASE-10592.00-0.98.patch, HBASE-10592.00.patch
>
>
> PerfEval is kind of a mess. It's painful to add new features because the test
> options are itemized and passed as parameters to internal methods.
> Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode
> is a chore because of it.
> This patch refactors the tool. Options are now passed around to methods and
> such as a POJO instead of one-by-one. Get rid of accessors that don't help
> anyone. On the mapreduce side, serialization is now handled using json
> (jackson is a dependency anyway) instead of the hand-rolled regex we used
> before. Also do away with custom InputSplit and FileFormat, instead using
> Text and NLineInputFormat. On the local mode side, combine 1 client and N
> clients into the same implementation. That implementation now uses an
> ExecutorService, so we can later decouple number of client workers from
> number of client tasks. Finally, drop a bunch of confusing local state,
> instead use the new TestOptions POJO as a parameter to static methods.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HBASE-10592) Refactor PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10592: - Attachment: HBASE-10592.00-0.98.patch [~jmspaggi] mind taking this for a spin on your real cluster? Here's a patch vs. 0.98, maybe it'll be easier to use. > Refactor PerformanceEvaluation tool > --- > > Key: HBASE-10592 > URL: https://issues.apache.org/jira/browse/HBASE-10592 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10592.00-0.98.patch, HBASE-10592.00.patch > > > PerfEval is kind of a mess. It's painful to add new features because the test > options are itemized and passed as parameters to internal methods. > Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode > is a chore because of it. > This patch refactors the tool. Options are now passed around to methods and > such as a POJO instead of one-by-one. Get rid of accessors that don't help > anyone. On the mapreduce side, serialization is now handled using json > (jackson is a dependency anyway) instead of the hand-rolled regex we used > before. Also do away with custom InputSplit and FileFormat, instead using > Text and NLineInputFormat. On the local mode side, combine 1 client and N > clients into the same implementation. That implementation now uses an > ExecutorService, so we can later decouple number of client workers from > number of client tasks. Finally, drop a bunch of confusing local state, > instead use the new TestOptions POJO as a parameter to static methods. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10592) Refactor PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10592: - Attachment: HBASE-10592.00.patch The EvaluationMapTask is designed to be overridden by a user-provided class, so technically this patch breaks API compatibility. Meaning, it shouldn't be applied to 0.96 or 0.98. However, it's only a testing tool so it won't break anyone's production code. This change makes the tool much easier to enhance, as I'll demonstrate on HBASE-9953, so I'd prefer to get the change applied to the previous branches anyway. However, it's up to you [~stack] and [~apurtell]. > Refactor PerformanceEvaluation tool > --- > > Key: HBASE-10592 > URL: https://issues.apache.org/jira/browse/HBASE-10592 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10592.00.patch > > > PerfEval is kind of a mess. It's painful to add new features because the test > options are itemized and passed as parameters to internal methods. > Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode > is a chore because of it. > This patch refactors the tool. Options are now passed around to methods and > such as a POJO instead of one-by-one. Get rid of accessors that don't help > anyone. On the mapreduce side, serialization is now handled using json > (jackson is a dependency anyway) instead of the hand-rolled regex we used > before. Also do away with custom InputSplit and FileFormat, instead using > Text and NLineInputFormat. On the local mode side, combine 1 client and N > clients into the same implementation. That implementation now uses an > ExecutorService, so we can later decouple number of client workers from > number of client tasks. Finally, drop a bunch of confusing local state, > instead use the new TestOptions POJO as a parameter to static methods. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10592) Refactor PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10592: - Status: Patch Available (was: Open) > Refactor PerformanceEvaluation tool > --- > > Key: HBASE-10592 > URL: https://issues.apache.org/jira/browse/HBASE-10592 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10592.00.patch > > > PerfEval is kind of a mess. It's painful to add new features because the test > options are itemized and passed as parameters to internal methods. > Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode > is a chore because of it. > This patch refactors the tool. Options are now passed around to methods and > such as a POJO instead of one-by-one. Get rid of accessors that don't help > anyone. On the mapreduce side, serialization is now handled using json > (jackson is a dependency anyway) instead of the hand-rolled regex we used > before. Also do away with custom InputSplit and FileFormat, instead using > Text and NLineInputFormat. On the local mode side, combine 1 client and N > clients into the same implementation. That implementation now uses an > ExecutorService, so we can later decouple number of client workers from > number of client tasks. Finally, drop a bunch of confusing local state, > instead use the new TestOptions POJO as a parameter to static methods. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HBASE-10592) Refactor PerformanceEvaluation tool
Nick Dimiduk created HBASE-10592: Summary: Refactor PerformanceEvaluation tool Key: HBASE-10592 URL: https://issues.apache.org/jira/browse/HBASE-10592 Project: HBase Issue Type: Improvement Components: test Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Fix For: 0.99.0 PerfEval is kind of a mess. It's painful to add new features because the test options are itemized and passed as parameters to internal methods. Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode is a chore because of it. This patch refactors the tool. Options are now passed around to methods and such as a POJO instead of one-by-one. Get rid of accessors that don't help anyone. On the mapreduce side, serialization is now handled using json (jackson is a dependency anyway) instead of the hand-rolled regex we used before. Also do away with custom InputSplit and FileFormat, instead using Text and NLineInputFormat. On the local mode side, combine 1 client and N clients into the same implementation. That implementation now uses an ExecutorService, so we can later decouple number of client workers from number of client tasks. Finally, drop a bunch of confusing local state, instead use the new TestOptions POJO as a parameter to static methods. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10584) Inconsistency between tableExists and listTables in implementation
[
https://issues.apache.org/jira/browse/HBASE-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-10584:
--
Attachment: HBASE-10584-trunk_v1.patch
> Inconsistency between tableExists and listTables in implementation
> --
>
> Key: HBASE-10584
> URL: https://issues.apache.org/jira/browse/HBASE-10584
> Project: HBase
> Issue Type: Bug
> Components: Client, master
>Reporter: Feng Honghua
>Assignee: Feng Honghua
> Attachments: HBASE-10584-trunk_v1.patch, HBASE-10584-trunk_v1.patch
>
>
> # HBaseAdmin.tableExists is implemented by scanning meta table
> # HBaseAdmin.listTables(and HBaseAdmin.getTableDescriptor) is implemented by
> talking with HMaster which responses by querying the FSTableDescriptors, and
> FSTableDescriptors return all tables by scanning all the table descriptor
> files in FS(cache also plays here, so most of time it can be satisfied by
> cache)...
> Actually HBaseAdmin requests HMaster to check if a table exists internally
> when implementing deleteTable(see below), then why does it use a
> different(scanning meta table) way to implementing tableExists() for outside
> user to use for the same purpose?
> {code}
> tableExists = false;
> GetTableDescriptorsResponse htds;
> MasterKeepAliveConnection master = connection.getKeepAliveMasterService();
> try {
> GetTableDescriptorsRequest req =
> RequestConverter.buildGetTableDescriptorsRequest(tableName);
> htds = master.getTableDescriptors(null, req);
> } catch (ServiceException se) {
> throw ProtobufUtil.getRemoteException(se);
> } finally {
> master.close();
> }
> tableExists = !htds.getTableSchemaList().isEmpty();
> {code}
> (Above verifying that table descriptor file is deleted can guarantee all
> items of this table are deleted from meta table...)
> Since creating table descriptor files and inserting item to meta table occur
> in different time without atomic semantic, this inconsistency in
> implementation can lead to confusing behaviors when create-table or
> delete-table fails midway, (before according cleanup is done) table
> descriptor file may exists while no item exists in meta table (for
> create-table where table descriptor file is created before inserting item to
> meta table), this leads to listTables including that table, while tableExists
> says no. Similar inconsistency if delete-table fails mid-way...
> Confusing behavior can happen during the process even though eventually it
> succeed:
> # During table creation, when a user calls listTables and then calls
> tableExists for this table after the table descriptor is created but before
> item is inserted to meta table. He will find the listTables includes a table
> but tableExists return false for that same table, this behavior is confusing
> and should only acceptable during the table is being deleted...
> # Similar behavior occurs during table deletion.
> Seems the benefit of implementing tableExists this way is we can avoid
> talking with HMaster, considering we talk with HMaster for listTables and
> getTableDescriptor, such benefit can't offset the drawback from inconsistency.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10529) Make Cell extend Cloneable
[ https://issues.apache.org/jira/browse/HBASE-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909428#comment-13909428 ] ramkrishna.s.vasudevan commented on HBASE-10529: bq. I guess I'm stuck on the fact that the memstore is deeply intertwined with KeyValue right now. That was for the above stmt from Matt. So was saying may be use same keyvalues in the memstore and use Cells only on the server side. > Make Cell extend Cloneable > -- > > Key: HBASE-10529 > URL: https://issues.apache.org/jira/browse/HBASE-10529 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0 > > Attachments: HBSE-10529.patch > > > Refer to the parent JIRA for discussion on making extending Cloneable. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10589) Reduce unnecessary TestRowProcessorEndpoint resource usage
[ https://issues.apache.org/jira/browse/HBASE-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909357#comment-13909357 ] Hudson commented on HBASE-10589: FAILURE: Integrated in hbase-0.96-hadoop2 #212 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/212/]) HBASE-10589 Reduce unnecessary TestRowProcessorEndpoint resource usage (apurtell: rev 1570768) * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.java > Reduce unnecessary TestRowProcessorEndpoint resource usage > -- > > Key: HBASE-10589 > URL: https://issues.apache.org/jira/browse/HBASE-10589 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10589.patch > > > We don't need 1000 concurrent threads when 100 will do. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10580) IntegrationTestingUtility#restoreCluster leak resource when running in a mini cluster mode
[ https://issues.apache.org/jira/browse/HBASE-10580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909358#comment-13909358 ] Hudson commented on HBASE-10580: FAILURE: Integrated in hbase-0.96-hadoop2 #212 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/212/]) HBASE-10580: IntegrationTestingUtility#restoreCluster leak resource when running in a mini cluster mode (jeffreyz: rev 1570774) * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestingUtility.java > IntegrationTestingUtility#restoreCluster leak resource when running in a mini > cluster mode > -- > > Key: HBASE-10580 > URL: https://issues.apache.org/jira/browse/HBASE-10580 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.96.1, 0.99.0 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: hbase-10580.patch > > > When the utility isn't run in distributed cluster mode, the restore only > shutdown MiniHBaseCluster not MiniDFSCluster & MiniZKCluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10589) Reduce unnecessary TestRowProcessorEndpoint resource usage
[ https://issues.apache.org/jira/browse/HBASE-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909300#comment-13909300 ] Hudson commented on HBASE-10589: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #165 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/165/]) HBASE-10589 Reduce unnecessary TestRowProcessorEndpoint resource usage (apurtell: rev 1570766) * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.java > Reduce unnecessary TestRowProcessorEndpoint resource usage > -- > > Key: HBASE-10589 > URL: https://issues.apache.org/jira/browse/HBASE-10589 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10589.patch > > > We don't need 1000 concurrent threads when 100 will do. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10580) IntegrationTestingUtility#restoreCluster leak resource when running in a mini cluster mode
[ https://issues.apache.org/jira/browse/HBASE-10580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909301#comment-13909301 ] Hudson commented on HBASE-10580: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #165 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/165/]) HBASE-10580: IntegrationTestingUtility#restoreCluster leak resource when running in a mini cluster mode (jeffreyz: rev 1570770) * /hbase/branches/0.98/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestingUtility.java > IntegrationTestingUtility#restoreCluster leak resource when running in a mini > cluster mode > -- > > Key: HBASE-10580 > URL: https://issues.apache.org/jira/browse/HBASE-10580 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.98.0, 0.96.0, 0.96.1, 0.99.0 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: hbase-10580.patch > > > When the utility isn't run in distributed cluster mode, the restore only > shutdown MiniHBaseCluster not MiniDFSCluster & MiniZKCluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
