[jira] [Commented] (HBASE-6455) org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path
[ https://issues.apache.org/jira/browse/HBASE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422953#comment-13422953 ] stack commented on HBASE-6455: -- +1 on patch > org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path > as a child of input path > -- > > Key: HBASE-6455 > URL: https://issues.apache.org/jira/browse/HBASE-6455 > Project: HBase > Issue Type: Bug > Components: performance >Affects Versions: 0.94.0, 0.96.0 >Reporter: Aditya Kishore >Assignee: Aditya Kishore >Priority: Minor > Labels: benchmark > Fix For: 0.96.0 > > Attachments: HBASE-6455_trunk.patch > > > I was the running PerformanceEvaluation test with a modified job > configuration where the job output path is created before the splits and that > unmasked the issue because the the PeInputFormat.getSplits() function expects > to find only files under the input path. > The attached patch addresses both the issues. > # Creates separate input and output path rooted under a single folder e.g. > "/performance_evaluation//inputs" and > "/performance_evaluation//outputs", and > # The PeInputFormat.getSplits(), now skips any folder found under the input > path and process only files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6460) hbck "-repareHoles" shortcut doesn't enable "-fixHdfsOrphans"
[ https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6460: - Attachment: hbase-6460.patch A quick and simple fix in the patch file > hbck "-repareHoles" shortcut doesn't enable "-fixHdfsOrphans" > - > > Key: HBASE-6460 > URL: https://issues.apache.org/jira/browse/HBASE-6460 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.94.0, 0.96.0 >Reporter: Jie Huang >Priority: Minor > Attachments: hbase-6460.patch > > > According to the hbck's help info, shortcut - "-repairHoles" will enable > "-fixHdfsOrphans" as below. > {noformat} > -repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles > -fixHdfsOrphans > {noformat} > However, in the implementation, the function "fsck.setFixHdfsOrphans(false);" > is called in "-repairHoles". This is not consistent with the usage > information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6453) Hbase Replication point in time feature
[ https://issues.apache.org/jira/browse/HBASE-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422872#comment-13422872 ] terry zhang commented on HBASE-6453: Hiļ¼Stack. I think we can use point in time feature with Snapshots feature (HBASE-6055) in below case. 1. Master cluster in data center China do a Snapshot at Time A 2. copy the Snapshot to Slave cluster in data center US and Set Replication Time stamp to A 3. Restore snapshots (HBASE-6230 ) for Slave cluster and start replication. Them Slave cluster data will be as same as Master Cluster . And data is more safe and US user can visit Slave cluster for getting or scanning data to decrease the stress for China data center. Enable replication can not control the accurate time so may be it will lose some data or replicate some useless data. Mysql also has point in time/position feature in it replication Framework. It is very convenience for data center administrate to use. We can give some better name for this operation cause I am not good at naming ... > Hbase Replication point in time feature > --- > > Key: HBASE-6453 > URL: https://issues.apache.org/jira/browse/HBASE-6453 > Project: HBase > Issue Type: New Feature > Components: replication >Affects Versions: 0.94.0 >Reporter: terry zhang >Assignee: terry zhang > Attachments: hbase-6453-v1.patch > > > Now we can not control when hbase replication start to work. this patch > support we set a time stamp filter . All the row which is below this time > stamp will not be replicated. We also can delete and show this time stamp in > hbase shell if we want to change it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Summary: Pluggable compaction policies via coprocessors (was: Pluggable policy for smallestReadPoint in HRegion) > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengsheng Huang updated HBASE-6456: Attachment: HBASE-6456.patch Attached a patch for it > Export Utility throws ClassNotFoundException > > > Key: HBASE-6456 > URL: https://issues.apache.org/jira/browse/HBASE-6456 > Project: HBase > Issue Type: Bug >Reporter: Shengsheng Huang >Priority: Minor > Attachments: HBASE-6456.patch > > > The command I ran is as below: > "bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1" > And I got the following exceptions: > attempt_201207261322_0002_m_00_0, Status : FAILED > Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.au > ... > This exception can be resolved by adding hbase common jar into > HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. > We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6458) reuse BinaryComparator
binlijin created HBASE-6458: --- Summary: reuse BinaryComparator Key: HBASE-6458 URL: https://issues.apache.org/jira/browse/HBASE-6458 Project: HBase Issue Type: Improvement Affects Versions: 0.94.1 Reporter: binlijin Fix For: 0.94.2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6456) Export Utility throws ClassNotFoundException
Shengsheng Huang created HBASE-6456: --- Summary: Export Utility throws ClassNotFoundException Key: HBASE-6456 URL: https://issues.apache.org/jira/browse/HBASE-6456 Project: HBase Issue Type: Bug Reporter: Shengsheng Huang Priority: Minor The command I ran is as below: "bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1" And I got the following exceptions: attempt_201207261322_0002_m_00_0, Status : FAILED Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.au ... This exception can be resolved by adding hbase common jar into HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6460) hbck "-repareHoles" shortcut doesn't enable "-fixHdfsOrphans"
Jie Huang created HBASE-6460: Summary: hbck "-repareHoles" shortcut doesn't enable "-fixHdfsOrphans" Key: HBASE-6460 URL: https://issues.apache.org/jira/browse/HBASE-6460 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jie Huang Priority: Minor According to the hbck's help info, shortcut - "-repairHoles" will enable "-fixHdfsOrphans" as below. {noformat} -repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans {noformat} However, in the implementation, the function "fsck.setFixHdfsOrphans(false);" is called in "-repairHoles". This is not consistent with the usage information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable policy for smallestReadPoint in HRegion
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Attachment: 6427-notReady.txt Here's an initial idea for a patch. For this to work coprocessors need to have access to KeyValueScanner, StoreScanner, etc. So that is violating/leaking some of the previous abstractions. This patch would allow a coprocessor to actually produce a scanner for a flush or a compaction, while not requiring it to reimplement all the logic. A coprocessor can now in fact just override the TTL and/or # versions for a flush/compaction. Please let me know what you think and how to improve this. And for the love of god, please think of better names for the two new hooks. > Pluggable policy for smallestReadPoint in HRegion > - > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422971#comment-13422971 ] stack commented on HBASE-5705: -- All of above seems reasonable. Here is some response. bq. ...Could you please point me to the block of code in the patch where this is. I do not see it now. bq. ...and in the case of ProtobufRpcEngine, they are done using RpcRequestWritable (a thin wrapper over PB objects). The Writable wrapper goes away when we move to pb engine? bq. ...In that sense, we need the message size. Its header length only right? Thats ok I'd say. But for the value, when its a 50M cell, we won't have to read it into a buffer to find its size before sending will we when we are pb? (pb does this internally probably but we do we have to do it again outside of pb to put it in rpc header?) > Introduce Protocol Buffer RPC engine > > > Key: HBASE-5705 > URL: https://issues.apache.org/jira/browse/HBASE-5705 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch > > > Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB > aware can be made to go through this RPC engine. The approach, in my current > thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422956#comment-13422956 ] stack commented on HBASE-6431: -- @Alex What do you think of Ted patch? > Some FilterList Constructors break addFilter > > > Key: HBASE-6431 > URL: https://issues.apache.org/jira/browse/HBASE-6431 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.92.1, 0.94.0 >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Minor > Attachments: > 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt > > > Some of the constructors for FilterList set the internal list of filters to > list types which don't support the add operation. As a result > FilterList(final List rowFilters) > FilterList(final Filter... rowFilters) > FilterList(final Operator operator, final List rowFilters) > FilterList(final Operator operator, final Filter... rowFilters) > may init private List filters = new ArrayList(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422873#comment-13422873 ] Hadoop QA commented on HBASE-6431: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537948/6431-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestAssignmentManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//console This message is automatically generated. > Some FilterList Constructors break addFilter > > > Key: HBASE-6431 > URL: https://issues.apache.org/jira/browse/HBASE-6431 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.92.1, 0.94.0 >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Minor > Attachments: > 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt > > > Some of the constructors for FilterList set the internal list of filters to > list types which don't support the add operation. As a result > FilterList(final List rowFilters) > FilterList(final Filter... rowFilters) > FilterList(final Operator operator, final List rowFilters) > FilterList(final Operator operator, final Filter... rowFilters) > may init private List filters = new ArrayList(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-6458: Attachment: HBASE-6458-0.94.patch > reuse BinaryComparator > -- > > Key: HBASE-6458 > URL: https://issues.apache.org/jira/browse/HBASE-6458 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.1 >Reporter: binlijin > Fix For: 0.94.2 > > Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422946#comment-13422946 ] wanbin commented on HBASE-6457: --- By modifying the file (add partitionid as suffix) name may be able to repair the problem, > when use bulkload, hundreds of reduce write hfile, maybe create files of the > same name, as a result some data loss > --- > > Key: HBASE-6457 > URL: https://issues.apache.org/jira/browse/HBASE-6457 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.92.1 >Reporter: wanbin > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanbin updated HBASE-6457: -- Affects Version/s: 0.92.1 > when use bulkload, hundreds of reduce write hfile, maybe create files of the > same name, as a result some data loss > --- > > Key: HBASE-6457 > URL: https://issues.apache.org/jira/browse/HBASE-6457 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.92.1 >Reporter: wanbin > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
[ https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423001#comment-13423001 ] zhou wenjian commented on HBASE-6459: - I'll give a patch later > improve speed of create table > - > > Key: HBASE-6459 > URL: https://issues.apache.org/jira/browse/HBASE-6459 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: zhou wenjian > Fix For: 0.94.1 > > > in CreateTableHandler > for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) { > HRegionInfo newRegion = this.newRegions[regionIdx]; > // 1. Create HRegion > HRegion region = HRegion.createHRegion(newRegion, > this.fileSystemManager.getRootDir(), this.conf, > this.hTableDescriptor, null, false, true); > regionInfos.add(region.getRegionInfo()); > if (regionIdx % batchSize == 0) { > // 2. Insert into META > MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos); > regionInfos.clear(); > } > // 3. Close the new region to flush to disk. Close log file too. > region.close(); > } > All the region will be create serially. > If we have thousands of regions, that will be a huge cost. > We can improve it by create the region in parallel -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengsheng Huang updated HBASE-6372: Attachment: HBASE-6372.patch I think this is useful. Attached a patch for it. > Add scanner batching to Export job > -- > > Key: HBASE-6372 > URL: https://issues.apache.org/jira/browse/HBASE-6372 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Lars George >Priority: Minor > Labels: newbie > Attachments: HBASE-6372.patch > > > When a single row is too large for the RS heap then an OOME can take out the > entire RS. Setting scanner batching in custom scans helps avoiding this > scenario, but for the supplied Export job this is not set. > Similar to HBASE-3421 we can set the batching to a low number - or if needed > make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
wanbin created HBASE-6457: - Summary: when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss Key: HBASE-6457 URL: https://issues.apache.org/jira/browse/HBASE-6457 Project: HBase Issue Type: Bug Components: mapreduce Reporter: wanbin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5991) Introduce sequential ZNode based read/write locks
[ https://issues.apache.org/jira/browse/HBASE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422909#comment-13422909 ] Alex Feinberg commented on HBASE-5991: -- Mostly done in terms of implementing the locks itself based on the recipe (with recoverable zooKeeper). Should have this integrated into HMaster (in place of my DistributedLock code) and have a diff ready soon. > Introduce sequential ZNode based read/write locks > -- > > Key: HBASE-5991 > URL: https://issues.apache.org/jira/browse/HBASE-5991 > Project: HBase > Issue Type: Improvement >Reporter: Alex Feinberg >Assignee: Alex Feinberg > > This is a continuation of HBASE-5494: > Currently table-level write locks have been implemented using non-sequential > ZNodes as part of HBASE-5494 and committed to 89-fb branch. This issue is to > track converting the table-level locks to sequential ZNodes and supporting > read-write locks, as to solve the issue of preventing schema changes during > region splits or merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-6458: Attachment: HBASE-6458-0.94-2.patch > reuse BinaryComparator > -- > > Key: HBASE-6458 > URL: https://issues.apache.org/jira/browse/HBASE-6458 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.1 >Reporter: binlijin > Fix For: 0.94.2 > > Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6459) improve speed of create table
zhou wenjian created HBASE-6459: --- Summary: improve speed of create table Key: HBASE-6459 URL: https://issues.apache.org/jira/browse/HBASE-6459 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: zhou wenjian Fix For: 0.94.1 in CreateTableHandler for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) { HRegionInfo newRegion = this.newRegions[regionIdx]; // 1. Create HRegion HRegion region = HRegion.createHRegion(newRegion, this.fileSystemManager.getRootDir(), this.conf, this.hTableDescriptor, null, false, true); regionInfos.add(region.getRegionInfo()); if (regionIdx % batchSize == 0) { // 2. Insert into META MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos); regionInfos.clear(); } // 3. Close the new region to flush to disk. Close log file too. region.close(); } All the region will be create serially. If we have thousands of regions, that will be a huge cost. We can improve it by create the region in parallel -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6455) org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path
[ https://issues.apache.org/jira/browse/HBASE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423137#comment-13423137 ] Zhihong Ted Yu commented on HBASE-6455: --- Integrated to trunk. Thanks for the patch, Aditya. Thanks for the review, Stack. > org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path > as a child of input path > -- > > Key: HBASE-6455 > URL: https://issues.apache.org/jira/browse/HBASE-6455 > Project: HBase > Issue Type: Bug > Components: performance >Affects Versions: 0.94.0, 0.96.0 >Reporter: Aditya Kishore >Assignee: Aditya Kishore >Priority: Minor > Labels: benchmark > Fix For: 0.96.0 > > Attachments: HBASE-6455_trunk.patch > > > I was the running PerformanceEvaluation test with a modified job > configuration where the job output path is created before the splits and that > unmasked the issue because the the PeInputFormat.getSplits() function expects > to find only files under the input path. > The attached patch addresses both the issues. > # Creates separate input and output path rooted under a single folder e.g. > "/performance_evaluation//inputs" and > "/performance_evaluation//outputs", and > # The PeInputFormat.getSplits(), now skips any folder found under the input > path and process only files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423161#comment-13423161 ] Lars Hofhansl commented on HBASE-6458: -- Yeah, that's stupid. +1 > reuse BinaryComparator > -- > > Key: HBASE-6458 > URL: https://issues.apache.org/jira/browse/HBASE-6458 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.1 >Reporter: binlijin > Fix For: 0.94.2 > > Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423200#comment-13423200 ] Jimmy Xiang commented on HBASE-6272: Patch 6 was uploaded RB to address Ram's comments. Please review. > In-memory region state is inconsistent > -- > > Key: HBASE-6272 > URL: https://issues.apache.org/jira/browse/HBASE-6272 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > AssignmentManger stores region state related information in several places: > regionsInTransition, regions (region info to server name map), and servers > (server name to region info set map). However the access to these places is > not coordinated properly. It leads to inconsistent in-memory region state > information. Sometimes, some region could even be offline, and not in > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423201#comment-13423201 ] Lars Hofhansl commented on HBASE-5954: -- Thinking more about HTable#hsync, I think that would hard to make useful to an application. The application would need to do know which RegionServers to hsync the WAL on (unless we want to flush for a Table, which means all RegionServer hosting regions for this table need to hsync their WAL, and that does not seem to be useful). A similar argument goes for the RPCs (which will be split to multiple RegionServers). So #6 and #7 are out I think. I think Todd was right after all (just took me a long time to come around to it), a flag per Put/Delete/Increment/Append/etc, would be best option. > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 5954-trunk-hdfs-trunk-v2.txt, > 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, > 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, > 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423241#comment-13423241 ] Lars Hofhansl commented on HBASE-5954: -- Another question: HBase 0.96 now has the hadoop{1|2}-compat projects. On the other hand I do not want to have completely diverging implementations for this for 0.94 and 0.96 (which would mean to use reflection in both branches). Any thoughts on that? > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 5954-trunk-hdfs-trunk-v2.txt, > 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, > 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, > 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423256#comment-13423256 ] Alex Newman commented on HBASE-6431: Just looked at it. It's fine! > Some FilterList Constructors break addFilter > > > Key: HBASE-6431 > URL: https://issues.apache.org/jira/browse/HBASE-6431 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.92.1, 0.94.0 >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Minor > Attachments: > 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt > > > Some of the constructors for FilterList set the internal list of filters to > list types which don't support the add operation. As a result > FilterList(final List rowFilters) > FilterList(final Filter... rowFilters) > FilterList(final Operator operator, final List rowFilters) > FilterList(final Operator operator, final Filter... rowFilters) > may init private List filters = new ArrayList(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423272#comment-13423272 ] Andrew Purtell commented on HBASE-5954: --- bq. a flag per Put/Delete/Increment/Append/etc, would be best option Makes sense, since both you and Todd got here after giving it consideration. bq. On the other hand I do not want to have completely diverging implementations for this for 0.94 and 0.96 I think we have to take this pain, a reflection based strategy in 0.94 and a module based strategy in 0.96. I this case I'd judge it worth it. But that's going to be a high bar for other things. > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 5954-trunk-hdfs-trunk-v2.txt, > 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, > 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, > 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-6435: --- Attachment: 6435.v2.patch > Reading WAL files after a recovery leads to time lost in HDFS timeouts when > using dead datanodes > > > Key: HBASE-6435 > URL: https://issues.apache.org/jira/browse/HBASE-6435 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal > Attachments: 6435.unfinished.patch, 6435.v2.patch > > > HBase writes a Write-Ahead-Log to revover from hardware failure. > This log is written with 'append' on hdfs. > Through ZooKeeper, HBase gets informed usually in 30s that it should start > the recovery process. > This means reading the Write-Ahead-Log to replay the edits on the other > servers. > In standards deployments, HBase process (regionserver) are deployed on the > same box as the datanodes. > It means that when the box stops, we've actually lost one of the edits, as we > lost both the regionserver and the datanode. > As HDFS marks a node as dead after ~10 minutes, it appears as available when > we try to read the blocks to recover. As such, we are delaying the recovery > process by 60 seconds as the read will usually fail with a socket timeout. If > the file is still opened for writing, it adds an extra 20s + a risk of losing > edits if we connect with ipc to the dead DN. > Possible solutions are: > - shorter dead datanodes detection by the NN. Requires a NN code change. > - better dead datanodes management in DFSClient. Requires a DFS code change. > - NN customisation to write the WAL files on another DN instead of the local > one. > - reordering the blocks returned by the NN on the client side to put the > blocks on the same DN as the dead RS at the end of the priority queue. > Requires a DFS code change or a kind of workaround. > The solution retained is the last one. Compared to what was discussed on the > mailing list, the proposed patch will not modify HDFS source code but adds a > proxy. This for two reasons: > - Some HDFS functions managing block orders are static > (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would > require to implement partially the fix, change the DFS interface to make this > function non static, or put the hook static. None of these solution is very > clean. > - Adding a proxy allows to put all the code in HBase, simplifying dependency > management. > Nevertheless, it would be better to have this in HDFS. But this solution > allows to target the last version only, and this could allow minimal > interface changes such as non static methods. > Moreover, writing the blocks to the non local DN would be an even better > solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423278#comment-13423278 ] stack commented on HBASE-6272: -- I'm up for committing this to trunk. What about you Ram? > In-memory region state is inconsistent > -- > > Key: HBASE-6272 > URL: https://issues.apache.org/jira/browse/HBASE-6272 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > AssignmentManger stores region state related information in several places: > regionsInTransition, regions (region info to server name map), and servers > (server name to region info set map). However the access to these places is > not coordinated properly. It leads to inconsistent in-memory region state > information. Sometimes, some region could even be offline, and not in > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423279#comment-13423279 ] Zhihong Ted Yu commented on HBASE-6431: --- Will integrate in the afternoon if there is no objection. > Some FilterList Constructors break addFilter > > > Key: HBASE-6431 > URL: https://issues.apache.org/jira/browse/HBASE-6431 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.92.1, 0.94.0 >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Minor > Attachments: > 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt > > > Some of the constructors for FilterList set the internal list of filters to > list types which don't support the add operation. As a result > FilterList(final List rowFilters) > FilterList(final Filter... rowFilters) > FilterList(final Operator operator, final List rowFilters) > FilterList(final Operator operator, final Filter... rowFilters) > may init private List filters = new ArrayList(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423280#comment-13423280 ] nkeywal commented on HBASE-6435: v2. May need some clean up on logs + a check to unactivate it for hadoop 2 for example. > Reading WAL files after a recovery leads to time lost in HDFS timeouts when > using dead datanodes > > > Key: HBASE-6435 > URL: https://issues.apache.org/jira/browse/HBASE-6435 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal > Attachments: 6435.unfinished.patch, 6435.v2.patch > > > HBase writes a Write-Ahead-Log to revover from hardware failure. > This log is written with 'append' on hdfs. > Through ZooKeeper, HBase gets informed usually in 30s that it should start > the recovery process. > This means reading the Write-Ahead-Log to replay the edits on the other > servers. > In standards deployments, HBase process (regionserver) are deployed on the > same box as the datanodes. > It means that when the box stops, we've actually lost one of the edits, as we > lost both the regionserver and the datanode. > As HDFS marks a node as dead after ~10 minutes, it appears as available when > we try to read the blocks to recover. As such, we are delaying the recovery > process by 60 seconds as the read will usually fail with a socket timeout. If > the file is still opened for writing, it adds an extra 20s + a risk of losing > edits if we connect with ipc to the dead DN. > Possible solutions are: > - shorter dead datanodes detection by the NN. Requires a NN code change. > - better dead datanodes management in DFSClient. Requires a DFS code change. > - NN customisation to write the WAL files on another DN instead of the local > one. > - reordering the blocks returned by the NN on the client side to put the > blocks on the same DN as the dead RS at the end of the priority queue. > Requires a DFS code change or a kind of workaround. > The solution retained is the last one. Compared to what was discussed on the > mailing list, the proposed patch will not modify HDFS source code but adds a > proxy. This for two reasons: > - Some HDFS functions managing block orders are static > (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would > require to implement partially the fix, change the DFS interface to make this > function non static, or put the hook static. None of these solution is very > clean. > - Adding a proxy allows to put all the code in HBase, simplifying dependency > management. > Nevertheless, it would be better to have this in HDFS. But this solution > allows to target the last version only, and this could allow minimal > interface changes such as non static methods. > Moreover, writing the blocks to the non local DN would be an even better > solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423282#comment-13423282 ] nkeywal commented on HBASE-6435: + I need to test it on a real cluster (emulating locations on a mini cluster can be dangerous...) > Reading WAL files after a recovery leads to time lost in HDFS timeouts when > using dead datanodes > > > Key: HBASE-6435 > URL: https://issues.apache.org/jira/browse/HBASE-6435 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal > Attachments: 6435.unfinished.patch, 6435.v2.patch > > > HBase writes a Write-Ahead-Log to revover from hardware failure. > This log is written with 'append' on hdfs. > Through ZooKeeper, HBase gets informed usually in 30s that it should start > the recovery process. > This means reading the Write-Ahead-Log to replay the edits on the other > servers. > In standards deployments, HBase process (regionserver) are deployed on the > same box as the datanodes. > It means that when the box stops, we've actually lost one of the edits, as we > lost both the regionserver and the datanode. > As HDFS marks a node as dead after ~10 minutes, it appears as available when > we try to read the blocks to recover. As such, we are delaying the recovery > process by 60 seconds as the read will usually fail with a socket timeout. If > the file is still opened for writing, it adds an extra 20s + a risk of losing > edits if we connect with ipc to the dead DN. > Possible solutions are: > - shorter dead datanodes detection by the NN. Requires a NN code change. > - better dead datanodes management in DFSClient. Requires a DFS code change. > - NN customisation to write the WAL files on another DN instead of the local > one. > - reordering the blocks returned by the NN on the client side to put the > blocks on the same DN as the dead RS at the end of the priority queue. > Requires a DFS code change or a kind of workaround. > The solution retained is the last one. Compared to what was discussed on the > mailing list, the proposed patch will not modify HDFS source code but adds a > proxy. This for two reasons: > - Some HDFS functions managing block orders are static > (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would > require to implement partially the fix, change the DFS interface to make this > function non static, or put the hook static. None of these solution is very > clean. > - Adding a proxy allows to put all the code in HBase, simplifying dependency > management. > Nevertheless, it would be better to have this in HDFS. But this solution > allows to target the last version only, and this could allow minimal > interface changes such as non static methods. > Moreover, writing the blocks to the non local DN would be an even better > solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423275#comment-13423275 ] Andrew Purtell commented on HBASE-5954: --- More on my comment above. We have two options: we can start breaking out reflection into modules in 0.94 too, or save all of that for 0.96. I don't have a strong opinion but if I had to make a choice the refactoring should be in next major version / trunk / currently unstable. > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 5954-trunk-hdfs-trunk-v2.txt, > 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, > 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, > 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423291#comment-13423291 ] stack commented on HBASE-6458: -- +1 on version2. Want to make one for trunk that we can commit Binlijin? > reuse BinaryComparator > -- > > Key: HBASE-6458 > URL: https://issues.apache.org/jira/browse/HBASE-6458 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.1 >Reporter: binlijin > Fix For: 0.94.2 > > Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423296#comment-13423296 ] stack commented on HBASE-6431: -- Patch is good by me. > Some FilterList Constructors break addFilter > > > Key: HBASE-6431 > URL: https://issues.apache.org/jira/browse/HBASE-6431 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.92.1, 0.94.0 >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Minor > Attachments: > 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt > > > Some of the constructors for FilterList set the internal list of filters to > list types which don't support the add operation. As a result > FilterList(final List rowFilters) > FilterList(final Filter... rowFilters) > FilterList(final Operator operator, final List rowFilters) > FilterList(final Operator operator, final Filter... rowFilters) > may init private List filters = new ArrayList(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423299#comment-13423299 ] stack commented on HBASE-6458: -- Binlijin Please fix the title and add a description. As is, the subject is too general (what BinaryComparator?), etc. Thanks. > reuse BinaryComparator > -- > > Key: HBASE-6458 > URL: https://issues.apache.org/jira/browse/HBASE-6458 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.1 >Reporter: binlijin > Fix For: 0.94.2 > > Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423302#comment-13423302 ] Alex Newman commented on HBASE-6431: Oh I should mention I ran the patch through our jenkins and it looks good. > Some FilterList Constructors break addFilter > > > Key: HBASE-6431 > URL: https://issues.apache.org/jira/browse/HBASE-6431 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.92.1, 0.94.0 >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Minor > Attachments: > 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt > > > Some of the constructors for FilterList set the internal list of filters to > list types which don't support the add operation. As a result > FilterList(final List rowFilters) > FilterList(final Filter... rowFilters) > FilterList(final Operator operator, final List rowFilters) > FilterList(final Operator operator, final Filter... rowFilters) > may init private List filters = new ArrayList(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423305#comment-13423305 ] Andrew Purtell commented on HBASE-6427: --- Or use polymorphism? {noformat} + public InternalScanner preFlush(final ObserverContext c, + Store store, KeyValueScanner scanner) throws IOException; + + @Deprecated public void preFlush(ObserverContext e) throws IOException; {noformat} {noformat} + public InternalScanner preCompact(final ObserverContext c, + Store store, List scanners, ScanType scanType, long earliestPutTs) + throws IOException; + + @Deprecated public void preCompact(ObserverContext e ... {noformat} > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423311#comment-13423311 ] Devaraj Das commented on HBASE-5705: bq. The Writable wrapper goes away when we move to pb engine? Yes (AFAICT). bq. Its header length only right? Thats ok I'd say. But for the value, when its a 50M cell, we won't have to read it into a buffer to find its size before sending will we when we are pb? (pb does this internally probably but we do we have to do it again outside of pb to put it in rpc header?) No we don't have to write to a buffer in the case of PB objects. There is a getSerializedSize method on PB objects (and assuming that the 50M cell is encapsulated in the PB object as a byte-array, PB would simply call byte-array.length...). > Introduce Protocol Buffer RPC engine > > > Key: HBASE-5705 > URL: https://issues.apache.org/jira/browse/HBASE-5705 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch > > > Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB > aware can be made to go through this RPC engine. The approach, in my current > thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423310#comment-13423310 ] stack commented on HBASE-6457: -- Wanbin Do we have this issue in trunk? Can you put up sample name-clash? Thanks. > when use bulkload, hundreds of reduce write hfile, maybe create files of the > same name, as a result some data loss > --- > > Key: HBASE-6457 > URL: https://issues.apache.org/jira/browse/HBASE-6457 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.92.1 >Reporter: wanbin > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6372: - Status: Patch Available (was: Open) Looks good (Has some small formatting issues... will fix on application). Running by hadoopqa to make sure its happy. > Add scanner batching to Export job > -- > > Key: HBASE-6372 > URL: https://issues.apache.org/jira/browse/HBASE-6372 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Lars George >Priority: Minor > Labels: newbie > Attachments: HBASE-6372.patch > > > When a single row is too large for the RS heap then an OOME can take out the > entire RS. Setting scanner batching in custom scans helps avoiding this > scenario, but for the supplied Export job this is not set. > Similar to HBASE-3421 we can set the batching to a low number - or if needed > make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423320#comment-13423320 ] ramkrishna.s.vasudevan commented on HBASE-6272: --- I just added minor comments. They are just questions. If they are fine i am ok with commit. May be once we get in this, we can fix any bugs in future if any. Thanks Jimmy and Stack. > In-memory region state is inconsistent > -- > > Key: HBASE-6272 > URL: https://issues.apache.org/jira/browse/HBASE-6272 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > AssignmentManger stores region state related information in several places: > regionsInTransition, regions (region info to server name map), and servers > (server name to region info set map). However the access to these places is > not coordinated properly. It leads to inconsistent in-memory region state > information. Sometimes, some region could even be offline, and not in > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423325#comment-13423325 ] Lars Hofhansl commented on HBASE-6427: -- Uh... I like that. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423331#comment-13423331 ] stack commented on HBASE-5705: -- I'm going to commit this in next day unless objection. > Introduce Protocol Buffer RPC engine > > > Key: HBASE-5705 > URL: https://issues.apache.org/jira/browse/HBASE-5705 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch > > > Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB > aware can be made to go through this RPC engine. The approach, in my current > thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423332#comment-13423332 ] stack commented on HBASE-6272: -- Jimmy, if you've addressed Ram's minor comments, commit I'd say. > In-memory region state is inconsistent > -- > > Key: HBASE-6272 > URL: https://issues.apache.org/jira/browse/HBASE-6272 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > AssignmentManger stores region state related information in several places: > regionsInTransition, regions (region info to server name map), and servers > (server name to region info set map). However the access to these places is > not coordinated properly. It leads to inconsistent in-memory region state > information. Sometimes, some region could even be offline, and not in > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Baranau updated HBASE-6411: Attachment: HBASE-6411-3.patch Tried to fix things we mentioned about. Created review request: https://reviews.apache.org/r/6161 Removed those incr() methods we were willing to (it turned out that they were only in hadoop 1.0 metrics2 lib, and were removed in hadoop 2.0, so we are ok removing it and no need to ask hadoop guys about it). Switched to using concurrent collections inside metric registries. It turned out that "synchronize-guarded" methods where only in hadoop 2.0, not in hadoop 1.0. In both cases replaced them with working with concurrent collections. Tried to remove throwing MetricsExceptions, but there are still cases where I throw them. Not sure what is the best way to avoid this in those particular cases. Advice is much appreciated. Two Qs remained: 1. what about old MasterMetrics? Are we sure we want to replace it? I though we want to leave it for some time 2. my Q from the comment above related to metricsSources interfaces - not critical, can be addressed later, I guess. > Move Master Metrics to metrics 2 > > > Key: HBASE-6411 > URL: https://issues.apache.org/jira/browse/HBASE-6411 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, > HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch > > > Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423348#comment-13423348 ] Elliott Clark commented on HBASE-6411: -- bq.what about old MasterMetrics? Are we sure we want to replace it? I though we want to leave it for some time The class is still there so we're not breaking compat. All that we're doing is changing which metrics system exports to file/jmx/ganglia. From the conversations I've had it seems like people are fine with changing to the new metrics system as the backing as long as we don't remove classes that could be used externally. eg. we'll have to leave the current implementation of MetricsHistorgram as it's possible that someone is using it outside of hbase internal code. (That's all just my understanding and if someone disagree's that's fine too.) bq.Should we probably unify that? E.g. push managing "sink.ageOfLastAppliedOp" metrics and such to xxxMetricsSource? Stuffing dealing with the replication source uses dynamic metric names so it relies on the more general interface and since we wanted those metrics under the same node in jmx, I think the way we have it is fine for now; or at least good enough to leave it as is until after your patch gets applied. > Move Master Metrics to metrics 2 > > > Key: HBASE-6411 > URL: https://issues.apache.org/jira/browse/HBASE-6411 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, > HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch > > > Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423351#comment-13423351 ] Jean-Daniel Cryans commented on HBASE-2214: --- Following this discussion on user@ http://permalink.gmane.org/gmane.comp.java.hadoop.hbase.user/27936 If the max scanner result size is only set on the region server it's possible miss rows. Even with the latest patch in this jira it's still possible since the client doesn't tell the server it's going to use the default instead of whatever the server is using. Maybe the {{ClientScanner}} could modify the {{Scan}} that it passes the max scanner result it uses to the region server > Do HBASE-1996 -- setting size to return in scan rather than count of rows -- > properly > - > > Key: HBASE-2214 > URL: https://issues.apache.org/jira/browse/HBASE-2214 > Project: HBase > Issue Type: New Feature >Reporter: stack > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt, > HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, > HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt > > > The notion that you set size rather than row count specifying how many rows a > scanner should return in each cycle was raised over in hbase-1966. Its a > good one making hbase "regular" though the data under it may vary. > HBase-1966 was committed but the patch was constrained by the fact that it > needed to not change RPC interface. This issue is about doing hbase-1966 for > 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Attachment: 6427-v1.txt New patch > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423365#comment-13423365 ] Lars Hofhansl commented on HBASE-6427: -- The part I still have think through is how to handle actual use scans. Be default a user scan will also filter TTL/Versions, so it's one thing to prevent the KVs from being compacted away and another to actually make them visible to user scans. A similar approach can be followed in preScannerOpen, as long as the coprocessor has enough access to internal region data structure to rebuild the default scanner. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423367#comment-13423367 ] Andrew Purtell commented on HBASE-6427: --- BaseRegionObserver should reimplement the default behavior in the new methods? Anybody who inherits would get it. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423368#comment-13423368 ] Andrew Purtell commented on HBASE-6427: --- Or, better yet, BaseRegionObserver calls out to a Store static method that does it, with some javadoc to make it clear what's going on? > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423370#comment-13423370 ] Jimmy Xiang commented on HBASE-6272: Thanks for review. I will commit it to trunk tomorrow if no objection. > In-memory region state is inconsistent > -- > > Key: HBASE-6272 > URL: https://issues.apache.org/jira/browse/HBASE-6272 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > > AssignmentManger stores region state related information in several places: > regionsInTransition, regions (region info to server name map), and servers > (server name to region info set map). However the access to these places is > not coordinated properly. It leads to inconsistent in-memory region state > information. Sometimes, some region could even be offline, and not in > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423372#comment-13423372 ] Lars Hofhansl commented on HBASE-6427: -- Hmm... The default behavior is followed when RegionObserver.pre{flush|compact} return a null scanner, which is what BaseRegionObserver does by default. BaseRegionObserver implementing the default behavior would not really buy anything (unless I am missing something). As for last comment above, I think we'd need a preStoreScannerOpen, which would be called in Store.getScanner (right before the new StoreScanner is created) to allow the coprocessor to return a custom scanner here too. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423373#comment-13423373 ] Alex Baranau commented on HBASE-6411: - Here's why I'm concerned about rewriting existing MasterMetrics class: as a part of "revising metrics" we are likely to change existing metrics (incl. names that we expose, esp. dependent on old stuff like MetricsTimeVaryingRate). Though we now just copied in the initial patch all metrics from old master metrics, they may change. That's why in my first version I just added "empty placeholder" class with single "example" metric (cluster_requests), so that after we revise existing we can fill in more (and they are likely to be different from the old Master/RegionServer metrics). I guess this was discussed a bit in HBASE-4050. May be that is more true for RegionServer metrics though where we have a lot of such old metrics types we don't want to support anymore. But I guess the approach should be the same w.r.t. "overwrite or add new metrics class". Thanx for your answers! > Move Master Metrics to metrics 2 > > > Key: HBASE-6411 > URL: https://issues.apache.org/jira/browse/HBASE-6411 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, > HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch > > > Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423374#comment-13423374 ] Andrew Purtell commented on HBASE-6427: --- bq. The default behavior is followed when RegionObserver.pre{flush|compact} return a null scanner, which is what BaseRegionObserver does by default. Fine, I was misled by the unit test code. bq. I think we'd need a preStoreScannerOpen, which would be called in Store.getScanner (right before the new StoreScanner is created) to allow the coprocessor to return a custom scanner here too. Sounds good to me. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423384#comment-13423384 ] Lars Hofhansl commented on HBASE-6427: -- Of course this has to be compared to simply making the ScanInfo pluggable in Store.java. What I want to achieve here is to have an external process (backup tool, transaction engine, etc) to be able to override HBase's default TTL/#Versions with very high fidelity (i.e. not via a dynamic schema change, which is too heavyweight/slow). The coprocessor approach is nice, because it provides a lot of flexibility for other future use cases and it does not invent a new concept. At the same time it adds complexity. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423387#comment-13423387 ] Andrew Purtell commented on HBASE-6427: --- bq. The coprocessor approach is nice, because it provides a lot of flexibility for other future use cases and it does not invent a new concept. At the same time it adds complexity. On balance the API change here is nice because it extends something that was too limited to address your use case such that now it works for you, and it also admits the possibility of others. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423392#comment-13423392 ] Lars Hofhansl commented on HBASE-2214: -- @Ferdy, are you sure you do not want to work on this anymore? I'll change Scan.java to be able to deal with two different version, so we can get this in 0.94 (admittedly, that is a small change only). bq. Maybe the ClientScanner could modify the Scan that it passes the max scanner result it uses to the region server Of course the requires the server to actually trust the client. Although I think with a rogue client it is quite easy to kill/mess-with a RegionServer in many other areas. > Do HBASE-1996 -- setting size to return in scan rather than count of rows -- > properly > - > > Key: HBASE-2214 > URL: https://issues.apache.org/jira/browse/HBASE-2214 > Project: HBase > Issue Type: New Feature >Reporter: stack > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt, > HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, > HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt > > > The notion that you set size rather than row count specifying how many rows a > scanner should return in each cycle was raised over in hbase-1966. Its a > good one making hbase "regular" though the data under it may vary. > HBase-1966 was committed but the patch was constrained by the fact that it > needed to not change RPC interface. This issue is about doing hbase-1966 for > 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6372: -- Affects Version/s: 0.94.2 0.96.0 Targeting for trunk and 0.94 if you don't mind [~lhofhansl] and [~stack] > Add scanner batching to Export job > -- > > Key: HBASE-6372 > URL: https://issues.apache.org/jira/browse/HBASE-6372 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 0.96.0, 0.94.2 >Reporter: Lars George >Priority: Minor > Labels: newbie > Attachments: HBASE-6372.patch > > > When a single row is too large for the RS heap then an OOME can take out the > entire RS. Setting scanner batching in custom scans helps avoiding this > scenario, but for the supplied Export job this is not set. > Similar to HBASE-3421 we can set the batching to a low number - or if needed > make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423399#comment-13423399 ] Elliott Clark commented on HBASE-6411: -- You're 100% correct we are for sure going to add, remove, and rename metrics; these changes are potentially a big deal. However, 0.96 is the singularity release so now's the time. Lars wanted a big metrics clean up so we might as well get started on that while moving over to metrics 2. > Move Master Metrics to metrics 2 > > > Key: HBASE-6411 > URL: https://issues.apache.org/jira/browse/HBASE-6411 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, > HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch > > > Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423406#comment-13423406 ] Elliott Clark commented on HBASE-6411: -- Thanks so much for all your work on the patch. Functionally it looks great. I'm +1 > Move Master Metrics to metrics 2 > > > Key: HBASE-6411 > URL: https://issues.apache.org/jira/browse/HBASE-6411 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, > HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch > > > Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
Elliott Clark created HBASE-6461: Summary: Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table. Key: HBASE-6461 URL: https://issues.apache.org/jira/browse/HBASE-6461 Project: HBase Issue Type: Bug Environment: hadoop-0.20.2-cdh3u3 HBase 0.94.1 RC1 Reporter: Elliott Clark Spun up a new dfs on hadoop-0.20.2-cdh3u3 Started hbase started running loadtest tool. killed rs and dn holding root with killall -9 java on server sv4r27s44 at about 2012-07-25 22:40:00 After things stabilize Root is in a bad state. Ran hbck and got: Exception in thread "main" org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in -ROOT- for region .META.,,1.1028785192 containing row at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232) at org.apache.hadoop.hbase.client.HTable.(HTable.java:172) at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236) hbase(main):001:0> scan '-ROOT-' ROW COLUMN+CELL 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing. .META.,,1column=info:regioninfo, timestamp=1343255838525, value={NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} .META.,,1column=info:v, timestamp=1343255838525, value=\x00\x00 1 row(s) in 0.5930 seconds Here's the master log: https://gist.github.com/3179194 I tried the same thing with 0.92.1 and I was able to get into a similar situation, so I don't think this is anything new. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423426#comment-13423426 ] Jonathan Leavitt commented on HBASE-6449: - @zhihong Do you think it is best to add it as a module as opposed to keeping it completely independent, and just having hbase depend on it? > Dapper like tracing > --- > > Key: HBASE-6449 > URL: https://issues.apache.org/jira/browse/HBASE-6449 > Project: HBase > Issue Type: New Feature > Components: client, ipc >Affects Versions: 0.96.0 >Reporter: Jonathan Leavitt > Labels: tracing > Attachments: htrace1.diff, trace.png > > > Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to > HBase. [Accumulo|http://accumulo.apache.org] added something similar with > their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423431#comment-13423431 ] Zhihong Ted Yu commented on HBASE-6449: --- A completely independent project is good to have but requires more logistics. We can refine the htrace module using this JIRA so that HBase is able to use it. Then we can propose the promotion of this feature as an independent project. > Dapper like tracing > --- > > Key: HBASE-6449 > URL: https://issues.apache.org/jira/browse/HBASE-6449 > Project: HBase > Issue Type: New Feature > Components: client, ipc >Affects Versions: 0.96.0 >Reporter: Jonathan Leavitt > Labels: tracing > Attachments: htrace1.diff, trace.png > > > Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to > HBase. [Accumulo|http://accumulo.apache.org] added something similar with > their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6455) org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path
[ https://issues.apache.org/jira/browse/HBASE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423444#comment-13423444 ] Hudson commented on HBASE-6455: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #111 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/111/]) HBASE-6455 org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path (Aditya) (Revision 1366034) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path > as a child of input path > -- > > Key: HBASE-6455 > URL: https://issues.apache.org/jira/browse/HBASE-6455 > Project: HBase > Issue Type: Bug > Components: performance >Affects Versions: 0.94.0, 0.96.0 >Reporter: Aditya Kishore >Assignee: Aditya Kishore >Priority: Minor > Labels: benchmark > Fix For: 0.96.0 > > Attachments: HBASE-6455_trunk.patch > > > I was the running PerformanceEvaluation test with a modified job > configuration where the job output path is created before the splits and that > unmasked the issue because the the PeInputFormat.getSplits() function expects > to find only files under the input path. > The attached patch addresses both the issues. > # Creates separate input and output path rooted under a single folder e.g. > "/performance_evaluation//inputs" and > "/performance_evaluation//outputs", and > # The PeInputFormat.getSplits(), now skips any folder found under the input > path and process only files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Attachment: 6427-v2.txt Add preStoreScannerOpen(...) to RegionObserver and related classes. Runs all of TestFromClientSide and TestCompaction with such a coprocessor. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Status: Patch Available (was: Open) > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-6427: Assignee: Lars Hofhansl > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423446#comment-13423446 ] Lars Hofhansl commented on HBASE-6372: -- 0.94 sounds good to me. > Add scanner batching to Export job > -- > > Key: HBASE-6372 > URL: https://issues.apache.org/jira/browse/HBASE-6372 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 0.96.0, 0.94.2 >Reporter: Lars George >Priority: Minor > Labels: newbie > Attachments: HBASE-6372.patch > > > When a single row is too large for the RS heap then an OOME can take out the > entire RS. Setting scanner batching in custom scans helps avoiding this > scenario, but for the supplied Export job this is not set. > Similar to HBASE-3421 we can set the batching to a low number - or if needed > make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423447#comment-13423447 ] Jonathan Leavitt commented on HBASE-6449: - Sounds good. I will submit a new patch today that deals with most of your and Todd's changes as well as making htrace a module of hbase. Let's just be careful going forward to ensure that the tracing library can easily be broken out when necessary. > Dapper like tracing > --- > > Key: HBASE-6449 > URL: https://issues.apache.org/jira/browse/HBASE-6449 > Project: HBase > Issue Type: New Feature > Components: client, ipc >Affects Versions: 0.96.0 >Reporter: Jonathan Leavitt > Labels: tracing > Attachments: htrace1.diff, trace.png > > > Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to > HBase. [Accumulo|http://accumulo.apache.org] added something similar with > their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6334) TestImprovement for TestHRegion.testWritesWhileGetting
[ https://issues.apache.org/jira/browse/HBASE-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-6334: -- Fix Version/s: 0.96.0 Adding 0.96.0 as this was committed to trunk. I also don't see this in the commit logs of 0.92. > TestImprovement for TestHRegion.testWritesWhileGetting > -- > > Key: HBASE-6334 > URL: https://issues.apache.org/jira/browse/HBASE-6334 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.1, 0.94.0, 0.96.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6334-92-v2.patch, HBASE-6334-94-v2.patch, > HBASE-6334-trunk-v2.patch, HBASE-6334-trunk.patch > > > Two improvements: > 1. I was running testWritesWhileGetting on 0.90.6 (which lacks HBASE-4195) > and I couldn't get the 20% failures mentioned in the JIRA. It was more like > 1%. So I increased the flush frequency and it failed most of the time. > 2. The check on failure is wrong: > {code} > assertEquals(previousKV.getValue(), thisValue); > {code} > Since this is a byte array, we should use Bytes.compareTo. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6334) TestImprovement for TestHRegion.testWritesWhileGetting
[ https://issues.apache.org/jira/browse/HBASE-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423453#comment-13423453 ] stack commented on HBASE-6334: -- Should be there now. Sorry about that Gregory. > TestImprovement for TestHRegion.testWritesWhileGetting > -- > > Key: HBASE-6334 > URL: https://issues.apache.org/jira/browse/HBASE-6334 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.1, 0.94.0, 0.96.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6334-92-v2.patch, HBASE-6334-94-v2.patch, > HBASE-6334-trunk-v2.patch, HBASE-6334-trunk.patch > > > Two improvements: > 1. I was running testWritesWhileGetting on 0.90.6 (which lacks HBASE-4195) > and I couldn't get the 20% failures mentioned in the JIRA. It was more like > 1%. So I increased the flush frequency and it failed most of the time. > 2. The check on failure is wrong: > {code} > assertEquals(previousKV.getValue(), thisValue); > {code} > Since this is a byte array, we should use Bytes.compareTo. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6462) TestAcidGuarantees failed on trunk
Enis Soztutar created HBASE-6462: Summary: TestAcidGuarantees failed on trunk Key: HBASE-6462 URL: https://issues.apache.org/jira/browse/HBASE-6462 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Enis Soztutar I've seen TestAcidGurantees fail with: {code} testGetAtomicity(org.apache.hadoop.hbase.IntegrationTestAcidGuaranteesWithChaosMonkey) Time elapsed: 42.611 sec <<< ERROR! java.lang.RuntimeException: Deferred at org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.checkException(MultithreadedTestUtil.java:76) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.stop(MultithreadedTestUtil.java:103) at org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:298) at org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:248) at org.apache.hadoop.hbase.IntegrationTestAcidGuaranteesWithChaosMonkey.testGetAtomicity(IntegrationTestAcidGuaranteesWithChaosMonkey.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47) at org.junit.rules.RunRules.evaluate(RunRules.java:18) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) Caused by: java.lang.RuntimeException: Failed after 147200!Expected=\x1BT\xC0i\x0CW\x9B\x108\xA0Got: test_row_0/A:col0/1343328428704/Put/vlen=10/ts=0 val= \x1BT\xC0i\x0CW\x9B\x108\xA0 test_row_0/A:col1/1343328428704/Put/vlen=10/ts=0 val= \x1BT\xC0i\x0CW\x9B\x108\xA0 test_row_0/A:col10/1343328428704/Put/vlen=10/ts=0 val= \x1BT\xC0i\x0CW\x9B\x108\xA0 ... test_row_0/B:col0/1343328425510/Put/vlen=10/ts=0 val= 4G\xE1T\x1B\xFDa\x98\xAC\xB6 test_row_0/B:col1/1343328425510/Put/vlen=10/ts=0 val= 4G\xE1T\x1B\xFDa\x98\xAC\xB6 test_row_0/B:col10/1343328425510/Put/vlen=10/ts=0 val= ... test_row_0/C:col0/1343328425510/Put/vlen=10/ts=0 val= 4G\xE1T\x1B\xFDa\x98\xAC\xB6 test_row_0/C:col1/1343328425510/Put/vlen=10/ts=0 val= 4G\xE1T\x1B\xFDa\x98\xAC\xB6 test_row_0/C:col10/1343328425510/Put/vlen=10/ts=0 val= {code} Might be related to HBASE-2856, but haven't had the time to check the root cause. The flusher thread was on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423454#comment-13423454 ] Jean-Daniel Cryans commented on HBASE-2214: --- bq. Of course the requires the server to actually trust the client. Isn't that what this jira does already? > Do HBASE-1996 -- setting size to return in scan rather than count of rows -- > properly > - > > Key: HBASE-2214 > URL: https://issues.apache.org/jira/browse/HBASE-2214 > Project: HBase > Issue Type: New Feature >Reporter: stack > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt, > HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, > HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt > > > The notion that you set size rather than row count specifying how many rows a > scanner should return in each cycle was raised over in hbase-1966. Its a > good one making hbase "regular" though the data under it may vary. > HBase-1966 was committed but the patch was constrained by the fact that it > needed to not change RPC interface. This issue is about doing hbase-1966 for > 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423458#comment-13423458 ] stack commented on HBASE-6449: -- bq. Do you think it is best to add it as a module as opposed to keeping it completely independent, and just having hbase depend on it? I think it best to have it as an independent project, at least at first (you don't need our blessing to do such a thing). Make a patch for hbase core that gives you the hooks you need and we'll commit that but better if all the rest can live outside especially as its being developed (no lag waiting on a committer to review and commit; just do it yourself out in github or wherever). What do you mean by having hbase dependent on htrace? > Dapper like tracing > --- > > Key: HBASE-6449 > URL: https://issues.apache.org/jira/browse/HBASE-6449 > Project: HBase > Issue Type: New Feature > Components: client, ipc >Affects Versions: 0.96.0 >Reporter: Jonathan Leavitt > Labels: tracing > Attachments: htrace1.diff, trace.png > > > Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to > HBase. [Accumulo|http://accumulo.apache.org] added something similar with > their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6431: -- Fix Version/s: 0.96.0 Integrated to trunk. Thanks for the patch, Alex. Thanks for the review, Stack. > Some FilterList Constructors break addFilter > > > Key: HBASE-6431 > URL: https://issues.apache.org/jira/browse/HBASE-6431 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.92.1, 0.94.0 >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Minor > Fix For: 0.96.0 > > Attachments: > 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt > > > Some of the constructors for FilterList set the internal list of filters to > list types which don't support the add operation. As a result > FilterList(final List rowFilters) > FilterList(final Filter... rowFilters) > FilterList(final Operator operator, final List rowFilters) > FilterList(final Operator operator, final Filter... rowFilters) > may init private List filters = new ArrayList(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423461#comment-13423461 ] Andrew Purtell commented on HBASE-6457: --- Our internal frankenBase uses MD5 calculated from time, hostname, and MAC address if available to name output HFiles. Seem reasonable? I can put up a patch. > when use bulkload, hundreds of reduce write hfile, maybe create files of the > same name, as a result some data loss > --- > > Key: HBASE-6457 > URL: https://issues.apache.org/jira/browse/HBASE-6457 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.92.1 >Reporter: wanbin > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423463#comment-13423463 ] Andrew Purtell commented on HBASE-6456: --- +1 seems helpful at best and harmless at worst. > Export Utility throws ClassNotFoundException > > > Key: HBASE-6456 > URL: https://issues.apache.org/jira/browse/HBASE-6456 > Project: HBase > Issue Type: Bug >Reporter: Shengsheng Huang >Priority: Minor > Attachments: HBASE-6456.patch > > > The command I ran is as below: > "bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1" > And I got the following exceptions: > attempt_201207261322_0002_m_00_0, Status : FAILED > Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.au > ... > This exception can be resolved by adding hbase common jar into > HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. > We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6456: -- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Export Utility throws ClassNotFoundException > > > Key: HBASE-6456 > URL: https://issues.apache.org/jira/browse/HBASE-6456 > Project: HBase > Issue Type: Bug >Reporter: Shengsheng Huang >Priority: Minor > Attachments: HBASE-6456.patch > > > The command I ran is as below: > "bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1" > And I got the following exceptions: > attempt_201207261322_0002_m_00_0, Status : FAILED > Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.au > ... > This exception can be resolved by adding hbase common jar into > HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. > We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu reassigned HBASE-6456: - Assignee: Shengsheng Huang > Export Utility throws ClassNotFoundException > > > Key: HBASE-6456 > URL: https://issues.apache.org/jira/browse/HBASE-6456 > Project: HBase > Issue Type: Bug >Reporter: Shengsheng Huang >Assignee: Shengsheng Huang >Priority: Minor > Attachments: HBASE-6456.patch > > > The command I ran is as below: > "bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1" > And I got the following exceptions: > attempt_201207261322_0002_m_00_0, Status : FAILED > Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.au > ... > This exception can be resolved by adding hbase common jar into > HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. > We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423469#comment-13423469 ] Andrew Purtell commented on HBASE-6427: --- lgtm, good tests > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
[ https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6461: - Priority: Critical (was: Major) Fix Version/s: 0.94.2 > Killing the HRegionServer and DataNode hosting ROOT can result in a malformed > root table. > - > > Key: HBASE-6461 > URL: https://issues.apache.org/jira/browse/HBASE-6461 > Project: HBase > Issue Type: Bug > Environment: hadoop-0.20.2-cdh3u3 > HBase 0.94.1 RC1 >Reporter: Elliott Clark >Priority: Critical > Fix For: 0.94.2 > > > Spun up a new dfs on hadoop-0.20.2-cdh3u3 > Started hbase > started running loadtest tool. > killed rs and dn holding root with killall -9 java on server sv4r27s44 at > about 2012-07-25 22:40:00 > After things stabilize Root is in a bad state. Ran hbck and got: > Exception in thread "main" > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in -ROOT- for region .META.,,1.1028785192 containing row > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810) > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:172) > at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236) > hbase(main):001:0> scan '-ROOT-' > ROW COLUMN+CELL > > > 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration > already set up for Hadoop, not re-installing. > .META.,,1column=info:regioninfo, > timestamp=1343255838525, value={NAME => '.META.,,1', STARTKEY => '', ENDKEY > => '', ENCODED => 1028785192,} > .META.,,1column=info:v, > timestamp=1343255838525, value=\x00\x00 > > 1 row(s) in 0.5930 seconds > Here's the master log: https://gist.github.com/3179194 > I tried the same thing with 0.92.1 and I was able to get into a similar > situation, so I don't think this is anything new. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
[ https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423474#comment-13423474 ] Lars Hofhansl commented on HBASE-6461: -- Marked Critical against 0.94.2. Could also see this as a blocker for 0.94.1 (Although 0.94.1 needs to go out at some point, because of HBASE-6311) > Killing the HRegionServer and DataNode hosting ROOT can result in a malformed > root table. > - > > Key: HBASE-6461 > URL: https://issues.apache.org/jira/browse/HBASE-6461 > Project: HBase > Issue Type: Bug > Environment: hadoop-0.20.2-cdh3u3 > HBase 0.94.1 RC1 >Reporter: Elliott Clark >Priority: Critical > Fix For: 0.94.2 > > > Spun up a new dfs on hadoop-0.20.2-cdh3u3 > Started hbase > started running loadtest tool. > killed rs and dn holding root with killall -9 java on server sv4r27s44 at > about 2012-07-25 22:40:00 > After things stabilize Root is in a bad state. Ran hbck and got: > Exception in thread "main" > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in -ROOT- for region .META.,,1.1028785192 containing row > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810) > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:172) > at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236) > hbase(main):001:0> scan '-ROOT-' > ROW COLUMN+CELL > > > 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration > already set up for Hadoop, not re-installing. > .META.,,1column=info:regioninfo, > timestamp=1343255838525, value={NAME => '.META.,,1', STARTKEY => '', ENDKEY > => '', ENCODED => 1028785192,} > .META.,,1column=info:v, > timestamp=1343255838525, value=\x00\x00 > > 1 row(s) in 0.5930 seconds > Here's the master log: https://gist.github.com/3179194 > I tried the same thing with 0.92.1 and I was able to get into a similar > situation, so I don't think this is anything new. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423476#comment-13423476 ] Zhihong Ted Yu commented on HBASE-6457: --- That would be nice, Andy. > when use bulkload, hundreds of reduce write hfile, maybe create files of the > same name, as a result some data loss > --- > > Key: HBASE-6457 > URL: https://issues.apache.org/jira/browse/HBASE-6457 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.92.1 >Reporter: wanbin > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6463) Support multiple memstore snapshots in order to support small/large flushes of cache.
Brian Nixon created HBASE-6463: -- Summary: Support multiple memstore snapshots in order to support small/large flushes of cache. Key: HBASE-6463 URL: https://issues.apache.org/jira/browse/HBASE-6463 Project: HBase Issue Type: Improvement Components: regionserver, util Affects Versions: 0.89-fb Reporter: Brian Nixon If cache is underutilized due to log size triggered flushes, should be able to buffer multiple snapshots in memory and flush all together into one file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423482#comment-13423482 ] Jonathan Leavitt commented on HBASE-6449: - bq. What do you mean by having hbase dependent on htrace? I meant having htrace as an independent maven project in the hbase source tree and add a for htrace in hbase's pom. The original plan was to get the tracing library committed to hbase first so the hbase committers could review and would be more likely to commit the necessary hooks into hbase. It would also save the initial hassle of getting the library into maven. But you would recommend putting it on github, and only submitting patches with the instrumentation for hbase? If htrace were on github and the hooks were in hbase, would those downloading hbase source just have to grab htrace off github and have maven install it locally? > Dapper like tracing > --- > > Key: HBASE-6449 > URL: https://issues.apache.org/jira/browse/HBASE-6449 > Project: HBase > Issue Type: New Feature > Components: client, ipc >Affects Versions: 0.96.0 >Reporter: Jonathan Leavitt > Labels: tracing > Attachments: htrace1.diff, trace.png > > > Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to > HBase. [Accumulo|http://accumulo.apache.org] added something similar with > their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423487#comment-13423487 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537954/HBASE-6372.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//console This message is automatically generated. > Add scanner batching to Export job > -- > > Key: HBASE-6372 > URL: https://issues.apache.org/jira/browse/HBASE-6372 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 0.96.0, 0.94.2 >Reporter: Lars George >Priority: Minor > Labels: newbie > Attachments: HBASE-6372.patch > > > When a single row is too large for the RS heap then an OOME can take out the > entire RS. Setting scanner batching in custom scans helps avoiding this > scenario, but for the supplied Export job this is not set. > Similar to HBASE-3421 we can set the batching to a low number - or if needed > make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423492#comment-13423492 ] Lars Hofhansl commented on HBASE-2214: -- Yep... Was just making a general observation. > Do HBASE-1996 -- setting size to return in scan rather than count of rows -- > properly > - > > Key: HBASE-2214 > URL: https://issues.apache.org/jira/browse/HBASE-2214 > Project: HBase > Issue Type: New Feature >Reporter: stack > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt, > HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, > HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt > > > The notion that you set size rather than row count specifying how many rows a > scanner should return in each cycle was raised over in hbase-1966. Its a > good one making hbase "regular" though the data under it may vary. > HBase-1966 was committed but the patch was constrained by the fact that it > needed to not change RPC interface. This issue is about doing hbase-1966 for > 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6462) TestAcidGuarantees failed on trunk
[ https://issues.apache.org/jira/browse/HBASE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423489#comment-13423489 ] Enis Soztutar commented on HBASE-6462: -- Another stack trace I got now has the 3rd CF with different values: {code} Caused by: java.lang.RuntimeException: Failed after 98200!Expected=\xFF[\x0B_\xAF\xCAQJ\xBDKGot: test_row_2/A:col0/1343337923715/Put/vlen=10/ts=0 val= \xFF[\x0B_\xAF\xCAQJ\xBDK test_row_2/A:col1/1343337923715/Put/vlen=10/ts=0 val= \xFF[\x0B_\xAF\xCAQJ\xBDK .. test_row_2/B:col8/1343337923715/Put/vlen=10/ts=0 val= \xFF[\x0B_\xAF\xCAQJ\xBDK test_row_2/B:col9/1343337923715/Put/vlen=10/ts=0 val= \xFF[\x0B_\xAF\xCAQJ\xBDK .. test_row_2/C:col0/1343337921472/Put/vlen=10/ts=0 val= \xEA\xD0\x15\xFB\xC0\xE7\xE3\xA0\xDB^ test_row_2/C:col1/1343337921472/Put/vlen=10/ts=0 val= \xEA\xD0\x15\xFB\xC0\xE7\xE3\xA0\xDB^ .. {code} > TestAcidGuarantees failed on trunk > -- > > Key: HBASE-6462 > URL: https://issues.apache.org/jira/browse/HBASE-6462 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Enis Soztutar > > I've seen TestAcidGurantees fail with: > {code} > testGetAtomicity(org.apache.hadoop.hbase.IntegrationTestAcidGuaranteesWithChaosMonkey) > Time elapsed: 42.611 sec <<< ERROR! > java.lang.RuntimeException: Deferred > at > org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.checkException(MultithreadedTestUtil.java:76) > at > org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.stop(MultithreadedTestUtil.java:103) > at > org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:298) > at > org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:248) > at > org.apache.hadoop.hbase.IntegrationTestAcidGuaranteesWithChaosMonkey.testGetAtomicity(IntegrationTestAcidGuaranteesWithChaosMonkey.java:58) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47) > at org.junit.rules.RunRules.evaluate(RunRules.java:18) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101) > at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) > Caused by: java.lang.RuntimeException: Failed after > 147200!Expected=\x1BT\xC0i\x0CW\x9B\x108\xA0Got: > test_row_0/A:col0/1343328428704/Put/vlen=10/ts=0 val= > \x1BT\xC0i\x0CW\x9B\x108\xA0 > test_row_0/A:c
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423501#comment-13423501 ] Hadoop QA commented on HBASE-6427: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538075/6427-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.catalog.TestMetaReaderEditor Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//console This message is automatically generated. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423506#comment-13423506 ] Jonathan Leavitt commented on HBASE-6449: - I updated the diff on [review board |https://reviews.apache.org/r/6127/] with many of Todd and Ted's suggestions. > Dapper like tracing > --- > > Key: HBASE-6449 > URL: https://issues.apache.org/jira/browse/HBASE-6449 > Project: HBase > Issue Type: New Feature > Components: client, ipc >Affects Versions: 0.96.0 >Reporter: Jonathan Leavitt > Labels: tracing > Attachments: htrace1.diff, trace.png > > > Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to > HBase. [Accumulo|http://accumulo.apache.org] added something similar with > their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2945) stop using string for block name in LRU block cache/hfile
[ https://issues.apache.org/jira/browse/HBASE-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-2945. -- Resolution: Duplicate This is a duplication of HBASE-5001 > stop using string for block name in LRU block cache/hfile > - > > Key: HBASE-2945 > URL: https://issues.apache.org/jira/browse/HBASE-2945 > Project: HBase > Issue Type: Bug > Components: performance >Affects Versions: 0.89.20100621 >Reporter: ryan rawson > > all my profiling runs indicate there is a large number of string/char[] > objects and string manipulation is taking a long time in profiling runs. > These come from the LRU block cache, block id. Also we should support the > eviction of blocks belonging to particular hfiles, and the current code > doesn't help with that right now. > Let's do something better. Whatever that might be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423508#comment-13423508 ] stack commented on HBASE-6449: -- bq. I meant having htrace as an independent maven project in the hbase source tree and add a for htrace in hbase's pom. But sounds like your intent is a generic tracing library going by statements above? Is that so? If that is the case, it seems odd dev'ing a generic lib inside in HBase core. You should do it as its own project I'd say; you can move faster and it'll be easier keep your generic focus. HBase project, IMO, will gladly be your guinea pig/proofing ground for your tracing lib work since we need tracing so bad. We can review/exercise your lib, we can add it as a dependency and make the necessary changes to pull it in and make it easy starting/stopping a trace, etc. but unless you think otherwise, lets keep the coupling loose. Correct me if I'm off Jonathan. > Dapper like tracing > --- > > Key: HBASE-6449 > URL: https://issues.apache.org/jira/browse/HBASE-6449 > Project: HBase > Issue Type: New Feature > Components: client, ipc >Affects Versions: 0.96.0 >Reporter: Jonathan Leavitt > Labels: tracing > Attachments: htrace1.diff, trace.png > > > Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to > HBase. [Accumulo|http://accumulo.apache.org] added something similar with > their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423509#comment-13423509 ] Hudson commented on HBASE-6431: --- Integrated in HBase-TRUNK #3173 (See [https://builds.apache.org/job/HBase-TRUNK/3173/]) HBASE-6431 Some FilterList Constructors break addFilter (Alex Newman) (Revision 1366198) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java > Some FilterList Constructors break addFilter > > > Key: HBASE-6431 > URL: https://issues.apache.org/jira/browse/HBASE-6431 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.92.1, 0.94.0 >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Minor > Fix For: 0.96.0 > > Attachments: > 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt > > > Some of the constructors for FilterList set the internal list of filters to > list types which don't support the add operation. As a result > FilterList(final List rowFilters) > FilterList(final Filter... rowFilters) > FilterList(final Operator operator, final List rowFilters) > FilterList(final Operator operator, final Filter... rowFilters) > may init private List filters = new ArrayList(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6455) org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path
[ https://issues.apache.org/jira/browse/HBASE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423510#comment-13423510 ] Hudson commented on HBASE-6455: --- Integrated in HBase-TRUNK #3173 (See [https://builds.apache.org/job/HBase-TRUNK/3173/]) HBASE-6455 org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path (Aditya) (Revision 1366034) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path > as a child of input path > -- > > Key: HBASE-6455 > URL: https://issues.apache.org/jira/browse/HBASE-6455 > Project: HBase > Issue Type: Bug > Components: performance >Affects Versions: 0.94.0, 0.96.0 >Reporter: Aditya Kishore >Assignee: Aditya Kishore >Priority: Minor > Labels: benchmark > Fix For: 0.96.0 > > Attachments: HBASE-6455_trunk.patch > > > I was the running PerformanceEvaluation test with a modified job > configuration where the job output path is created before the splits and that > unmasked the issue because the the PeInputFormat.getSplits() function expects > to find only files under the input path. > The attached patch addresses both the issues. > # Creates separate input and output path rooted under a single folder e.g. > "/performance_evaluation//inputs" and > "/performance_evaluation//outputs", and > # The PeInputFormat.getSplits(), now skips any folder found under the input > path and process only files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423512#comment-13423512 ] Hadoop QA commented on HBASE-6411: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538040/HBASE-6411-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 45 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause mvn compile goal to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2442//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2442//console This message is automatically generated. > Move Master Metrics to metrics 2 > > > Key: HBASE-6411 > URL: https://issues.apache.org/jira/browse/HBASE-6411 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, > HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch > > > Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423520#comment-13423520 ] stack commented on HBASE-6449: -- One more thing, I'd say HBase wants the generic lib. An hbase-specific would be great but won't be as effective as one that does NOT stop dead in its tracks at the HDFS API. A trace that goes all the ways down would be the cats pyjamas. > Dapper like tracing > --- > > Key: HBASE-6449 > URL: https://issues.apache.org/jira/browse/HBASE-6449 > Project: HBase > Issue Type: New Feature > Components: client, ipc >Affects Versions: 0.96.0 >Reporter: Jonathan Leavitt > Labels: tracing > Attachments: htrace1.diff, trace.png > > > Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to > HBase. [Accumulo|http://accumulo.apache.org] added something similar with > their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
[ https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423522#comment-13423522 ] Lars Hofhansl commented on HBASE-6461: -- Any chance to repeat this with distributed log splitting disabled? > Killing the HRegionServer and DataNode hosting ROOT can result in a malformed > root table. > - > > Key: HBASE-6461 > URL: https://issues.apache.org/jira/browse/HBASE-6461 > Project: HBase > Issue Type: Bug > Environment: hadoop-0.20.2-cdh3u3 > HBase 0.94.1 RC1 >Reporter: Elliott Clark >Priority: Critical > Fix For: 0.94.2 > > > Spun up a new dfs on hadoop-0.20.2-cdh3u3 > Started hbase > started running loadtest tool. > killed rs and dn holding root with killall -9 java on server sv4r27s44 at > about 2012-07-25 22:40:00 > After things stabilize Root is in a bad state. Ran hbck and got: > Exception in thread "main" > org.apache.hadoop.hbase.client.NoServerForRegionException: No server address > listed in -ROOT- for region .META.,,1.1028785192 containing row > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810) > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:172) > at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241) > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236) > hbase(main):001:0> scan '-ROOT-' > ROW COLUMN+CELL > > > 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration > already set up for Hadoop, not re-installing. > .META.,,1column=info:regioninfo, > timestamp=1343255838525, value={NAME => '.META.,,1', STARTKEY => '', ENDKEY > => '', ENCODED => 1028785192,} > .META.,,1column=info:v, > timestamp=1343255838525, value=\x00\x00 > > 1 row(s) in 0.5930 seconds > Here's the master log: https://gist.github.com/3179194 > I tried the same thing with 0.92.1 and I was able to get into a similar > situation, so I don't think this is anything new. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira