[jira] [Commented] (HBASE-9791) MR initializes scanner twice

2013-10-17 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798048#comment-13798048
 ] 

stack commented on HBASE-9791:
--

+1

 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,903 INFO [main]

[jira] [Commented] (HBASE-9791) MR initializes scanner twice

2013-10-17 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798198#comment-13798198
 ] 

Hadoop QA commented on HBASE-9791:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12608957/trunk-9791.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//console

This message is automatically generated.

 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)

[jira] [Commented] (HBASE-9791) MR initializes scanner twice

2013-10-17 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798455#comment-13798455
 ] 

Hudson commented on HBASE-9791:
---

FAILURE: Integrated in HBase-TRUNK #4625 (See 
[https://builds.apache.org/job/HBase-TRUNK/4625/])
HBASE-9791 MR initializes scanner twice (jxiang: rev 1533213)
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java


 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for

[jira] [Commented] (HBASE-9791) MR initializes scanner twice

2013-10-17 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798554#comment-13798554
 ] 

Hudson commented on HBASE-9791:
---

SUCCESS: Integrated in hbase-0.96 #146 (See 
[https://builds.apache.org/job/hbase-0.96/146/])
HBASE-9791 MR initializes scanner twice (jxiang: rev 1533214)
* 
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java
* 
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java


 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for

[jira] [Commented] (HBASE-9791) MR initializes scanner twice

2013-10-17 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798779#comment-13798779
 ] 

Hudson commented on HBASE-9791:
---

FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #797 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/797/])
HBASE-9791 MR initializes scanner twice (jxiang: rev 1533213)
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java


 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for

[jira] [Commented] (HBASE-9791) MR initializes scanner twice

2013-10-17 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798820#comment-13798820
 ] 

Hudson commented on HBASE-9791:
---

FAILURE: Integrated in hbase-0.96-hadoop2 #92 (See 
[https://builds.apache.org/job/hbase-0.96-hadoop2/92/])
HBASE-9791 MR initializes scanner twice (jxiang: rev 1533214)
* 
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java
* 
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java


 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for