[
https://issues.apache.org/jira/browse/HBASE-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975935#comment-14975935
]
Hadoop QA commented on HBASE-14696:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12768890/14696-branch-1-v2.txt
against branch-1 branch at commit 496d20cfca5a30bc72a29e4ef893424964f9fa91.
ATTACHMENT ID: 12768890
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat
org.apache.hadoop.hbase.mapreduce.TestHashTable
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithTTLs
org.apache.hadoop.hbase.mapreduce.TestCellCounter
org.apache.hadoop.hbase.mapreduce.TestCopyTable
org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
org.apache.hadoop.hbase.mapreduce.TestWALPlayer
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2
org.apache.hadoop.hbase.mapreduce.TestRowCounter
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
org.apache.hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat
org.apache.hadoop.hbase.mapreduce.TestSyncTable
org.apache.hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16241//testReport/
Release Findbugs (version 2.0.3) warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16241//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16241//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16241//console
This message is automatically generated.
> Support setting allowPartialResults in mapreduce Mappers
> --------------------------------------------------------
>
> Key: HBASE-14696
> URL: https://issues.apache.org/jira/browse/HBASE-14696
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
> Affects Versions: 2.0.0, 1.1.0
> Reporter: Mindaugas Kairys
> Assignee: Ted Yu
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14696-branch-1-v1.txt, 14696-branch-1-v2.txt,
> 14696-v1.txt, 14696-v2.txt
>
>
> It is currently impossible to get partial results in mapreduce mapper jobs.
> When setting setAllowPartialResults(true) for scan jobs, they still fail with
> OOME on large rows.
> The reason is that Scan field allowPartialResults is lost during job creation:
> 1. User creates a Job and sets a scan object via
> TableMapReduceUtil.initTableMapperJob(table_name, scanObj,...) -> which puts
> a result of TableMapReduceUtil.convertScanToString(scanObj) to the job config.
> 2. When the job starts - method TableInputFormat.setConfig retrieves a scan
> string from config and converts it to Scan object by calling
> TableMapReduceUtil.convertStringToScan - which results in a Scan object with
> a field allowPartialResults always set to false.
> I have tried to experiment and modify a TableInputFormat method setConfig()
> by forcing all scans to allow partial results and after this all jobs
> succeeded with no more OOME and I also noticed that mappers began to get
> partial results (Result.isPartial()).
> My use case is very simple - I just have large rows and expect a mapper to
> get them partially - to get same rowid several times with different key/value
> records.
> This would allow me not to worry about implementing my own result
> partitioning solution, which i would encounter in case the big amount of
> result key values could be transparently returned for a single large row.
> And from the other side - if a Scan object can return several records for the
> same rowid (partial results), perhaps the mapper should do the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
