[ https://issues.apache.org/jira/browse/MAPREDUCE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391345#comment-15391345 ]
Tsuyoshi Ozawa commented on MAPREDUCE-4522: ------------------------------------------- [~shyam_gav] Thanks for updating and sorry for my delay. Following lines exceeds 80LOC, so could you fix them? {quote} + List<List<K>> subLists = Lists.partition(records, context.getConfiguration().getInt(MR_DB_OUTPUTFORMAT_BATCH_SIZE,1000)); {quote} {quote} + public static final String MR_DB_OUTPUTFORMAT_BATCH_SIZE="mapreduce.output.dboutputformat.batch-size"; {quote} {quote} + <description>The batch size of SQL statements that will be executed before reporting progress. Default is 1000</description> {quote} > DBOutputFormat Times out on large batch inserts > ----------------------------------------------- > > Key: MAPREDUCE-4522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller > Affects Versions: 0.20.205.0 > Reporter: Nathan Jarus > Assignee: Shyam Gavulla > Labels: newbie > Attachments: MAPREDUCE-4522.001.patch > > > In DBRecordWriter#close(), progress is never updated. In large batch inserts, > this can cause the reduce task to time out due to the amount of time it takes > the SQL engine to process that insert. > Potential solutions I can see: > Don't batch inserts; do the insert when DBRecordWriter#write() is called > (awful) > Spin up a thread in DBRecordWriter#close() and update progress in that. > (gross) > I can provide code for either if you're interested. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org