[jira] [Commented] (MAPREDUCE-3384) Add warning message for underflow or overflow

[Brahma Reddy Battula (JIRA)]

    [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585835#comment-14585835
 ] 

Brahma Reddy Battula commented on MAPREDUCE-3384:
-------------------------------------------------

[~devaraj.k] thanks for review..
{quote}Can we also have key as part of the warning message?{quote}

Ok.we can have.

{quote}
I am thinking, if we have more number of values where overflow/underflow can 
happen, we would be filling with these logs. Can we limit having a some flag in 
the reduce method to log only once for overflow and underflow for each key? any 
thoughts?{quote}

I am thinking like following,, please let me know your opinion...

{code}
public void reduce(Key key, Iterable<IntWritable> values,
                     Context context) throws IOException, InterruptedException {
    int sum = 0;
    boolean isOverflow=false;
    boolean isUnderflow=false;
    for (IntWritable val : values) {
      int tempVal = val.get();
      int oldSum = sum;
      sum += tempVal;
      if ((!isOverflow && oldSum > 0 && tempVal > 0) && (oldSum > sum)) {
        LOG.warn("Arithmetic overflow occurs while reducing"+tempVal);
        isOverflow=True;
      }
      if ((!isUnderflow && oldSum < 0 && tempVal < 0) && (oldSum < sum)) {
        LOG.warn("Arithmetic underflow occurs while reducing"+tempVal);
        isUnderflow=True;
      }
    }
    result.set(sum);
    context.write(key, result);
  }
{code}

> Add warning message for underflow or overflow
> ---------------------------------------------
>
>                 Key: MAPREDUCE-3384
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3384
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: JiangKai
>            Assignee: Brahma Reddy Battula
>         Attachments: MAPREDUCE-3384-002.patch, MAPREDUCE-3384-003.patch, 
> MAPREDUCE-3384.patch
>
>
> When we call the function reduce() of LongSumReducer,the result may overflow 
> or underflow.
> We should have a warning message to users if overflow/underflow occurs for 
> all these classes,
> {code:xml}
> /java/org/apache/hadoop/mapred/lib/LongSumReducer.java
>  .../apache/hadoop/mapreduce/lib/reduce/IntSumReducer.java
>  .../apache/hadoop/mapreduce/lib/reduce/LongSumReducer.java
>  .../test/java/org/apache/hadoop/fs/AccumulatingReducer.java
>  .../hadoop/mapred/nativetask/kvtest/HashSumReducer.java
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)