[jira] [Commented] (GIRAPH-1147) Store timestamps when various fractions of input were done
[ https://issues.apache.org/jira/browse/GIRAPH-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016758#comment-16016758 ] ASF GitHub Bot commented on GIRAPH-1147: Github user asfgit closed the pull request at: https://github.com/apache/giraph/pull/38 > Store timestamps when various fractions of input were done > -- > > Key: GIRAPH-1147 > URL: https://issues.apache.org/jira/browse/GIRAPH-1147 > Project: Giraph > Issue Type: New Feature >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In order to evaluate how read stragglers affect job performance, add a way to > expose timestamps when various fractions of input were done reading through > counters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (GIRAPH-1147) Store timestamps when various fractions of input were done
[ https://issues.apache.org/jira/browse/GIRAPH-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016742#comment-16016742 ] ASF GitHub Bot commented on GIRAPH-1147: Github user majakabiljo commented on the issue: https://github.com/apache/giraph/pull/38 Ah didn't know, will do in the future > Store timestamps when various fractions of input were done > -- > > Key: GIRAPH-1147 > URL: https://issues.apache.org/jira/browse/GIRAPH-1147 > Project: Giraph > Issue Type: New Feature >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In order to evaluate how read stragglers affect job performance, add a way to > expose timestamps when various fractions of input were done reading through > counters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (GIRAPH-1147) Store timestamps when various fractions of input were done
[ https://issues.apache.org/jira/browse/GIRAPH-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016675#comment-16016675 ] ASF GitHub Bot commented on GIRAPH-1147: Github user dlogothetis commented on the issue: https://github.com/apache/giraph/pull/38 Looks ok to me. Btw, did you squash the commits? You don't need to, they are squashed automatically when the pull request is merged. And we don't loose the diff between updates. > Store timestamps when various fractions of input were done > -- > > Key: GIRAPH-1147 > URL: https://issues.apache.org/jira/browse/GIRAPH-1147 > Project: Giraph > Issue Type: New Feature >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In order to evaluate how read stragglers affect job performance, add a way to > expose timestamps when various fractions of input were done reading through > counters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (GIRAPH-1147) Store timestamps when various fractions of input were done
[ https://issues.apache.org/jira/browse/GIRAPH-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016604#comment-16016604 ] ASF GitHub Bot commented on GIRAPH-1147: Github user majakabiljo commented on the issue: https://github.com/apache/giraph/pull/38 Updated with comments and tested again > Store timestamps when various fractions of input were done > -- > > Key: GIRAPH-1147 > URL: https://issues.apache.org/jira/browse/GIRAPH-1147 > Project: Giraph > Issue Type: New Feature >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In order to evaluate how read stragglers affect job performance, add a way to > expose timestamps when various fractions of input were done reading through > counters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (GIRAPH-1147) Store timestamps when various fractions of input were done
[ https://issues.apache.org/jira/browse/GIRAPH-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016535#comment-16016535 ] ASF GitHub Bot commented on GIRAPH-1147: Github user dlogothetis commented on a diff in the pull request: https://github.com/apache/giraph/pull/38#discussion_r117364342 --- Diff: giraph-core/src/main/java/org/apache/giraph/master/input/MasterInputSplitsHandler.java --- @@ -56,16 +69,39 @@ /** Latches to say when one input splits type is ready to be accessed */ private Map latchesMap = new EnumMap<>(InputType.class); + /** Context for accessing counters */ + private final Mapper.Context context; + /** How many splits per type are there total */ + private final Map numSplitsPerType = + new EnumMap<>(InputType.class); + /** How many splits per type have been read so far */ + private final Map numSplitsReadPerType = + new EnumMap<>(InputType.class); + /** + * Store in counters timestamps when we finished reading + * these fractions of input + */ + private final double[] doneFractionsToStoreInCoutners; --- End diff -- Typo in field name. > Store timestamps when various fractions of input were done > -- > > Key: GIRAPH-1147 > URL: https://issues.apache.org/jira/browse/GIRAPH-1147 > Project: Giraph > Issue Type: New Feature >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In order to evaluate how read stragglers affect job performance, add a way to > expose timestamps when various fractions of input were done reading through > counters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (GIRAPH-1147) Store timestamps when various fractions of input were done
[ https://issues.apache.org/jira/browse/GIRAPH-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016531#comment-16016531 ] ASF GitHub Bot commented on GIRAPH-1147: Github user dlogothetis commented on the issue: https://github.com/apache/giraph/pull/38 As opposed to timestamps why not set the counters to the time passed between the different fractions? That's going to be easier to parse quickly. > Store timestamps when various fractions of input were done > -- > > Key: GIRAPH-1147 > URL: https://issues.apache.org/jira/browse/GIRAPH-1147 > Project: Giraph > Issue Type: New Feature >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In order to evaluate how read stragglers affect job performance, add a way to > expose timestamps when various fractions of input were done reading through > counters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (GIRAPH-1138) Don't wrap exceptions from executor service
[ https://issues.apache.org/jira/browse/GIRAPH-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo resolved GIRAPH-1138. -- Resolution: Fixed > Don't wrap exceptions from executor service > --- > > Key: GIRAPH-1138 > URL: https://issues.apache.org/jira/browse/GIRAPH-1138 > Project: Giraph > Issue Type: Improvement >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In ProgressableUtils.getResultsWithNCallables we wrap exceptions from > underlying threads, making logs hard to read. We should re-throw original > exception when possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (GIRAPH-1146) Keep track of number of supersteps when possible
[ https://issues.apache.org/jira/browse/GIRAPH-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo resolved GIRAPH-1146. -- Resolution: Fixed > Keep track of number of supersteps when possible > > > Key: GIRAPH-1146 > URL: https://issues.apache.org/jira/browse/GIRAPH-1146 > Project: Giraph > Issue Type: New Feature >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In many cases we know how many supersteps are there going to be. We can keep > track of it and log it with progress. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (GIRAPH-1147) Store timestamps when various fractions of input were done
[ https://issues.apache.org/jira/browse/GIRAPH-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015991#comment-16015991 ] ASF GitHub Bot commented on GIRAPH-1147: GitHub user majakabiljo opened a pull request: https://github.com/apache/giraph/pull/38 [GIRAPH-1147] Store timestamps when various fractions of input were done Summary: In order to evaluate how read stragglers affect job performance, add a way to expose timestamps when various fractions of input were done reading through counters. Test Plan: Ran a big job and verified counters are set correctly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/majakabiljo/giraph inputCounters Alternatively you can review and apply these changes as the patch at: https://github.com/apache/giraph/pull/38.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #38 commit 56305ecfeb223313fc63e331cb815bfdc2430731 Author: Maja Kabiljo Date: 2017-05-18T15:30:05Z [GIRAPH-1147] Store timestamps when various fractions of input were done Summary: In order to evaluate how read stragglers affect job performance, add a way to expose timestamps when various fractions of input were done reading through counters. Test Plan: Ran a big job and verified counters are set correctly. > Store timestamps when various fractions of input were done > -- > > Key: GIRAPH-1147 > URL: https://issues.apache.org/jira/browse/GIRAPH-1147 > Project: Giraph > Issue Type: New Feature >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > > In order to evaluate how read stragglers affect job performance, add a way to > expose timestamps when various fractions of input were done reading through > counters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (GIRAPH-1147) Store timestamps when various fractions of input were done
Maja Kabiljo created GIRAPH-1147: Summary: Store timestamps when various fractions of input were done Key: GIRAPH-1147 URL: https://issues.apache.org/jira/browse/GIRAPH-1147 Project: Giraph Issue Type: New Feature Reporter: Maja Kabiljo Assignee: Maja Kabiljo Priority: Minor In order to evaluate how read stragglers affect job performance, add a way to expose timestamps when various fractions of input were done reading through counters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
