[ https://issues.apache.org/jira/browse/GIRAPH-274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaeho Shin updated GIRAPH-274: ------------------------------ Attachment: GIRAPH-274.patch Here is our patch that adds several progress() calls after a careful code review with Greg Malewicz. It seems missing progress() call from BspServiceWorker#reserveInputSplit() was causing the timeout for idle workers during the INPUT_SUPERSTEP. There were many more spots where Giraph is doing a blocking call, but we left comments due to either not having access to the Context or the source code. This seems to be an endless effort and it'll only pollute Giraph's codebase as we try to fix more timeout cases. We definitely need a better systematic way to keep our Giraph jobs from timing out. One possibility is to run a separate thread from GraphMapper#run() which reports progress as long as the task don't crash, and stop worry about calling progress(). Do you think this is a good idea? Will this cause any trouble in the underlying Hadoop/MapReduce stack? If we're using MapReduce only for scheduling resources, then I believe there should be no reason for us to conform to MapReduce conventions of not using threads. > Jobs still failing due to tasks timeout during INPUT_SUPERSTEP > -------------------------------------------------------------- > > Key: GIRAPH-274 > URL: https://issues.apache.org/jira/browse/GIRAPH-274 > Project: Giraph > Issue Type: Bug > Affects Versions: 0.2.0 > Reporter: Jaeho Shin > Assignee: Jaeho Shin > Fix For: 0.2.0 > > Attachments: GIRAPH-274.patch > > > Even after GIRAPH-267, jobs were failing during INPUT_SUPERSTEP when some > workers don't get to reserve an input split, while others were loading > vertices for a long time. (related to GIRAPH-246 and GIRAPH-267) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira