[ https://issues.apache.org/jira/browse/SPARK-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568309#comment-14568309 ]
Sean Owen commented on SPARK-8024: ---------------------------------- OK, but what is this about that is different? You should just comment on the other JIRA, I think. > Luigi triggering resolved Blockmanager bug > ------------------------------------------ > > Key: SPARK-8024 > URL: https://issues.apache.org/jira/browse/SPARK-8024 > Project: Spark > Issue Type: Bug > Components: Block Manager > Affects Versions: 1.3.1 > Reporter: Cory Nguyen > > We are using Luigi with Spark to manage our jobs > However we run into a unique rare case with the following conditions that > trigger the resolved Block Manger Bug: > - Dataset is relatively large ~ 1.5TB > - Spark job is ran with Luigi > - save to local HDFS > The spark job would process data and mappings just fine, until the very end > when it proceeds to save the files to local hdfs this is when it triggers > this bug. > However, the job saves and complete data successfully if it was saved to > s3:// location. > wondering what might cause this resolved bug to trigger when ran with luigi > saving to local hdfs but not trigger when saved to s3 with luigi or ran > without luigi? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org