[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861467#action_12861467 ] Ning Zhang commented on HIVE-1326: -- Michael, the temporary directory is needed for a conditional task to handle skewed joins (that's why the JobConf is cloned but with a different path). The idea here is that whenever there many too many rows for a particular job key to be held in memory, they will be first write to local disk via RowContainer. If there are still too many rows to local disk, they will be write to a DFS location and a contitional task will be triggered to handle this skewed key. So if we create a temp directory in local disk and let all mappers write their temp files in that directory, then later when we want to move the data to HDFS, we just move the whole directory instead of moving individual files. +1 on v2. Will commit if tests pass. > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hadoop Hive > Issue Type: Bug > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt > Attachments: rowcontainer.patch, rowcontainer_v2.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977) > at > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343) > at > org.apache.hado
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861449#action_12861449 ] Michael Klatt commented on HIVE-1326: - Hmm, on second thought, the parentFile (directory) is referenced in several places. I'm not familiar enough with Hive to know what these lines do: HiveConf.setVar(jobCloneUsingLocalFs, HiveConf.ConfVars.HADOOPMAPREDINPUTDIR, org.apache.hadoop.util.StringUtils.escapeString(parentFile .getAbsolutePath())); If it weren't for these lines I could remove the parentFile variable all together. > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hadoop Hive > Issue Type: Bug > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt > Attachments: rowcontainer.patch, rowcontainer_v2.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977) > at > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456) > at > org.apache.hadoop.hive.ql.exec.Exe
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861444#action_12861444 ] Michael Klatt commented on HIVE-1326: - The reason the parentFile.delete() call is there is because the File.createTempFile method actually creates a file. The code, as it currently is, creates a temporary directory to hold the rowcontainer file and I made the smallest change possible to continue to support this behavior. Looking at the code, it appears that the createTempFile mechanism is used several lines down to actually create the temporary file (within the new temporary directory). I'm not sure why a temporary directory is created first, but I'll submit a new patch which doesn't try to create a temporary directory at all. > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hadoop Hive > Issue Type: Bug > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt > Attachments: rowcontainer.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977) > at > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.ja
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861370#action_12861370 ] He Yongqiang commented on HIVE-1326: Agreed with Ning, looks fine to me. > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hadoop Hive > Issue Type: Bug > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt > Attachments: rowcontainer.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977) > at > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) > at org.apache.hadoop.mapred.Child.main(Child.java:158) > Caused by: java.io.IOException: No space left on device > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStrea
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861188#action_12861188 ] Ning Zhang commented on HIVE-1326: -- I guess this is due to https://issues.apache.org/jira/browse/HADOOP-2735 which set java.io.tmpdir to be dependent on mapred.tmp.dir. In that case the change of using createTempFile looks fine with me. Also can you remove the parentFile.delete() in your patch? If we set java.io.tmpdir to work/tmp then we don't have file name conflicts, But if the user choose to use shared /tmp then it may remove files created by other tasks. Yongqiang, any further comments? > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hadoop Hive > Issue Type: Bug > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt > Attachments: rowcontainer.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977) > at > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.r
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861080#action_12861080 ] Michael Klatt commented on HIVE-1326: - In my brief experiments, I found that File.createTempFile() used the work/tmp path for the individual task (in Hadoop). I suspect that the java.io.tmpdir is being set by the TaskTracker. This has the advantage of letting Hadoop do the temporary file cleanup when the task cleanup happens. I think it would be great if this was the default behavior, but admins could override that with a hive specific parameter. I don't have any attachment to the actual implementation. I can spend more time to produce a different patch if needed, but I suspected someone would want to implement it differently. > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hadoop Hive > Issue Type: Bug > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt > Attachments: rowcontainer.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977) > at > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861054#action_12861054 ] Ning Zhang commented on HIVE-1326: -- The general idea of why we put temp files and directories under /tmp is two fold: 1) it allows easy clean up in case of any unexpected interruptions to the JVM. In the regular exits, these temp files/directories should be removed by JVM by deleteOnExit(), but if the JVM was killed/interrupted unexpected, these temp files could eventually take up all disk spaces and make the whole cluster hard to recover (imagine finding these temp files in thousands of machines). By putting it in /tmp, all these will be automatically removed whenever the machine is restarted, or cleaned up manually by just removing all files in /tmp. That's much easier in terms of Hadoop administration. 2) /tmp seems to be a universally writable mounted point on all unix-based systems. So it should be a safe place to put the temporary files. But I agree that it would a good idea to introduce a new Hive parameter to let the user choose a customizable tmp directory, which should address Edward's point. I think we use File.createTempFile() previously and Yongqiang added another directory structure (parentFile). I think we can use File.createTempFile to create this directory as well (Yongqiang, please correct me if I'm wrong), there there are several questions about the patch: 1) File.createTempFile() will by default create temp files in java.io.tmpdir (/tmp or /var/tmp) according to JDK. I suspect most of the case JVM on Linux will use /tmp as the java.io.tmpdir, so in that caes your change won't achieve your purpose in most cases. 2) the random number generator is not needed if you can use File.createTempFile to create parentFile, because File.createTempFile guarantees its uniqueness for this JVM. The only case you need to prevent is the collision between different JVMs. This is why there is a check for parentFile.mkdir(). It actually acts like a write lock to the directory so that other JVM won't overwrite it. So parentFile.delete() should not be added because it may delete the directory created by another JVM who is still running. > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hadoop Hive > Issue Type: Bug > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt > Attachments: rowcontainer.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) >
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/' path for temporary files
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860929#action_12860929 ] Edward Capriolo commented on HIVE-1326: --- I am +1 on the concept. Many times /tmp can be a ramdisk or mounted with some size restrictions. This type of bug an be very painful to track down when it happens. > RowContainer uses hard-coded '/tmp/' path for temporary files > - > > Key: HIVE-1326 > URL: https://issues.apache.org/jira/browse/HIVE-1326 > Project: Hadoop Hive > Issue Type: Bug > Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, > but that doesn't seem relevant. >Reporter: Michael Klatt > Attachments: rowcontainer.patch > > > In our production hadoop environment, the "/tmp/" is actually pretty small, > and we encountered a problem when a query used the RowContainer class and > filled up the /tmp/ partition. I tracked down the cause to the RowContainer > class putting temporary files in the '/tmp/' path instead of using the > configured Hadoop temporary path. I've attached a patch to fix this. > Here's the traceback: > 2010-04-25 12:05:05,120 INFO > org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created > temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp > 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 1000 > rows: used memory = 385520312 > 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 1100 > rows: used memory = 341780472 > 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 1200 > rows: used memory = 301446768 > 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 1300 > rows: used memory = 399208768 > 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 1400 > rows: used memory = 364507216 > 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 1500 > rows: used memory = 332907280 > 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 1600 > rows: used memory = 298774096 > 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 1700 > rows: used memory = 396505408 > 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 1800 > rows: used memory = 362477288 > 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 1900 > rows: used memory = 327229744 > 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 2000 > rows: used memory = 296051904 > 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013) > at > org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977) > at > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343) > at > org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) > at org.apache.hadoop.mapred.Child.main(Child.java:158) > Caused by: java.io.IOException: No space left on device > at java.io.FileOutputStream.writeBytes(Native Method) >
