[
https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830207#action_12830207
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
After the patch, there are some improvement but archive still uses double
memory of ls.
- archive
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 658875 42680832 [C
2: 1434792 34435008 java.lang.String
3: 255174 20413920 java.net.URI
4: 255163 16330432 org.apache.hadoop.fs.FileStatus
5: 200001 4800024 org.apache.hadoop.fs.permission.FsPermission
6: 255172 4082752 org.apache.hadoop.fs.Path
{noformat}
- ls
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 304186 21086344 [C
2: 804264 19302336 java.lang.String
3: 100009 8000720 java.net.URI
4: 100001 6400064 org.apache.hadoop.fs.FileStatus
5: 100002 2400048 org.apache.hadoop.fs.permission.FsPermission
6: 100008 1600128 org.apache.hadoop.fs.Path
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch,
> MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at
> org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at
> org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
