[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524969#comment-14524969
]
Hadoop QA commented on MAPREDUCE-5969:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12673831/MAPREDUCE-5969.branch1.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5577/console |
This message was automatically generated.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.1.patch,
MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
{code}
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
{code}
The second time we add file size is at
setSize:
{code}
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
{code}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525025#comment-14525025
]
Hadoop QA commented on MAPREDUCE-5969:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12673831/MAPREDUCE-5969.branch1.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5580/console |
This message was automatically generated.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.1.patch,
MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
{code}
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
{code}
The second time we add file size is at
setSize:
{code}
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
{code}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164772#comment-14164772
]
zhihai xu commented on MAPREDUCE-5969:
--
update the patch MAPREDUCE-5969.branch1.1.patch based on the latest branch-1
code.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.1.patch,
MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
{code}
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
{code}
The second time we add file size is at
setSize:
{code}
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
{code}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164773#comment-14164773
]
Hadoop QA commented on MAPREDUCE-5969:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673831/MAPREDUCE-5969.branch1.1.patch
against trunk revision 2a51494.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4945//console
This message is automatically generated.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.1.patch,
MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
{code}
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
{code}
The second time we add file size is at
setSize:
{code}
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
{code}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14095019#comment-14095019
]
zhihai xu commented on MAPREDUCE-5969:
--
[~kasha] - I checked the MR2(trunk/branch-2) source code, the implementation is
totally different from MR1(branch-1).
The MR2(trunk/branch-2) use LocalizedResource to manage the cache size. The
LocalizedResource is created in LocalResourcesTrackerImpl after receive
ContainerLocalizationRequestEvent(ContainerInitEvent) to request LocalResource
from ContainerLaunchContext (container.launchContext). ContainerLaunchContext
is created in TaskAttemptImpl.java(createContainerLaunchContext) and
YARNRunner.java(createApplicationSubmissionContext).
LocalResource in ContainerLaunchContext is created by {code}
MRApps.setupDistributedCache(conf, localResources) {code}.
So MR2(trunk/branch-2) doesn't have this issue.
The following is the size calculation after received ResourceLocalizedEvent in
LocalizedResource.java:
{code}
private static class FetchSuccessTransition extends ResourceTransition {
@Override
public void transition(LocalizedResource rsrc, ResourceEvent event) {
ResourceLocalizedEvent locEvent = (ResourceLocalizedEvent) event;
rsrc.localPath =
Path.getPathWithoutSchemeAndAuthority(locEvent.getLocation());
rsrc.size = locEvent.getSize();
for (ContainerId container : rsrc.ref) {
rsrc.dispatcher.getEventHandler().handle(
new ContainerResourceLocalizedEvent(
container, rsrc.rsrc, rsrc.localPath));
}
}
}
{code}
the size in ResourceLocalizedEvent is in the following
code(ResourceLocalizationService.java):
For public resource:
{code}
publicRsrc.handle(new ResourceLocalizedEvent(key, local, FileUtil
.getDU(new File(local.toUri();
{code}
For private resource:
{code}
getLocalResourcesTracker(req.getVisibility(), user, applicationId)
.handle(
new ResourceLocalizedEvent(req, ConverterUtils
.getPathFromYarnURL(stat.getLocalPath()),
stat.getLocalSize()));
{code}
The cache cleanup is at the following code:
{code}
// from ResourceLocalizationService.java
private void handleCacheCleanup(LocalizationEvent event) {
ResourceRetentionSet retain =
new ResourceRetentionSet(delService, cacheTargetSize);
retain.addResources(publicRsrc);
LOG.debug(Resource cleanup (public) + retain);
for (LocalResourcesTracker t : privateRsrc.values()) {
retain.addResources(t);
LOG.debug(Resource cleanup + t.getUser() + : + retain);
}
//TODO Check if appRsrcs should also be added to the retention set.
}
// from ResourceRetentionSet.java
public void addResources(LocalResourcesTracker newTracker) {
for (LocalizedResource resource : newTracker) {
currentSize += resource.getSize();
if (resource.getRefCount() 0) {
// always retain resources in use
continue;
}
retain.put(resource, newTracker);
}
for (IteratorMap.EntryLocalizedResource,LocalResourcesTracker i =
retain.entrySet().iterator();
currentSize - delSize targetSize i.hasNext();) {
Map.EntryLocalizedResource,LocalResourcesTracker rsrc = i.next();
LocalizedResource resource = rsrc.getKey();
LocalResourcesTracker tracker = rsrc.getValue();
if (tracker.remove(resource, delService)) {
delSize += resource.getSize();
i.remove();
}
}
}
{code}
And It should be only one copy of LocalizedResource for each
LocalResourceRequest is saved in publicRsrc or privateRsrc.
So this issue should only happen for MR1(branch-1).
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092216#comment-14092216
]
Karthik Kambatla commented on MAPREDUCE-5969:
-
[~zxu] - thanks for reporting and working on this issue. Can you also check if
this is an issue in trunk/branch-2 so we can fix that first? Thanks.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
{code}
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
{code}
The second time we add file size is at
setSize:
{code}
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
{code}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081388#comment-14081388
]
zhihai xu commented on MAPREDUCE-5969:
--
I add unit test: testFileNotDoubleCounted to cover this error case. Without the
fix, the test will fail.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
{code}
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
{code}
The second time we add file size is at
setSize:
{code}
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
{code}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081860#comment-14081860
]
Hadoop QA commented on MAPREDUCE-5969:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12658993/MAPREDUCE-5969.branch1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4782//console
This message is automatically generated.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
{code}
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
{code}
The second time we add file size is at
setSize:
{code}
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
{code}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062903#comment-14062903
]
zhihai xu commented on MAPREDUCE-5969:
--
I updated the patch. The new patch is to delete the old file size before add
the new file size for Private non-Archive Files.
With the new patch, It will work even you change the file size multiple times
for the same file.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
The second time we add file size is at
setSize:
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062908#comment-14062908
]
Hadoop QA commented on MAPREDUCE-5969:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655931/MAPREDUCE-5969.branch1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4738//console
This message is automatically generated.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
The second time we add file size is at
setSize:
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061554#comment-14061554
]
zhihai xu commented on MAPREDUCE-5969:
--
I submit patch for review:
I tested this patch. The following log show the total file size in the cache is
6260 and total file number in cache is 2.
, Which is correct. Without this patch, the total file size in the cache is
12588 and total file number in cache is 4.
addCacheInfoUpdate size: 2395 num: 1
addCacheInfoUpdate size: 6260 num: 2
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
The second time we add file size is at
setSize:
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5969) Private non-Archive Files' size add twice in Distributed Cache directory size calculation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061555#comment-14061555
]
Hadoop QA commented on MAPREDUCE-5969:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655668/MAPREDUCE-5969.branch1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4732//console
This message is automatically generated.
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation.
--
Key: MAPREDUCE-5969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-5969.branch1.patch
Private non-Archive Files' size add twice in Distributed Cache directory size
calculation. Private non-Archive Files list is passed in by -files command
line option. The Distributed Cache directory size is used to check whether
the total cache files size exceed the cache size limitation, the default
cache size limitation is 10G.
I add log in addCacheInfoUpdate and setSize in
TrackerDistributedCacheManager.java.
I use the following command to test:
hadoop jar ./wordcount.jar org.apache.hadoop.examples.WordCount -files
hdfs://host:8022/tmp/zxu/WordCount.java,hdfs://host:8022/tmp/zxu/wordcount.jar
/tmp/zxu/test_in/ /tmp/zxu/test_out
to add two files into distributed cache:WordCount.java and wordcount.jar.
WordCount.java file size is 2395 byes and wordcount.jar file size is 3865
bytes. The total should be 6260.
The log show these files size added twice:
add one time before download to local node and add second time after download
to local node, so total file number becomes 4 instead of 2:
addCacheInfoUpdate size: 6260 num: 2 baseDir: /mapred/local
addCacheInfoUpdate size: 8683 num: 3 baseDir: /mapred/local
addCacheInfoUpdate size: 12588 num: 4 baseDir: /mapred/local
In the code, for Private non-Archive File, the first time we add file size is
at
getLocalCache:
if (!isArchive) {
//for private archives, the lengths come over RPC from the
//JobLocalizer since the JobLocalizer is the one who expands
//archives and gets the total length
lcacheStatus.size = fileStatus.getLen();
LOG.info(getLocalCache: + localizedPath + size =
+ lcacheStatus.size);
// Increase the size and sub directory count of the cache
// from baseDirSize and baseDirNumberSubDir.
baseDirManager.addCacheInfoUpdate(lcacheStatus);
}
The second time we add file size is at
setSize:
synchronized (status) {
status.size = size;
baseDirManager.addCacheInfoUpdate(status);
}
The fix is not to add the file size for for Private non-Archive File after
download(downloadCacheObject).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
