[jira] [Updated] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Han updated YARN-1130:
---
Attachment: YARN-1130.patch
Improve the log flushing for tasks when mapred.userlog.limit.kb is set
--
Key: YARN-1130
URL: https://issues.apache.org/jira/browse/YARN-1130
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Paul Han
Assignee: Paul Han
Fix For: 2.0.5-alpha
Attachments: YARN-1130.patch, YARN-1130.patch, YARN-1130.patch
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for
user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
not flush to file until the container shutdown and logmanager close all
appenders. If user likes to see the log during task execution, it doesn't
support it.
Will propose a patch to add a flush mechanism and also flush the log when
task is done.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Han updated YARN-1130:
---
Attachment: YARN-1130.patch
Improve the log flushing for tasks when mapred.userlog.limit.kb is set
--
Key: YARN-1130
URL: https://issues.apache.org/jira/browse/YARN-1130
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Paul Han
Assignee: Paul Han
Fix For: 2.0.5-alpha
Attachments: YARN-1130.patch, YARN-1130.patch
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for
user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
not flush to file until the container shutdown and logmanager close all
appenders. If user likes to see the log during task execution, it doesn't
support it.
Will propose a patch to add a flush mechanism and also flush the log when
task is done.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated YARN-1130:
Assignee: Paul Han
Improve the log flushing for tasks when mapred.userlog.limit.kb is set
--
Key: YARN-1130
URL: https://issues.apache.org/jira/browse/YARN-1130
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Paul Han
Assignee: Paul Han
Fix For: 2.0.5-alpha
Attachments: YARN-1130.patch
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for
user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
not flush to file until the container shutdown and logmanager close all
appenders. If user likes to see the log during task execution, it doesn't
support it.
Will propose a patch to add a flush mechanism and also flush the log when
task is done.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Han updated YARN-1130:
---
Attachment: YARN-1130.patch
Improve the log flushing for tasks when mapred.userlog.limit.kb is set
--
Key: YARN-1130
URL: https://issues.apache.org/jira/browse/YARN-1130
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Paul Han
Attachments: YARN-1130.patch
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for
user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
not flush to file until the container shutdown and logmanager close all
appenders. If user likes to see the log during task execution, it doesn't
support it.
Will propose a patch to add a flush mechanism and also flush the log when
task is done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Han updated YARN-1130:
---
Attachment: YARN-1130.patch
Improve the log flushing for tasks when mapred.userlog.limit.kb is set
--
Key: YARN-1130
URL: https://issues.apache.org/jira/browse/YARN-1130
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Paul Han
Fix For: 2.0.5-alpha
Attachments: YARN-1130.patch
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for
user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
not flush to file until the container shutdown and logmanager close all
appenders. If user likes to see the log during task execution, it doesn't
support it.
Will propose a patch to add a flush mechanism and also flush the log when
task is done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Han updated YARN-1130:
---
Attachment: (was: YARN-1130.patch)
Improve the log flushing for tasks when mapred.userlog.limit.kb is set
--
Key: YARN-1130
URL: https://issues.apache.org/jira/browse/YARN-1130
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Paul Han
Fix For: 2.0.5-alpha
Attachments: YARN-1130.patch
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for
user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
not flush to file until the container shutdown and logmanager close all
appenders. If user likes to see the log during task execution, it doesn't
support it.
Will propose a patch to add a flush mechanism and also flush the log when
task is done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Han updated YARN-1130:
---
Description:
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not
flush to file until the container shutdown and logmanager close all appenders.
If user likes to see the log during task execution, it doesn't support it.
Will propose a patch to add a flush mechanism and also flush the log when task
is done.
was:
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) will not write to log until
the container shutdown and logmanager close all appenders. If user likes to see
the log during task execution, it doesn't support it.
Will propose a patch to add a flush mechanism and also flush the log when task
is done.
Improve the log flushing for tasks when mapred.userlog.limit.kb is set
--
Key: YARN-1130
URL: https://issues.apache.org/jira/browse/YARN-1130
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.5-alpha
Reporter: Paul Han
When userlog limit is set with something like this:
{code}
property
namemapred.userlog.limit.kb/name
value2048/value
descriptionThe maximum size of user-logs of each task in KB. 0 disables the
cap.
/description
/property
{code}
the log entry will be truncated randomly for the jobs.
The log size is left between 1.2MB to 1.6MB.
Since the log is already limited, avoid the log truncation is crucial for
user.
The other issue with the current
impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
not flush to file until the container shutdown and logmanager close all
appenders. If user likes to see the log during task execution, it doesn't
support it.
Will propose a patch to add a flush mechanism and also flush the log when
task is done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
