[jira] [Updated] (MAPREDUCE-7249) Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated MAPREDUCE-7249:
-
Fix Version/s: 2.10.1
Pushed to branch-2.10
> Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes
> job failure
>
>
> Key: MAPREDUCE-7249
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7249
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 3.1.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Critical
> Labels: Reviewed
> Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1
>
> Attachments: MAPREDUCE-7249-001.patch, MAPREDUCE-7249-002.patch,
> MAPREDUCE-7249-branch-3.2.001.patch
>
>
> Same issue as in MAPREDUCE-7240 but this one has a different state in which
> the Exception {{TA_TOO_MANY_FETCH_FAILURE}} event is received:
> {code}
> 2019-11-18 23:03:40,270 ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle
> this event at current state for attempt_1568654141590_630203_m_003108_1
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1183)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:148)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1380)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> {code}
> The stack trace is from a CDH release which is highly patched 2.6 release.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7240) Exception ' Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER' cause job error
[
https://issues.apache.org/jira/browse/MAPREDUCE-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated MAPREDUCE-7240:
-
Fix Version/s: 2.10.1
Pushed to branch-2.10
> Exception ' Invalid event: TA_TOO_MANY_FETCH_FAILURE at
> SUCCESS_FINISHING_CONTAINER' cause job error
>
>
> Key: MAPREDUCE-7240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.2
>Reporter: luhuachao
>Assignee: luhuachao
>Priority: Critical
> Labels: Reviewed, applicationmaster, mrv2
> Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1
>
> Attachments: MAPREDUCE-7240-001.patch, MAPREDUCE-7240-002.patch,
> MAPREDUCE-7240-branch-3.1.001.patch, MAPREDUCE-7240-branch-3.2.001.patch,
> MAPREDUCE-7240-branch-3.2.001.patch, application_1566552310686_260041.log
>
>
> *log in appmaster*
> {noformat}
> 2019-09-03 17:18:43,090 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures
> for output of task attempt: attempt_1566552310686_260041_m_52_0 ...
> raising fetch failure to map
> 2019-09-03 17:18:43,091 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures
> for output of task attempt: attempt_1566552310686_260041_m_49_0 ...
> raising fetch failure to map
> 2019-09-03 17:18:43,091 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures
> for output of task attempt: attempt_1566552310686_260041_m_51_0 ...
> raising fetch failure to map
> 2019-09-03 17:18:43,091 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures
> for output of task attempt: attempt_1566552310686_260041_m_50_0 ...
> raising fetch failure to map
> 2019-09-03 17:18:43,091 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures
> for output of task attempt: attempt_1566552310686_260041_m_53_0 ...
> raising fetch failure to map
> 2019-09-03 17:18:43,092 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1566552310686_260041_m_52_0 transitioned from state SUCCEEDED to
> FAILED, event type is TA_TOO_MANY_FETCH_FAILURE and nodeId=yarn095:45454
> 2019-09-03 17:18:43,092 ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle
> this event at current state for attempt_1566552310686_260041_m_49_0
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:146)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1458)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1450)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
> at java.lang.Thread.run(Thread.java:745)
> 2019-09-03 17:18:43,093 ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle
> this event at current state for attempt_1566552310686_260041_m_51_0
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:146)
> at
> org.apache.had
[jira] [Updated] (MAPREDUCE-7208) Tuning TaskRuntimeEstimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated MAPREDUCE-7208: - Fix Version/s: (was: 2.11.0) > Tuning TaskRuntimeEstimator > > > Key: MAPREDUCE-7208 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7208 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1 > > Attachments: MAPREDUCE-7208-branch-2.10.001.patch, > MAPREDUCE-7208-branch-2.10.002.patch, MAPREDUCE-7208.001.patch, > MAPREDUCE-7208.002.patch, MAPREDUCE-7208.003.patch, MAPREDUCE-7208.004.patch, > smoothing-exponential.md > > > By default, MR uses LegacyTaskRuntimeEstimator to get an estimate of the > runtime. The estimator does not adjust dynamically to the progress rate of > the tasks. On the other hand, the existing alternative > "ExponentiallySmoothedTaskRuntimeEstimator" behavior in unpredictable. > > There are several dimensions to improve the exponential implementation: > # Exponential shooting needs a warmup period. Otherwise, the estimate will > be affected by the initial values. > # Using a single smoothing factor (Lambda) does not work well for all the > tasks. To increase the level of smoothing across the majority of tasks, we > need to give a range of flexibility to dynamically adjust the smoothing > factor based on the history of the task progress. > # Design wise, it is better to separate between the statistical model and > the MR interface. We need to have a way to evaluate estimators statistically, > without the need to run MR. For example, an estimator can be evaluated as a > black box by using a stream of raw data as input and testing the accuracy of > the generated stream of estimates. > # The exponential estimator speculates frequently and fails to detect > slowing tasks. It does not detect slowing tasks. As a result, a taskAttempt > that does not do any progress won't trigger a new speculation. > > The file [^smoothing-exponential.md] describes how Simple Exponential > smoothing factor works. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7208) Tuning TaskRuntimeEstimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991889#comment-16991889 ] Jonathan Hung commented on MAPREDUCE-7208: -- Removing 2.11.0 fix version after branch-2 -> branch-2.10 rename > Tuning TaskRuntimeEstimator > > > Key: MAPREDUCE-7208 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7208 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1 > > Attachments: MAPREDUCE-7208-branch-2.10.001.patch, > MAPREDUCE-7208-branch-2.10.002.patch, MAPREDUCE-7208.001.patch, > MAPREDUCE-7208.002.patch, MAPREDUCE-7208.003.patch, MAPREDUCE-7208.004.patch, > smoothing-exponential.md > > > By default, MR uses LegacyTaskRuntimeEstimator to get an estimate of the > runtime. The estimator does not adjust dynamically to the progress rate of > the tasks. On the other hand, the existing alternative > "ExponentiallySmoothedTaskRuntimeEstimator" behavior in unpredictable. > > There are several dimensions to improve the exponential implementation: > # Exponential shooting needs a warmup period. Otherwise, the estimate will > be affected by the initial values. > # Using a single smoothing factor (Lambda) does not work well for all the > tasks. To increase the level of smoothing across the majority of tasks, we > need to give a range of flexibility to dynamically adjust the smoothing > factor based on the history of the task progress. > # Design wise, it is better to separate between the statistical model and > the MR interface. We need to have a way to evaluate estimators statistically, > without the need to run MR. For example, an estimator can be evaluated as a > black box by using a stream of raw data as input and testing the accuracy of > the generated stream of estimates. > # The exponential estimator speculates frequently and fails to detect > slowing tasks. It does not detect slowing tasks. As a result, a taskAttempt > that does not do any progress won't trigger a new speculation. > > The file [^smoothing-exponential.md] describes how Simple Exponential > smoothing factor works. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7185) Parallelize part files move in FileOutputCommitter
[
https://issues.apache.org/jira/browse/MAPREDUCE-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated MAPREDUCE-7185:
-
Target Version/s: 3.3.0, 2.10.1 (was: 2.10.0, 3.3.0)
> Parallelize part files move in FileOutputCommitter
> --
>
> Key: MAPREDUCE-7185
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7185
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 3.2.0, 2.9.2
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Attachments: MAPREDUCE-7185.patch
>
>
> If map task outputs multiple files it could be slow to move them from temp
> directory to output directory in object stores (GCS, S3, etc).
> To improve performance we need to parallelize move of more than 1 file in
> FileOutputCommitter.
> Repro:
> Start spark-shell:
> {code}
> spark-shell --num-executors 2 --executor-memory 10G --executor-cores 4 --conf
> spark.dynamicAllocation.maxExecutors=2
> {code}
> From spark-shell:
> {code}
> val df = (1 to 1).toList.toDF("value").withColumn("p", $"value" %
> 10).repartition(50)
> df.write.partitionBy("p").mode("overwrite").format("parquet").options(Map("path"
> -> s"gs://some/path")).saveAsTable("parquet_partitioned_bench")
> {code}
> With the fix execution time reduces from 130 seconds to 50 seconds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015044#comment-16015044 ] Jonathan Hung commented on MAPREDUCE-6304: -- +1 (non-binding) for the 2.7.4 patch. > Specifying node labels when submitting MR jobs > -- > > Key: MAPREDUCE-6304 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Jian Fang >Assignee: Naganarasimha G R > Labels: mapreduce, release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6304.20150410-1.patch, > MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch, > MAPREDUCE-6304.20150510-1.patch, MAPREDUCE-6304.20150511-1.patch, > MAPREDUCE-6304.20150512-1.patch, MAPREDUCE-6304-branch-2.7.patch > > > Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify > node labels when submitting MR jobs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6885) JobHistory event handling does not complete if handling event throws exception on shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated MAPREDUCE-6885: - Description: If eventHandlingThread handles an event which causes it to throw an exception (e.g. if it is unable to flush an event to HDFS), the thread dies. The events are enqueued and eventually handled when JobHistoryEventHandler stops. If handling these events also throws an exception, the remaining events are lost. This can for example cause moving job history files to mapreduce.jobhistory.done-dir to not occur. There should be some fail-proof logic here to prevent these events from being lost. Should also be careful that the same exception is not thrown for each event to prevent the logs from being cluttered with the same stacktrace. Perhaps we can set a configurable number of failed handleEvent calls before finally giving up a clean shutdown. was: If eventHandlingThread handles an event which causes it to throw an exception (e.g. if it is unable to flush an event to HDFS), the thread dies. This thread is responsible for moving job history files to mapreduce.jobhistory.done-dir, if an exception is thrown the files will not be moved here, which is bad. We should catch these exceptions so that the thread can still move these files when the job is complete. > JobHistory event handling does not complete if handling event throws > exception on shutdown > -- > > Key: MAPREDUCE-6885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6885 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Hung > > If eventHandlingThread handles an event which causes it to throw an exception > (e.g. if it is unable to flush an event to HDFS), the thread dies. The events > are enqueued and eventually handled when JobHistoryEventHandler stops. If > handling these events also throws an exception, the remaining events are > lost. This can for example cause moving job history files to > mapreduce.jobhistory.done-dir to not occur. > There should be some fail-proof logic here to prevent these events from being > lost. Should also be careful that the same exception is not thrown for each > event to prevent the logs from being cluttered with the same stacktrace. > Perhaps we can set a configurable number of failed handleEvent calls before > finally giving up a clean shutdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6885) JobHistory event handling does not complete if handling event throws exception on shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated MAPREDUCE-6885: - Summary: JobHistory event handling does not complete if handling event throws exception on shutdown (was: JobHistory event handler thread should not die if exception thrown) > JobHistory event handling does not complete if handling event throws > exception on shutdown > -- > > Key: MAPREDUCE-6885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6885 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Hung > > If eventHandlingThread handles an event which causes it to throw an exception > (e.g. if it is unable to flush an event to HDFS), the thread dies. This > thread is responsible for moving job history files to > mapreduce.jobhistory.done-dir, if an exception is thrown the files will not > be moved here, which is bad. > We should catch these exceptions so that the thread can still move these > files when the job is complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6885) JobHistory event handler thread should not die if exception thrown
Jonathan Hung created MAPREDUCE-6885: Summary: JobHistory event handler thread should not die if exception thrown Key: MAPREDUCE-6885 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6885 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Hung If eventHandlingThread handles an event which causes it to throw an exception (e.g. if it is unable to flush an event to HDFS), the thread dies. This thread is responsible for moving job history files to mapreduce.jobhistory.done-dir, if an exception is thrown the files will not be moved here, which is bad. We should catch these exceptions so that the thread can still move these files when the job is complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6860) User intermediate-done-dir permissions should use history permissions configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated MAPREDUCE-6860: - Summary: User intermediate-done-dir permissions should use history permissions configuration (was: User intermediate-done-dir permissions should use history file permissions configuration) > User intermediate-done-dir permissions should use history permissions > configuration > --- > > Key: MAPREDUCE-6860 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6860 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Hung > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6860) User intermediate-done-dir permissions should use history file permissions configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904062#comment-15904062
]
Jonathan Hung commented on MAPREDUCE-6860:
--
Oh, I just realized this is a change we have made internally that I was not
aware of. Sorry for the confusion. Will close this ticket.
> User intermediate-done-dir permissions should use history file permissions
> configuration
>
>
> Key: MAPREDUCE-6860
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6860
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jonathan Hung
>
> Currently {{JobHistoryEventHandler}} creates the user intermediate-done-dir
> directory here: {noformat} doneDirPrefixPath =
> FileContext.getFileContext(conf).makeQualified(new
> Path(userDoneDirStr));
> mkdir(doneDirFS, doneDirPrefixPath, new FsPermission(
>
> JobHistoryUtils.HISTORY_INTERMEDIATE_USER_DIR_PERMISSIONS));{noformat} which
> is hardcoded to 770. But the summary, history, and conf files under this user
> dir are configurable via
> {{mapreduce.jobhistory.intermediate-done-dir.file.permission}}. So if the
> configured permissions has read/write/execute permissions for "other" users,
> they will still not have access to these files due to the 770 permission on
> the user dir.
> I see two options here:
> # Reuse {{mapreduce.jobhistory.intermediate-done-dir.file.permission}} as the
> permissions for the user dir
> # Create a new config for the user dir permissions, using 770 as the default
> The latter makes more sense to me.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6860) User intermediate-done-dir permissions should use history file permissions configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung resolved MAPREDUCE-6860.
--
Resolution: Not A Bug
> User intermediate-done-dir permissions should use history file permissions
> configuration
>
>
> Key: MAPREDUCE-6860
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6860
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jonathan Hung
>
> Currently {{JobHistoryEventHandler}} creates the user intermediate-done-dir
> directory here: {noformat} doneDirPrefixPath =
> FileContext.getFileContext(conf).makeQualified(new
> Path(userDoneDirStr));
> mkdir(doneDirFS, doneDirPrefixPath, new FsPermission(
>
> JobHistoryUtils.HISTORY_INTERMEDIATE_USER_DIR_PERMISSIONS));{noformat} which
> is hardcoded to 770. But the summary, history, and conf files under this user
> dir are configurable via
> {{mapreduce.jobhistory.intermediate-done-dir.file.permission}}. So if the
> configured permissions has read/write/execute permissions for "other" users,
> they will still not have access to these files due to the 770 permission on
> the user dir.
> I see two options here:
> # Reuse {{mapreduce.jobhistory.intermediate-done-dir.file.permission}} as the
> permissions for the user dir
> # Create a new config for the user dir permissions, using 770 as the default
> The latter makes more sense to me.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6860) User intermediate-done-dir permissions should use history file permissions configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated MAPREDUCE-6860:
-
Description:
Currently {{JobHistoryEventHandler}} creates the user intermediate-done-dir
directory here: {noformat} doneDirPrefixPath =
FileContext.getFileContext(conf).makeQualified(new
Path(userDoneDirStr));
mkdir(doneDirFS, doneDirPrefixPath, new FsPermission(
JobHistoryUtils.HISTORY_INTERMEDIATE_USER_DIR_PERMISSIONS));{noformat} which is
hardcoded to 770. But the summary, history, and conf files under this user dir
are configurable via
{{mapreduce.jobhistory.intermediate-done-dir.file.permission}}. So if the
configured permissions has read/write/execute permissions for "other" users,
they will still not have access to these files due to the 770 permission on the
user dir.
I see two options here:
# Reuse {{mapreduce.jobhistory.intermediate-done-dir.file.permission}} as the
permissions for the user dir
# Create a new config for the user dir permissions, using 770 as the default
The latter makes more sense to me.
was:
Currently {{JobHistoryEventHandler}} creates the user intermediate-done-dir
directory here: {noformat} doneDirPrefixPath =
FileContext.getFileContext(conf).makeQualified(new
Path(userDoneDirStr));
mkdir(doneDirFS, doneDirPrefixPath, new FsPermission(
JobHistoryUtils.HISTORY_INTERMEDIATE_USER_DIR_PERMISSIONS));{noformat} which is
hardcoded to 770. But the summary, history, and conf files under this user dir
are configurable via
{{mapreduce.jobhistory.intermediate-done-dir.file.permission}}. So if the
configured permissions has read/write/execute permissions for "other" users,
they will still not have access to these files due to the 770 permission on the
user dir.
I see two options here:
# Reuse {{mapreduce.jobhistory.intermediate-done-dir.file.permission}} as the
permissions for the user dir
# Create a new config for the user dir permissions, using 770 as the default
> User intermediate-done-dir permissions should use history file permissions
> configuration
>
>
> Key: MAPREDUCE-6860
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6860
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jonathan Hung
>
> Currently {{JobHistoryEventHandler}} creates the user intermediate-done-dir
> directory here: {noformat} doneDirPrefixPath =
> FileContext.getFileContext(conf).makeQualified(new
> Path(userDoneDirStr));
> mkdir(doneDirFS, doneDirPrefixPath, new FsPermission(
>
> JobHistoryUtils.HISTORY_INTERMEDIATE_USER_DIR_PERMISSIONS));{noformat} which
> is hardcoded to 770. But the summary, history, and conf files under this user
> dir are configurable via
> {{mapreduce.jobhistory.intermediate-done-dir.file.permission}}. So if the
> configured permissions has read/write/execute permissions for "other" users,
> they will still not have access to these files due to the 770 permission on
> the user dir.
> I see two options here:
> # Reuse {{mapreduce.jobhistory.intermediate-done-dir.file.permission}} as the
> permissions for the user dir
> # Create a new config for the user dir permissions, using 770 as the default
> The latter makes more sense to me.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6860) User intermediate-done-dir permissions should use history file permissions configuration
Jonathan Hung created MAPREDUCE-6860:
Summary: User intermediate-done-dir permissions should use history
file permissions configuration
Key: MAPREDUCE-6860
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6860
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jonathan Hung
Currently {{JobHistoryEventHandler}} creates the user intermediate-done-dir
directory here: {noformat} doneDirPrefixPath =
FileContext.getFileContext(conf).makeQualified(new
Path(userDoneDirStr));
mkdir(doneDirFS, doneDirPrefixPath, new FsPermission(
JobHistoryUtils.HISTORY_INTERMEDIATE_USER_DIR_PERMISSIONS));{noformat} which is
hardcoded to 770. But the summary, history, and conf files under this user dir
are configurable via
{{mapreduce.jobhistory.intermediate-done-dir.file.permission}}. So if the
configured permissions has read/write/execute permissions for "other" users,
they will still not have access to these files due to the 770 permission on the
user dir.
I see two options here:
# Reuse {{mapreduce.jobhistory.intermediate-done-dir.file.permission}} as the
permissions for the user dir
# Create a new config for the user dir permissions, using 770 as the default
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
