[
https://issues.apache.org/jira/browse/TEZ-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013442#comment-14013442
]
Mohammad Kamrul Islam commented on TEZ-1106:
--------------------------------------------
{quote}
I believe a slight modification of Option 2 should work. Support both
create_dir function and a get_dir functions. createDir() should be called once
and only once in the client. It will create the dir if it doesn't exist and do
the necessary permission checks ( logging messages when either the dir is not
present or perms need to be reset). createDir() could also return the fully
resolved path back to the user. getDir() should be called after this point in
the AM and the path obtained re-used elsewhere as needed.
{quote}
Q1: Where/when to log the message? in createDir() or getDir() or both?
Q2: Will getDir() checks the existence of the path and creates if needed? OR it
will just return the path assuming it was already created.
Q3: Important: path permission check. Will it be little tricky to compare the
file's current permission with TEZ_AM_DIR_PERMISSION/TEZ_AM_FILE_PERMISSION?
FSPermission.equals() checks the *exact* same permission. In reality, umask
could play some role in determining the final permission, Is it correct?
Q4: How to test the whole thing? In addition to unit test, I can run some
examples code to see if it works fine.
> Tez framework should use a unique subdir when creating new files in staging
> -----------------------------------------------------------------------------
>
> Key: TEZ-1106
> URL: https://issues.apache.org/jira/browse/TEZ-1106
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Attachments: TEZ-1106.1.patch, TEZ-1106.2.patch, TEZ-1106.3.patch,
> TEZ-1106.4.patch
>
>
> Currently the files are created in different sub-directories. It is hard to
> manage and cleanup at the end.
> The proposal is to create a new subdir : $STAGE_DIR/<APP_ID>/
> All recovery files will go under : $STAGE_DIR/<APP_ID>/recovery/<attemp_num>/
> All confs will go under: $STAGE_DIR/<APP_ID>/conf/
> All dagplans will go: $STAGE_DIR/<APP_ID>/dag_id/plan/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
