[jira] [Updated] (HADOOP-12830) Bash environment for quick command operations
[
https://issues.apache.org/jira/browse/HADOOP-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12830:
--
Attachment: HADOOP-12830.003.patch
I removed the test case about multi bytes characters. The test seems to be
passed in specific environments.
> Bash environment for quick command operations
> -
>
> Key: HADOOP-12830
> URL: https://issues.apache.org/jira/browse/HADOOP-12830
> Project: Hadoop Common
> Issue Type: New Feature
> Components: bin
>Reporter: Kazuho Fujii
>Assignee: Kazuho Fujii
> Attachments: HADOOP-12830.001.patch, HADOOP-12830.002.patch,
> HADOOP-12830.003.patch
>
>
> Hadoop file system shell commands are slow. This issue is about building a
> shell environment for quick command operations.
> Previously an interactive shell is tried to build in HADOOP-6541. But, it
> seems to be poor because users are used to powerful shells like bash. This
> issue is not about creating a new shell, but just opening a new bash process.
> Therefore, user can operate commands as before.
> {code}
> fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
> fjk@x240 hadoop> hadoop fs -ls /
> Found 2 items
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file1
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file2
> {code}
> The shell has a mini daemon process that is living until the shell is closed.
> The hadoop fs command delegates the operation to the daemon. They communicate
> with named pipes. The daemon conducts the operation and returns the result to
> the command.
> In this shell the hadoop fs commands operation becomes quick. In a local
> environment, "hadoop fs -ls" command is about 100 times faster than the
> normal command.
> {code}
> fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
> real 0m0.021s
> user 0m0.003s
> sys 0m0.011s
> {code}
> Using bash's function, commands and file names are automatically completed.
> {code}
> fjk@x240 hadoop> hadoop fs -ch
> -checksum -chgrp -chmod -chown
> fjk@x240 hadoop> hadoop fs -ls /file
> /file1 /file2 /file3
> {code}
> Additionally, we can make equivalents with bash build-in commands, e.g., cd,
> umask. In this shell, they can work because the daemon remembers the state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12830) Bash environment for quick command operations
[
https://issues.apache.org/jira/browse/HADOOP-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12830:
--
Attachment: HADOOP-12830.002.patch
Hi,
I attached the second patch. Differences from the previous patch is as
following:
- Follow the style guide, shellcheck, checkstyle, findbugs
- Partly merge hadoop-shell.sh into the main hadoop script
- Do not use flock command
- Enable to work stdin such as hadoop fs -put - /
- Enable to work in the batch mode
- Add unit tests
The first patch calls hadoop-shell.sh twice as executable and as bashrc. I
rethink this is strange. So, the new patch creates the work files, starts the
daemon, starts bash, stops the daemon in the main hadoop script. The bashrc
part is separated because Bash requires and the script might be larger for the
future.
For avoiding two processes write into the fifo at the same time, we should lock
hadoop fs command. Previously, I used flock command because I could not find a
simple way to take atomic lock. (I guess this is the reason flock command
exists.) The new patch does not use flock, but takes a lock with a lock file. I
think it will works any UNIX.
The new patch works in the batch mode with specifying BASH_ENV. I think this is
useful.
I added unit tests. I tried to write integration test, but it is difficult for
me because there is no integration tests for the normal hadoop fs command.
The security issue is not solved yet. I have confidence of convenience of this
feature, but security sensitive users can not use this now. I do not have the
fundamental resolution. I am planning to temporarily print warning or block the
feature for Kerberos users. If you have a good resolution, could you tell me?
> Bash environment for quick command operations
> -
>
> Key: HADOOP-12830
> URL: https://issues.apache.org/jira/browse/HADOOP-12830
> Project: Hadoop Common
> Issue Type: New Feature
> Components: bin
>Reporter: Kazuho Fujii
>Assignee: Kazuho Fujii
> Attachments: HADOOP-12830.001.patch, HADOOP-12830.002.patch
>
>
> Hadoop file system shell commands are slow. This issue is about building a
> shell environment for quick command operations.
> Previously an interactive shell is tried to build in HADOOP-6541. But, it
> seems to be poor because users are used to powerful shells like bash. This
> issue is not about creating a new shell, but just opening a new bash process.
> Therefore, user can operate commands as before.
> {code}
> fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
> fjk@x240 hadoop> hadoop fs -ls /
> Found 2 items
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file1
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file2
> {code}
> The shell has a mini daemon process that is living until the shell is closed.
> The hadoop fs command delegates the operation to the daemon. They communicate
> with named pipes. The daemon conducts the operation and returns the result to
> the command.
> In this shell the hadoop fs commands operation becomes quick. In a local
> environment, "hadoop fs -ls" command is about 100 times faster than the
> normal command.
> {code}
> fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
> real 0m0.021s
> user 0m0.003s
> sys 0m0.011s
> {code}
> Using bash's function, commands and file names are automatically completed.
> {code}
> fjk@x240 hadoop> hadoop fs -ch
> -checksum -chgrp -chmod -chown
> fjk@x240 hadoop> hadoop fs -ls /file
> /file1 /file2 /file3
> {code}
> Additionally, we can make equivalents with bash build-in commands, e.g., cd,
> umask. In this shell, they can work because the daemon remembers the state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12830) Bash environment for quick command operations
[
https://issues.apache.org/jira/browse/HADOOP-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158879#comment-15158879
]
Kazuho Fujii commented on HADOOP-12830:
---
[~aw],
bq. Not necessarily. If Kerberos is enabled, keys are being stored locked in
memory, etc, then su isn't guaranteed to work.
Thanks for teaching. If so, the security is a critical issue. I do not want to
downgrade the security level with this feature. I try to search a solution.
I will rewrite the patch following your advice. Thanks a lot.
> Bash environment for quick command operations
> -
>
> Key: HADOOP-12830
> URL: https://issues.apache.org/jira/browse/HADOOP-12830
> Project: Hadoop Common
> Issue Type: New Feature
> Components: bin
>Reporter: Kazuho Fujii
>Assignee: Kazuho Fujii
> Attachments: HADOOP-12830.001.patch
>
>
> Hadoop file system shell commands are slow. This issue is about building a
> shell environment for quick command operations.
> Previously an interactive shell is tried to build in HADOOP-6541. But, it
> seems to be poor because users are used to powerful shells like bash. This
> issue is not about creating a new shell, but just opening a new bash process.
> Therefore, user can operate commands as before.
> {code}
> fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
> fjk@x240 hadoop> hadoop fs -ls /
> Found 2 items
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file1
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file2
> {code}
> The shell has a mini daemon process that is living until the shell is closed.
> The hadoop fs command delegates the operation to the daemon. They communicate
> with named pipes. The daemon conducts the operation and returns the result to
> the command.
> In this shell the hadoop fs commands operation becomes quick. In a local
> environment, "hadoop fs -ls" command is about 100 times faster than the
> normal command.
> {code}
> fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
> real 0m0.021s
> user 0m0.003s
> sys 0m0.011s
> {code}
> Using bash's function, commands and file names are automatically completed.
> {code}
> fjk@x240 hadoop> hadoop fs -ch
> -checksum -chgrp -chmod -chown
> fjk@x240 hadoop> hadoop fs -ls /file
> /file1 /file2 /file3
> {code}
> Additionally, we can make equivalents with bash build-in commands, e.g., cd,
> umask. In this shell, they can work because the daemon remembers the state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12830) Bash environment for quick command operations
[
https://issues.apache.org/jira/browse/HADOOP-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156780#comment-15156780
]
Kazuho Fujii commented on HADOOP-12830:
---
[~aw], thank you very much for many suggestions. I will rewrite the source code.
I am worried about security issues. But, I think the fifo itself can not be a
security hole. It has 0600 mode and the parent directory has 0700 mode.
{code}
chmod 700 ${HSH_TMP_DIR}
{code}
{code}
mkfifo --mode 600 ${fifo_names}
{code}
A malicious root user can attack more directly with "su -c".
I think the attack from root is unavoidable.
> Bash environment for quick command operations
> -
>
> Key: HADOOP-12830
> URL: https://issues.apache.org/jira/browse/HADOOP-12830
> Project: Hadoop Common
> Issue Type: New Feature
> Components: bin
>Reporter: Kazuho Fujii
>Assignee: Kazuho Fujii
> Attachments: HADOOP-12830.001.patch
>
>
> Hadoop file system shell commands are slow. This issue is about building a
> shell environment for quick command operations.
> Previously an interactive shell is tried to build in HADOOP-6541. But, it
> seems to be poor because users are used to powerful shells like bash. This
> issue is not about creating a new shell, but just opening a new bash process.
> Therefore, user can operate commands as before.
> {code}
> fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
> fjk@x240 hadoop> hadoop fs -ls /
> Found 2 items
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file1
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file2
> {code}
> The shell has a mini daemon process that is living until the shell is closed.
> The hadoop fs command delegates the operation to the daemon. They communicate
> with named pipes. The daemon conducts the operation and returns the result to
> the command.
> In this shell the hadoop fs commands operation becomes quick. In a local
> environment, "hadoop fs -ls" command is about 100 times faster than the
> normal command.
> {code}
> fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
> real 0m0.021s
> user 0m0.003s
> sys 0m0.011s
> {code}
> Using bash's function, commands and file names are automatically completed.
> {code}
> fjk@x240 hadoop> hadoop fs -ch
> -checksum -chgrp -chmod -chown
> fjk@x240 hadoop> hadoop fs -ls /file
> /file1 /file2 /file3
> {code}
> Additionally, we can make equivalents with bash build-in commands, e.g., cd,
> umask. In this shell, they can work because the daemon remembers the state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12830) Bash environment for quick command operations
[
https://issues.apache.org/jira/browse/HADOOP-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12830:
--
Attachment: HADOOP-12830.001.patch
> Bash environment for quick command operations
> -
>
> Key: HADOOP-12830
> URL: https://issues.apache.org/jira/browse/HADOOP-12830
> Project: Hadoop Common
> Issue Type: New Feature
> Components: bin
>Reporter: Kazuho Fujii
>Assignee: Kazuho Fujii
> Attachments: HADOOP-12830.001.patch
>
>
> Hadoop file system shell commands are slow. This issue is about building a
> shell environment for quick command operations.
> Previously an interactive shell is tried to build in HADOOP-6541. But, it
> seems to be poor because users are used to powerful shells like bash. This
> issue is not about creating a new shell, but just opening a new bash process.
> Therefore, user can operate commands as before.
> {code}
> fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
> fjk@x240 hadoop> hadoop fs -ls /
> Found 2 items
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file1
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file2
> {code}
> The shell has a mini daemon process that is living until the shell is closed.
> The hadoop fs command delegates the operation to the daemon. They communicate
> with named pipes. The daemon conducts the operation and returns the result to
> the command.
> In this shell the hadoop fs commands operation becomes quick. In a local
> environment, "hadoop fs -ls" command is about 100 times faster than the
> normal command.
> {code}
> fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
> real 0m0.021s
> user 0m0.003s
> sys 0m0.011s
> {code}
> Using bash's function, commands and file names are automatically completed.
> {code}
> fjk@x240 hadoop> hadoop fs -ch
> -checksum -chgrp -chmod -chown
> fjk@x240 hadoop> hadoop fs -ls /file
> /file1 /file2 /file3
> {code}
> Additionally, we can make equivalents with bash build-in commands, e.g., cd,
> umask. In this shell, they can work because the daemon remembers the state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12830) Bash environment for quick command operations
[
https://issues.apache.org/jira/browse/HADOOP-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12830:
--
Attachment: (was: HADOOP-12830.001.path)
> Bash environment for quick command operations
> -
>
> Key: HADOOP-12830
> URL: https://issues.apache.org/jira/browse/HADOOP-12830
> Project: Hadoop Common
> Issue Type: New Feature
> Components: bin
>Reporter: Kazuho Fujii
>Assignee: Kazuho Fujii
> Attachments: HADOOP-12830.001.patch
>
>
> Hadoop file system shell commands are slow. This issue is about building a
> shell environment for quick command operations.
> Previously an interactive shell is tried to build in HADOOP-6541. But, it
> seems to be poor because users are used to powerful shells like bash. This
> issue is not about creating a new shell, but just opening a new bash process.
> Therefore, user can operate commands as before.
> {code}
> fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
> fjk@x240 hadoop> hadoop fs -ls /
> Found 2 items
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file1
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file2
> {code}
> The shell has a mini daemon process that is living until the shell is closed.
> The hadoop fs command delegates the operation to the daemon. They communicate
> with named pipes. The daemon conducts the operation and returns the result to
> the command.
> In this shell the hadoop fs commands operation becomes quick. In a local
> environment, "hadoop fs -ls" command is about 100 times faster than the
> normal command.
> {code}
> fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
> real 0m0.021s
> user 0m0.003s
> sys 0m0.011s
> {code}
> Using bash's function, commands and file names are automatically completed.
> {code}
> fjk@x240 hadoop> hadoop fs -ch
> -checksum -chgrp -chmod -chown
> fjk@x240 hadoop> hadoop fs -ls /file
> /file1 /file2 /file3
> {code}
> Additionally, we can make equivalents with bash build-in commands, e.g., cd,
> umask. In this shell, they can work because the daemon remembers the state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12830) Bash environment for quick command operations
[
https://issues.apache.org/jira/browse/HADOOP-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12830:
--
Attachment: HADOOP-12830.001.path
Hi,
I attached a prototype.
Note that it has some issues:
- stdin of hadoop command does not work
- no test code
I tested this on Ubuntu 14.04. By using a Linux' own command (flock), it would
not work on Mac.
I built this for using by myself, but I want to hear whether other users want
it. As HADOOP-6541 remains opening, I'm not sure this feature should be
included Hadoop's code base or not.
If interested, could you give me a comment.
Regard,
> Bash environment for quick command operations
> -
>
> Key: HADOOP-12830
> URL: https://issues.apache.org/jira/browse/HADOOP-12830
> Project: Hadoop Common
> Issue Type: New Feature
> Components: bin
>Reporter: Kazuho Fujii
>Assignee: Kazuho Fujii
> Attachments: HADOOP-12830.001.path
>
>
> Hadoop file system shell commands are slow. This issue is about building a
> shell environment for quick command operations.
> Previously an interactive shell is tried to build in HADOOP-6541. But, it
> seems to be poor because users are used to powerful shells like bash. This
> issue is not about creating a new shell, but just opening a new bash process.
> Therefore, user can operate commands as before.
> {code}
> fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
> fjk@x240 hadoop> hadoop fs -ls /
> Found 2 items
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file1
> -rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file2
> {code}
> The shell has a mini daemon process that is living until the shell is closed.
> The hadoop fs command delegates the operation to the daemon. They communicate
> with named pipes. The daemon conducts the operation and returns the result to
> the command.
> In this shell the hadoop fs commands operation becomes quick. In a local
> environment, "hadoop fs -ls" command is about 100 times faster than the
> normal command.
> {code}
> fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
> real 0m0.021s
> user 0m0.003s
> sys 0m0.011s
> {code}
> Using bash's function, commands and file names are automatically completed.
> {code}
> fjk@x240 hadoop> hadoop fs -ch
> -checksum -chgrp -chmod -chown
> fjk@x240 hadoop> hadoop fs -ls /file
> /file1 /file2 /file3
> {code}
> Additionally, we can make equivalents with bash build-in commands, e.g., cd,
> umask. In this shell, they can work because the daemon remembers the state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12830) Bash environment for quick command operations
Kazuho Fujii created HADOOP-12830:
-
Summary: Bash environment for quick command operations
Key: HADOOP-12830
URL: https://issues.apache.org/jira/browse/HADOOP-12830
Project: Hadoop Common
Issue Type: New Feature
Components: bin
Reporter: Kazuho Fujii
Assignee: Kazuho Fujii
Hadoop file system shell commands are slow. This issue is about building a
shell environment for quick command operations.
Previously an interactive shell is tried to build in HADOOP-6541. But, it seems
to be poor because users are used to powerful shells like bash. This issue is
not about creating a new shell, but just opening a new bash process. Therefore,
user can operate commands as before.
{code}
fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
fjk@x240 hadoop> hadoop fs -ls /
Found 2 items
-rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file1
-rw-r--r-- 3 fjk supergroup 0 2016-02-21 00:26 /file2
{code}
The shell has a mini daemon process that is living until the shell is closed.
The hadoop fs command delegates the operation to the daemon. They communicate
with named pipes. The daemon conducts the operation and returns the result to
the command.
In this shell the hadoop fs commands operation becomes quick. In a local
environment, "hadoop fs -ls" command is about 100 times faster than the normal
command.
{code}
fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
real0m0.021s
user0m0.003s
sys 0m0.011s
{code}
Using bash's function, commands and file names are automatically completed.
{code}
fjk@x240 hadoop> hadoop fs -ch
-checksum -chgrp -chmod -chown
fjk@x240 hadoop> hadoop fs -ls /file
/file1 /file2 /file3
{code}
Additionally, we can make equivalents with bash build-in commands, e.g., cd,
umask. In this shell, they can work because the daemon remembers the state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12796) FsPermission#applyUMask drops the sticky bit
[
https://issues.apache.org/jira/browse/HADOOP-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12796:
--
Status: Patch Available (was: Open)
> FsPermission#applyUMask drops the sticky bit
>
>
> Key: HADOOP-12796
> URL: https://issues.apache.org/jira/browse/HADOOP-12796
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Kazuho Fujii
> Attachments: HADOOP-12796.001.patch
>
>
> The sticky bit has no effect specified at create and mkdirs.
> The sticky bit is dropped when umask is applied. {{applyUMask}} method does
> not take over the sticky bit from the original {{FsPermission}} class.
> In contrast, the sticky bit specified in POSIX' create or mkdir system call
> is always reflected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12796) FsPermission#applyUMask drops the sticky bit
[
https://issues.apache.org/jira/browse/HADOOP-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12796:
--
Attachment: HADOOP-12796.001.patch
I attached a patch. Please review it.
> FsPermission#applyUMask drops the sticky bit
>
>
> Key: HADOOP-12796
> URL: https://issues.apache.org/jira/browse/HADOOP-12796
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Kazuho Fujii
> Attachments: HADOOP-12796.001.patch
>
>
> The sticky bit has no effect specified at create and mkdirs.
> The sticky bit is dropped when umask is applied. {{applyUMask}} method does
> not take over the sticky bit from the original {{FsPermission}} class.
> In contrast, the sticky bit specified in POSIX' create or mkdir system call
> is always reflected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12796) FsPermission#applyUMask drops the sticky bit
Kazuho Fujii created HADOOP-12796:
-
Summary: FsPermission#applyUMask drops the sticky bit
Key: HADOOP-12796
URL: https://issues.apache.org/jira/browse/HADOOP-12796
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Kazuho Fujii
The sticky bit has no effect specified at create and mkdirs.
The sticky bit is dropped when umask is applied. {{applyUMask}} method does not
take over the sticky bit from the original {{FsPermission}} class.
In contrast, the sticky bit specified in POSIX' create or mkdir system call is
always reflected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12043) Display warning if defaultFs is not set when running fs commands.
[
https://issues.apache.org/jira/browse/HADOOP-12043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110717#comment-15110717
]
Kazuho Fujii commented on HADOOP-12043:
---
Hi,
This function is useful. But, I think "fs.defaultFs" in the warning message
should be the correct name "fs.defaultFS". A user who does not set fs.defaultFS
property or mistypes the property name gets confused when seeing the message.
At least I was misled.
How do you think?
> Display warning if defaultFs is not set when running fs commands.
> -
>
> Key: HADOOP-12043
> URL: https://issues.apache.org/jira/browse/HADOOP-12043
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.7.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8322.000.patch, HDFS-8322.001.patch,
> HDFS-8322.002.patch, HDFS-8322.003.patch, HDFS-8322.003.patch,
> HDFS-8322.004.patch, HDFS-8322.005.patch, HDFS-8322.006.patch
>
>
> Using {{LocalFileSystem}} is rarely the intention of running {{hadoop fs
> -ls}}.
> This JIRA proposes displaying a warning message if hadoop fs -ls is showing
> the local filesystem or using default fs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12543) Path#isRoot() returns true if the path is ".".
[
https://issues.apache.org/jira/browse/HADOOP-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988648#comment-14988648
]
Kazuho Fujii commented on HADOOP-12543:
---
Hi, thanks for the comment.
{{new Path(new Path("/tmp"), new Path(".."))}} equals {{new Path("/")}}. {{new
Path("/").isRoot()}} equals true. This is OK.
The behavior for relative path is misreading. {{new Path("..").getParent()}}
equals {{new Path(".")}}. {{new Path(".").isRoot()}} always return true in the
current implementation.
A Path class object doesn't know the working directory. It can not say whether
it repesents the root directory for relative path.
Just seeing the method name and the comment, I misunderstood it returns true
iif the path is "/".
> Path#isRoot() returns true if the path is ".".
> --
>
> Key: HADOOP-12543
> URL: https://issues.apache.org/jira/browse/HADOOP-12543
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Kazuho Fujii
> Attachments: HADOOP-12543.001.patch
>
>
> {{Path#isRoot()}} method is expected to return true if and only if the path
> represents the root of a file system. But, it returns true in the case where
> the path is ".". This is because {{getParent()}} method returns null when the
> path in URI is empty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12543) Path#isRoot() returns true if the path is ".".
[
https://issues.apache.org/jira/browse/HADOOP-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12543:
--
Description: {{Path#isRoot()}} method is expected to return true if and
only if the path represents the root of a file system. But, it returns true in
the case where the path is ".". This is because {{getParent()}} method returns
null when the path in URI is empty. (was: {{Path#isRoot()}} methods is
expected to return true if and only if the path represents the root of a file
system. But, it returns true in the case where the path is ".". This is because
{{getParent()}} method returns null when the path in URI is empty.)
> Path#isRoot() returns true if the path is ".".
> --
>
> Key: HADOOP-12543
> URL: https://issues.apache.org/jira/browse/HADOOP-12543
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Kazuho Fujii
> Attachments: HADOOP-12543.001.patch
>
>
> {{Path#isRoot()}} method is expected to return true if and only if the path
> represents the root of a file system. But, it returns true in the case where
> the path is ".". This is because {{getParent()}} method returns null when the
> path in URI is empty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12543) Path#isRoot() returns true if the path is ".".
[
https://issues.apache.org/jira/browse/HADOOP-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12543:
--
Attachment: HADOOP-12543.001.patch
I think the method should be like the attached patch. I tested it only in a
Linux environment.
> Path#isRoot() returns true if the path is ".".
> --
>
> Key: HADOOP-12543
> URL: https://issues.apache.org/jira/browse/HADOOP-12543
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Kazuho Fujii
> Attachments: HADOOP-12543.001.patch
>
>
> {{Path#isRoot()}} methods is expected to return true if and only if the path
> represents the root of a file system. But, it returns true in the case where
> the path is ".". This is because {{getParent()}} method returns null when the
> path in URI is empty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12543) Path#isRoot() returns true if the path is ".".
Kazuho Fujii created HADOOP-12543:
-
Summary: Path#isRoot() returns true if the path is ".".
Key: HADOOP-12543
URL: https://issues.apache.org/jira/browse/HADOOP-12543
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Kazuho Fujii
{{Path#isRoot()}} methods is expected to return true if and only if the path
represents the root of a file system. But, it returns true in the case where
the path is ".". This is because {{getParent()}} method returns null when the
path in URI is empty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Attachment: HADOOP-12045.006.patch [~cnauroth], thanks for testing the patch. I updated the patch again. I override new test methods and check if runnnig on Windows. Is it OK? Thank you for repeatedly helping. > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Assignee: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch, HADOOP-12045.002.patch, > HADOOP-12045.003.patch, HADOOP-12045.004-1.patch, HADOOP-12045.004-2.patch, > HADOOP-12045.005-1.patch, HADOOP-12045.005-2.patch, HADOOP-12045.006.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594665#comment-14594665 ] Kazuho Fujii commented on HADOOP-12045: --- Hi [~cnauroth]. Thank you for the comment. bq. 1. Nitpick: In testSetTimesSymlinkToDir, please use "dir" instead of "file" for the sub-directory name. I fixed it. Thanks for pointing it out. bq. 2. I think we also can update TestCopyPreserveFlag as part of this patch. Currently, it only checks mtime, but now it can check atime too. I updated it. I refactored the code at the same time. Is this acceptable? The just refactored version is included in HADOOP-12045.005-1.patch. > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Assignee: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch, HADOOP-12045.002.patch, > HADOOP-12045.003.patch, HADOOP-12045.004-1.patch, HADOOP-12045.004-2.patch, > HADOOP-12045.005-1.patch, HADOOP-12045.005-2.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Attachment: HADOOP-12045.005-2.patch HADOOP-12045.005-1.patch > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Assignee: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch, HADOOP-12045.002.patch, > HADOOP-12045.003.patch, HADOOP-12045.004-1.patch, HADOOP-12045.004-2.patch, > HADOOP-12045.005-1.patch, HADOOP-12045.005-2.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Attachment: HADOOP-12045.004-2.patch HADOOP-12045.004-1.patch Hi, [~cnauroth]. Thanks for a kind comment. I attached two more patches along your advise: * HADOOP-12045.004-1.patch * HADOOP-12045.004-2.patch . The former includes additional tests for the current setTimes method; the later includes the tests and the new setTimes method. We can confirm the backword-compatibility with the them. > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Assignee: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch, HADOOP-12045.002.patch, > HADOOP-12045.003.patch, HADOOP-12045.004-1.patch, HADOOP-12045.004-2.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12058) Link error in the commands manual site
[
https://issues.apache.org/jira/browse/HADOOP-12058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuho Fujii updated HADOOP-12058:
--
Attachment: HADOOP-12058.002.patch
Hi [~ajisakaa], thanks for the comment.
I fixed {{MapredCommand.md}}. I couldn't notice it.
> Link error in the commands manual site
> --
>
> Key: HADOOP-12058
> URL: https://issues.apache.org/jira/browse/HADOOP-12058
> Project: Hadoop Common
> Issue Type: Bug
> Components: site
>Affects Versions: 2.7.0
>Reporter: Kazuho Fujii
>Priority: Minor
> Attachments: HADOOP-12058.001.patch, HADOOP-12058.002.patch
>
>
> The links to DistCp or Hadoop Archives is broken in the commands reference
> site:
> [http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/CommandsManual.html].
> The destinations were moved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12058) Link error in the commands manual site
[ https://issues.apache.org/jira/browse/HADOOP-12058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12058: -- Description: The links to DistCp or Hadoop Archives are broken in the commands reference site: [http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/CommandsManual.html]. The destinations were moved. (was: The links to DistCp or Hadoop Archives is broken in the commands reference site: [http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/CommandsManual.html]. The destinations were moved. ) > Link error in the commands manual site > -- > > Key: HADOOP-12058 > URL: https://issues.apache.org/jira/browse/HADOOP-12058 > Project: Hadoop Common > Issue Type: Bug > Components: site >Affects Versions: 2.7.0 >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12058.001.patch, HADOOP-12058.002.patch > > > The links to DistCp or Hadoop Archives are broken in the commands reference > site: > [http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/CommandsManual.html]. > The destinations were moved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12058) Link error in the commands manual site
[ https://issues.apache.org/jira/browse/HADOOP-12058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12058: -- Attachment: HADOOP-12058.001.patch > Link error in the commands manual site > -- > > Key: HADOOP-12058 > URL: https://issues.apache.org/jira/browse/HADOOP-12058 > Project: Hadoop Common > Issue Type: Bug > Components: site >Affects Versions: 2.7.0 >Reporter: Kazuho Fujii >Priority: Trivial > Attachments: HADOOP-12058.001.patch > > > The links to DistCp or Hadoop Archives is broken in the commands reference > site: > [http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/CommandsManual.html]. > The destinations were moved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12058) Link error in the commands manual site
Kazuho Fujii created HADOOP-12058: - Summary: Link error in the commands manual site Key: HADOOP-12058 URL: https://issues.apache.org/jira/browse/HADOOP-12058 Project: Hadoop Common Issue Type: Bug Components: site Affects Versions: 2.7.0 Reporter: Kazuho Fujii Priority: Trivial The links to DistCp or Hadoop Archives is broken in the commands reference site: [http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/CommandsManual.html]. The destinations were moved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Attachment: HADOOP-12045.003.patch Excuse me. I made a mistake. > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch, HADOOP-12045.002.patch, > HADOOP-12045.003.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568974#comment-14568974 ] Kazuho Fujii commented on HADOOP-12045: --- Hi [~zxu], thanks for the comment. Certainly, the code repeating was not good. I attached a fixed patch. I have not test the patch at Windows yet. I don't use classes depending on a platform. So, I think it works. Anyway, we should test it on Windows platform. > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch, HADOOP-12045.002.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Attachment: HADOOP-12045.002.patch > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch, HADOOP-12045.002.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Status: Patch Available (was: Open) > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Status: Open (was: Patch Available) > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Status: Patch Available (was: Open) > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Attachment: HADOOP-12045.001.patch I attached a simple patch. The change in DeprecatedRawLocalFileStatus is for testing. > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Status: Open (was: Patch Available) > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > Attachments: HADOOP-12045.001.patch > > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
[ https://issues.apache.org/jira/browse/HADOOP-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuho Fujii updated HADOOP-12045: -- Status: Patch Available (was: Open) > Enable LocalFileSystem#setTimes to change atime > --- > > Key: HADOOP-12045 > URL: https://issues.apache.org/jira/browse/HADOOP-12045 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Kazuho Fujii >Priority: Minor > > LocalFileSystem#setTimes method can not change the last access time currently. > With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12045) Enable LocalFileSystem#setTimes to change atime
Kazuho Fujii created HADOOP-12045: - Summary: Enable LocalFileSystem#setTimes to change atime Key: HADOOP-12045 URL: https://issues.apache.org/jira/browse/HADOOP-12045 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Kazuho Fujii Priority: Minor LocalFileSystem#setTimes method can not change the last access time currently. With java.nio.file package in Java 7, we can implement the function easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
