[jira] [Commented] (HDFS-15072) HDFS MiniCluster fails to start when run in directory path with a %
[ https://issues.apache.org/jira/browse/HDFS-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010290#comment-17010290 ] Hudson commented on HDFS-15072: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17825 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17825/]) HDFS-15072. HDFS MiniCluster fails to start when run in directory path (aajisaka: rev a43c177f1d4c2b6149a2680dd23d91103eca3be0) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java > HDFS MiniCluster fails to start when run in directory path with a % > --- > > Key: HDFS-15072 > URL: https://issues.apache.org/jira/browse/HDFS-15072 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5, 3.3.0 > Environment: I encountered this on a Mac while running an HBase > minicluster that was using Hadoop 2.7.5. However, the code looks the same in > trunk so it likely affects most or all current versions. >Reporter: Geoffrey Jacoby >Assignee: Masatake Iwasaki >Priority: Minor > > FSVolumeImpl.initializeCacheExecutor calls Guava's ThreadPoolExecutorBuilder. > setNameFormat, passing in the String representation of the parent File. Guava > will take the String whole and pass it to String.format, which uses % as a > special character. That means that if parent.toString() contains a percentage > sign, followed by a character that's illegal to use as a formatter in > String.format(), you'll get an exception that stops the MiniCluster from > starting up. > I did not check to see if this would also happen on a normal DataNode daemon. > initializeCacheExecutor should escape the parent file name before passing it > in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15072) HDFS MiniCluster fails to start when run in directory path with a %
[
https://issues.apache.org/jira/browse/HDFS-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000919#comment-17000919
]
Masatake Iwasaki commented on HDFS-15072:
-
While I don't thinks its practical to use path contiang {{%}} for data dir, it
would be harmless to escape the path in
{{FSVolumeImpl#initializeCacheExecutor}} because it uses the path just for the
name of thread.
> HDFS MiniCluster fails to start when run in directory path with a %
> ---
>
> Key: HDFS-15072
> URL: https://issues.apache.org/jira/browse/HDFS-15072
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.5
> Environment: I encountered this on a Mac while running an HBase
> minicluster that was using Hadoop 2.7.5. However, the code looks the same in
> trunk so it likely affects most or all current versions.
>Reporter: Geoffrey Jacoby
>Priority: Minor
>
> FSVolumeImpl.initializeCacheExecutor calls Guava's ThreadPoolExecutorBuilder.
> setNameFormat, passing in the String representation of the parent File. Guava
> will take the String whole and pass it to String.format, which uses % as a
> special character. That means that if parent.toString() contains a percentage
> sign, followed by a character that's illegal to use as a formatter in
> String.format(), you'll get an exception that stops the MiniCluster from
> starting up.
> I did not check to see if this would also happen on a normal DataNode daemon.
> initializeCacheExecutor should escape the parent file name before passing it
> in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15072) HDFS MiniCluster fails to start when run in directory path with a %
[
https://issues.apache.org/jira/browse/HDFS-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000914#comment-17000914
]
Masatake Iwasaki commented on HDFS-15072:
-
{quote}FSVolumeImpl.initializeCacheExecutor calls Guava's
ThreadPoolExecutorBuilder. setNameFormat, passing in the String representation
of the parent File.
{quote}
This means the scope is not limited to MiniDFSCluster. DataNode does not allow
path containing {{%}} for {{dfs.datanode.data.dir}}.
{noformat}
$ grep -A1 dfs.datanode.data.dir etc/hadoop/hdfs-site.xml
dfs.datanode.data.dir
/tmp/data%-
$ bin/hdfs datanode
...(snip)
2019-12-20 21:48:51,970 WARN datanode.DataNode: Unexpected exception in block
pool Block pool (Datanode Uuid
e0b42c6d-b28a-4cd7-8e70-82d8a3a8faac) service to localhost/127.0.0.1:8020
java.util.DuplicateFormatFlagsException: Flags = '-'
at java.util.Formatter$Flags.parse(Formatter.java:4443)
at java.util.Formatter$FormatSpecifier.flags(Formatter.java:2640)
at java.util.Formatter$FormatSpecifier.(Formatter.java:2709)
at java.util.Formatter.parse(Formatter.java:2560)
at java.util.Formatter.format(Formatter.java:2501)
at java.util.Formatter.format(Formatter.java:2455)
at java.lang.String.format(String.java:2981)
at
com.google.common.util.concurrent.ThreadFactoryBuilder.format(ThreadFactoryBuilder.java:182)
at
com.google.common.util.concurrent.ThreadFactoryBuilder.setNameFormat(ThreadFactoryBuilder.java:70)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.initializeCacheExecutor(FsVolumeImpl.java:208)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.(FsVolumeImpl.java:183)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImplBuilder.build(FsVolumeImplBuilder.java:90)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.addVolume(FsDatasetImpl.java:458)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.(FsDatasetImpl.java:348)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)
...(snip)
/
SHUTDOWN_MSG: Shutting down DataNode at localhost/127.0.0.1
/
{noformat}
> HDFS MiniCluster fails to start when run in directory path with a %
> ---
>
> Key: HDFS-15072
> URL: https://issues.apache.org/jira/browse/HDFS-15072
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.5
> Environment: I encountered this on a Mac while running an HBase
> minicluster that was using Hadoop 2.7.5. However, the code looks the same in
> trunk so it likely affects most or all current versions.
>Reporter: Geoffrey Jacoby
>Priority: Minor
>
> FSVolumeImpl.initializeCacheExecutor calls Guava's ThreadPoolExecutorBuilder.
> setNameFormat, passing in the String representation of the parent File. Guava
> will take the String whole and pass it to String.format, which uses % as a
> special character. That means that if parent.toString() contains a percentage
> sign, followed by a character that's illegal to use as a formatter in
> String.format(), you'll get an exception that stops the MiniCluster from
> starting up.
> I did not check to see if this would also happen on a normal DataNode daemon.
> initializeCacheExecutor should escape the parent file name before passing it
> in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
