Syed Shameerur Rahman created HADOOP-19309:
----------------------------------------------
Summary: S3A CopyFromLocalFile operation fails when the source
file does not start with file scheme.
Key: HADOOP-19309
URL: https://issues.apache.org/jira/browse/HADOOP-19309
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.3.6, 3.4.0
Reporter: Syed Shameerur Rahman
Assignee: Syed Shameerur Rahman
When the sourcePath does not contain any file scheme information, S3A
CopyFromLocalFile operation fails with the following exception stack trace.
{code:java}
at
org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.getFinalPath(CopyFromLocalOperation.java:360)
at
org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.uploadSourceFromFS(CopyFromLocalOperation.java:222)
at
org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.execute(CopyFromLocalOperation.java:169)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFromLocalFile$23(S3AFileSystem.java:4217)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2871)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2890)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.copyFromLocalFile(S3AFileSystem.java:4209)
{code}
Additionally the failure is seen only when
{color:#172b4d}*fs.s3a.optimized.copy.from.local.enabled* is enabled (which is
by default). This happens only when the local source file is given without any
file scheme for example : /tmp/file.txt instead of file:///tmp/file.txt.
{color}
{color:#172b4d}The proposal here is to add file scheme to the source if the
source path does not contain the same.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
