[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-11724:
---
Status: Patch Available (was: Open)
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Attachments: HADOOP-11724.000.patch, HADOOP-11724.001.patch,
HADOOP-11724.002.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO mapred.CopyCommitter: Remove parent: null for
hdfs://localhost/
15/03/17 11:17:46 WARN mapred.CopyCommitter: Unable to cleanup temp files
java.lang.NullPointerException
at
[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-11724:
---
Status: Open (was: Patch Available)
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Attachments: HADOOP-11724.000.patch, HADOOP-11724.001.patch,
HADOOP-11724.002.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO mapred.CopyCommitter: Remove parent: null for
hdfs://localhost/
15/03/17 11:17:46 WARN mapred.CopyCommitter: Unable to cleanup temp files
java.lang.NullPointerException
at
[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-11724:
---
Attachment: HADOOP-11724.002.patch
Hi, [~yzhangal]
Thanks for reviewing it. I removed the logs from the patch. The two alternative
logs are similar, I think it would be better to defer writing logs in the
future.
Would it be good to you?
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Attachments: HADOOP-11724.000.patch, HADOOP-11724.001.patch,
HADOOP-11724.002.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO
[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-11724:
---
Attachment: HADOOP-11724.002.patch
Re-upload the patch to trigger jenkin job
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Attachments: HADOOP-11724.000.patch, HADOOP-11724.001.patch,
HADOOP-11724.002.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO mapred.CopyCommitter: Remove parent: null for
hdfs://localhost/
15/03/17 11:17:46 WARN mapred.CopyCommitter: Unable to cleanup temp files
[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-11724:
---
Attachment: (was: HADOOP-11724.002.patch)
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Attachments: HADOOP-11724.000.patch, HADOOP-11724.001.patch,
HADOOP-11724.002.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO mapred.CopyCommitter: Remove parent: null for
hdfs://localhost/
15/03/17 11:17:46 WARN mapred.CopyCommitter: Unable to cleanup temp files
java.lang.NullPointerException
at
[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-11724:
---
Resolution: Fixed
Fix Version/s: 2.8.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Fix For: 2.8.0
Attachments: HADOOP-11724.000.patch, HADOOP-11724.001.patch,
HADOOP-11724.002.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO mapred.CopyCommitter: Remove parent: null for
hdfs://localhost/
15/03/17 11:17:46 WARN
[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-11724:
---
Attachment: HADOOP-11724.001.patch
Thanks a lot [~yzhangal]. It is a great suggestion. I updated the patch for
your comments. Would you mind take another look?
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Attachments: HADOOP-11724.000.patch, HADOOP-11724.001.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO mapred.CopyCommitter: Remove parent: null for
hdfs://localhost/
15/03/17 11:17:46 WARN
[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-11724:
---
Attachment: HADOOP-11724.000.patch
Check whether parent directory is {{null}} in
{{CopyCommitter#cleanupTempFiles}}.
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Attachments: HADOOP-11724.000.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO mapred.CopyCommitter: Remove parent: null for
hdfs://localhost/
15/03/17 11:17:46 WARN mapred.CopyCommitter: Unable to cleanup temp files
java.lang.NullPointerException
[jira] [Updated] (HADOOP-11724) DistCp throws NPE when the target directory is root.
[
https://issues.apache.org/jira/browse/HADOOP-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-11724:
---
Status: Patch Available (was: Open)
DistCp throws NPE when the target directory is root.
Key: HADOOP-11724
URL: https://issues.apache.org/jira/browse/HADOOP-11724
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
Attachments: HADOOP-11724.000.patch
Distcp throws NPE when the target directory is root. It is due to
{{CopyCommitter#cleanupTempFiles}} attempts to delete parent directory of
root, which is {{null}}:
{code}
$ hadoop distcp pom.xml hdfs://localhost/
15/03/17 11:17:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/17 11:17:45 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[pom.xml],
targetPath=hdfs://localhost/, targetPathExists=true, preserveRawXattrs=false}
15/03/17 11:17:45 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/03/17 11:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
15/03/17 11:17:45 INFO Configuration.deprecation: io.sort.factor is
deprecated. Instead, use mapreduce.task.io.sort.factor
15/03/17 11:17:45 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: number of splits:1
15/03/17 11:17:45 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: The url to track the job:
http://localhost:8080/
15/03/17 11:17:46 INFO tools.DistCp: DistCp job-id: job_local992233322_0001
15/03/17 11:17:46 INFO mapreduce.Job: Running job: job_local992233322_0001
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.tools.mapred.CopyCommitter
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Waiting for map tasks
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Starting task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree
currently is supported only on Linux.
15/03/17 11:17:46 INFO mapred.Task: Using ResourceCalculatorProcessTree :
null
15/03/17 11:17:46 INFO mapred.MapTask: Processing split:
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/fileList.seq:0+220
15/03/17 11:17:46 INFO output.FileOutputCommitter: File Output Committer
Algorithm version is 1
15/03/17 11:17:46 INFO mapred.CopyMapper: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.CopyMapper: Skipping copy of
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task:
Task:attempt_local992233322_0001_m_00_0 is done. And is in the process of
committing
15/03/17 11:17:46 INFO mapred.LocalJobRunner:
15/03/17 11:17:46 INFO mapred.Task: Task
attempt_local992233322_0001_m_00_0 is allowed to commit now
15/03/17 11:17:46 INFO output.FileOutputCommitter: Saved output of task
'attempt_local992233322_0001_m_00_0' to
file:/tmp/hadoop/mapred/staging/lei2046334351/.staging/_distcp-1889397390/_logs/_temporary/0/task_local992233322_0001_m_00
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Copying
file:/Users/lei/work/cloudera/s3a_cp_target/pom.xml to
hdfs://localhost/pom.xml
15/03/17 11:17:46 INFO mapred.Task: Task
'attempt_local992233322_0001_m_00_0' done.
15/03/17 11:17:46 INFO mapred.LocalJobRunner: Finishing task:
attempt_local992233322_0001_m_00_0
15/03/17 11:17:46 INFO mapred.LocalJobRunner: map task executor complete.
15/03/17 11:17:46 INFO mapred.CopyCommitter: Remove parent: null for
hdfs://localhost/
15/03/17 11:17:46 WARN mapred.CopyCommitter: Unable to cleanup temp files
java.lang.NullPointerException
at org.apache.hadoop.fs.Path.init(Path.java:104)
at
