Exception while running distcp on YARN

2015-08-20 Thread Varun Sharma
Hi, I am getting the following error when trying to distcp a file from S3 to an HDFS cluster. The job is running on YARN. It seems the containers are not being setup appropriately. When I browse logs on node manager, it says the container was not found meaning that it never launched. I see the sam

Specifying replication factor & block size during distcp

2015-08-19 Thread Varun Sharma
Hi, I am running a Distcp programmatically from Hadoop cluster to another - using Hadoop 2.7 and distcp v2. I would like to set a custom block size and replication factor for my files. How can I achieve that ? Thanks ! Varun