You might pipe ls -R into that command but you’ll want access to multiple machines to run a slew of parallel cp jobs. GNU parallel can use a hosts file to start remote jobs. Each of those jobs needs a supply of files to copy
> On Nov 16, 2018, at 12:13 PM, Michael A Nesmith <[email protected]> > wrote: > > Hi, everyone, > > I am new to both AWS and GNU Parallel software. > > Please pardon a beginner's questions. > > I need to copy/move files from my AWS Linux EC2 instance (Redhat 6.10) to an > S3 bucket that was previously created. > > Normally the command to copy a file (for example, messages) would look like > this: > > aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3 > > To verify the file is in the S3 bucket, I use the command: > > aws s3 ls hhs-cms-epl-epor-stg-predev-v3 | grep messages > > The problem is that I need to upload a lot of files (100,000 or so) and some > of them are quite big (1 GB to 5 GB), and it takes a long time to upload so > many files or very big ones. > I need to speed this process up. > > The questions are: > > 1) Would GNU parallel software speed up this copy process significantly? > > 2) What is the syntax of the command when using S3 buckets? > > > If I understand the man page and the beginner youtube page, to use the > parallel option one should use something similar to: > > parallel aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3 > > However the output I get is: > > # parallel aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3 > parallel: Warning: Input is read from the terminal. You either know what you > parallel: Warning: are doing (in which case: YOU ARE AWESOME!) or you forgot > parallel: Warning: ::: or :::: or to pipe data into parallel. If so > parallel: Warning: consider going through the tutorial: man parallel_tutorial > parallel: Warning: Press CTRL-D to exit. > [root@ixe-1a-nfs-10618 tmp2]# > > The file does not copy. > > Any advice on how to do this properly? > > Thank you in advance to any who have time to help. > > Yours, > Mike > > > > <Outlook-y0fjbxd4.png> > > Michael A. NeSmith• AWS UNIX Cloud Administrator > > C-HIT |Enterprise Portal Contract > > 301.799.7864 | [email protected] > > 10480 Little Patuxent Pkwy, Ste 800, Columbia MD 21044 > > CMMI ML Dev 3 & Srv 3 > > Splunk Certified > > > > “This electronic mail and any files transmitted within are confidential and > is intended solely for the individual or entity to whom it is addressed. This > message contains information that is privileged and is otherwise protected > from disclosure to anyone other than the intended recipient. Please notify > the sender immediately by e-mail if you have received this e-mail by mistake > and delete this e-mail from your system. If you are not the named addressee > you should not disseminate, distribute or copy this e-mail and you are > notified that disclosing, copying, distributing or taking any action in > reliance on the contents of this information is strictly prohibited.” > >
