Transferring the small files with AFM + NFS over high latency networks is always a challenge. For example, for each small file replication AFM performs a lookup, create, write and set mtime operation. If the latency is 10ms, replication of each file takes minimum (10 * 4 = 40 ms) amount of time. AFM is not a network acceleration tool and also it does not use compression. If the file sizes are big, AFM parallel IO and parallel mounts feature can be used. Aspera can be used to transfer the small files over high latency network with better utilization of the network bandwidth.
https://www.ibm.com/support/knowledgecenter/no/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/b1lins_afmparalleldatatransferwithremotemounts.htm https://www.ibm.com/support/knowledgecenter/no/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_paralleldatatransfersafm.htm ~Venkat (vpuvv...@in.ibm.com) From: Chris Schlipalius <chris.schlipal...@pawsey.org.au> To: <gpfsug-discuss@spectrumscale.org> Date: 02/27/2020 05:54 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] AFM Alternative? Aspera? Sent by: gpfsug-discuss-boun...@spectrumscale.org Maybe the following would assist? I do think tarring up files first is best, but you could always check out: http://www.redbooks.ibm.com/redpapers/pdfs/redp5527.pdf https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spectrumscaleug.org_wp-2Dcontent_uploads_2019_05_SSSD19DE-2DDay-2D2-2DB02-2DIntegration-2Dof-2DSpectrum-2DScale-2Dand-2DAspera-2DSync.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A&s=tRCxd4SimJH_eycqekhzM0Qp3TB3NtaIYWBvyQnrIiM&e= Aspera sync integration (non html links added for your use – how they don’t get scrubbed: www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B02-Integration-of-Spectrum-Scale-and-Aspera-Sync.pdf www.redbooks.ibm.com/redpapers/pdfs/redp5527.pdf ) Regards, Chris Schlipalius Team Lead, Data Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO) 1 Bryce Avenue Kensington WA 6151 Australia Tel +61 8 6436 8815 Email chris.schlipal...@pawsey.org.au Web www.pawsey.org.au < https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pawsey.org.au_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A&s=Xkm8VFy3l6nyD40yhONihsKcqmwRhy4SZyd0lwHf1GA&e= > On 26/2/20, 9:39 pm, "gpfsug-discuss-boun...@spectrumscale.org on behalf of gpfsug-discuss-requ...@spectrumscale.org" <gpfsug-discuss-boun...@spectrumscale.org on behalf of gpfsug-discuss-requ...@spectrumscale.org> wrote: Re: AFM Alternative? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A&s=mYK1ZsVgtsM6HntRMLPS49tKvEhhgGAdWF2qniyn9Ko&e=
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
