Re: [Tutor] file move with wait period

[banda gunda]

Thanks Cameron (I hope I address your first name correctly).
Get Outlook for iOS<https://aka.ms/o0ukef>

From: Cameron Simpson <c...@cskk.id.au>
Sent: Friday, August 4, 2017 5:18:58 AM
To: banda gunda
Cc: tutor@python.org
Subject: Re: [Tutor] file move with wait period

On 03Aug2017 11:22, banda gunda <bandagu...@hotmail.com> wrote:
>Dear tutors,
>
>I am interested to move a set of files, periodically, from source_folder to 
>dest_folder.
>source_folder receives processed files from a ‘decompressing program’.  The 
>files are deposited from the ‘decompressing program’ at periodic random 
>intervals.  Furthermore some of the files dropped in the source_folder are 
>large, of typical size 800MB.  These files take time before it is completely 
>deposited in the source_folder.
>
>I could use shutil.move (source_folder, dest_folder).  But then how?

You'd normally use shutil.move(source_file, dest_file) i.e. move individual
files. Or os.rename, but that requires that the source and dest are the same
filesystem.

>The frequency of moving files from source_folder to dest_folder is flexible.  
>It could be once in 15 minutes (or even lengthier).
>Challenge for me:  How could the code include the wait statement to make sure 
>that the file in the source_folder is completely formed before attempting to 
>move to dest_folder?
>Thanks in advance for your comments/guidance.

That is the tricky thing; this shows up a lot, also with FTP upload locations
and so forth.

The problem, as you state, is knowing when the file is complete. There are 2
basic approaches: do not put a fil into the transfer directory before it is
complete, or to monitor the files for changes. On the premise that the file
will be being actively written until it is complete you could keep a list if
the files in the directory. For each name, record the file's size and
modification time. Wait for that to be unchanged "long enough"; you might then
decide it is ready to move.

Cheers,
Cameron Simpson <c...@cskk.id.au> (formerly c...@zip.com.au)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] file move with wait period

[Mats Wichmann]

On 08/03/2017 05:22 AM, banda gunda wrote:
> Dear tutors,
> 
> I am interested to move a set of files, periodically, from source_folder to 
> dest_folder.
> source_folder receives processed files from a ‘decompressing program’.  The 
> files are deposited from the ‘decompressing program’ at periodic random 
> intervals.  Furthermore some of the files dropped in the source_folder are 
> large, of typical size 800MB.  These files take time before it is completely 
> deposited in the source_folder.
> 
> I could use shutil.move (source_folder, dest_folder).  But then how?
> 
> The frequency of moving files from source_folder to dest_folder is flexible.  
> It could be once in 15 minutes (or even lengthier).
> Challenge for me:  How could the code include the wait statement to make sure 
> that the file in the source_folder is completely formed before attempting to 
> move to dest_folder?

Well, we cannot answer that question with the information you have given.

"How can you know" is actually the key question to explore. You need to
answer it before you worry about any Pythonic implementation details of
solving your problem.

- is the decompressor initiated by your program?  In that case there
will be techniques to maintain communication with it to find out when it
is done (perhaps as simple as waiting for it to quit-with-success).
- if it launches completely independently, how do you find out that it
has deposited files? Do you intend to just can periodically? Or is there
a way that the system causing the files to be generated can trigger
something that your program could be asked to be notified about when a
file is available?
- is it possible to convice the compressor to give files a certain
suffix or other recognizable pattern, but only when they are complete?
(downloader systems often work like this... while they're writing the
file, it has some temporary name, then when it finishes it is renamed to
the intended target name).

and so on.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] file move with wait period

[Cameron Simpson]

On 03Aug2017 11:22, banda gunda  wrote:

Dear tutors,

I am interested to move a set of files, periodically, from source_folder to 
dest_folder.
source_folder receives processed files from a ‘decompressing program’.  The 
files are deposited from the ‘decompressing program’ at periodic random 
intervals.  Furthermore some of the files dropped in the source_folder are 
large, of typical size 800MB.  These files take time before it is completely 
deposited in the source_folder.

I could use shutil.move (source_folder, dest_folder).  But then how?


You'd normally use shutil.move(source_file, dest_file) i.e. move individual 
files. Or os.rename, but that requires that the source and dest are the same 
filesystem.



The frequency of moving files from source_folder to dest_folder is flexible.  
It could be once in 15 minutes (or even lengthier).
Challenge for me:  How could the code include the wait statement to make sure 
that the file in the source_folder is completely formed before attempting to 
move to dest_folder?
Thanks in advance for your comments/guidance.


That is the tricky thing; this shows up a lot, also with FTP upload locations 
and so forth.


The problem, as you state, is knowing when the file is complete. There are 2 
basic approaches: do not put a fil into the transfer directory before it is 
complete, or to monitor the files for changes. On the premise that the file 
will be being actively written until it is complete you could keep a list if 
the files in the directory. For each name, record the file's size and 
modification time. Wait for that to be unchanged "long enough"; you might then 
decide it is ready to move.


Cheers,
Cameron Simpson  (formerly c...@zip.com.au)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] file move with wait period

[banda gunda]

Dear tutors,

I am interested to move a set of files, periodically, from source_folder to 
dest_folder.
source_folder receives processed files from a ‘decompressing program’.  The 
files are deposited from the ‘decompressing program’ at periodic random 
intervals.  Furthermore some of the files dropped in the source_folder are 
large, of typical size 800MB.  These files take time before it is completely 
deposited in the source_folder.

I could use shutil.move (source_folder, dest_folder).  But then how?

The frequency of moving files from source_folder to dest_folder is flexible.  
It could be once in 15 minutes (or even lengthier).
Challenge for me:  How could the code include the wait statement to make sure 
that the file in the source_folder is completely formed before attempting to 
move to dest_folder?
Thanks in advance for your comments/guidance.

Best,
Kumar.
+

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor