When implementing the producer/consumer pattern within a file system I
use these guidelines.
Pick a directory writeable by the producer, readable by the consumer
(this method only uses one directory to avoid file system boundary
issues). Have the producer write a file with a name that matches this
template:
B_<group_prefix><sequence_number>.<suffix>
Call it the "before" image. When the write is complete, rename the file
removing the B_ prefix. The consumer polls for files matching the pattern:
<group_prefix>[0-9]+\.<suffix>
The group prefix allows for multiple streams of files to coexist in the
same directory. Useful if the producer is on a remote host since only
one identity and writeable directory has to set up to enable multiple
streams.
If the sequence number is chosen to be fixed length and ascending then
it defines the processing order. After the consumer has processed the
file have it rename the file prefixing it with A_, the "after" image.
Select the suffix to be association table friendly for clicking on the
files in a GUI environment. The use of prefixes rather than appending a
".tmp" suffix for the before image leaves all three file states with the
correct suffix for the association table.
This approach has a number of useful properties.
The state is given by; ls -1. It's output groups the before, during and
after images. Within each group file are listed in processing order.
Reprocessing one or more files is achieved by renaming them, striping
the A_ prefix (much easier than "touching" a list of files with pauses
which is what happens if you use mtime).
Before image files that are "old" should be investigated and deleted
since their continued existence indicates an error. After image files
that are "old" can archived/deleted. Files without prefixes that become
"old" indicate that the consumer has stopped processing.
Works using FTP, ssh/scp, local processes or anything that has a
functional rename command and is dependant only on the atomicity of that
command.
--
Regards
