Hi Samer,
it would look easier to me to just check the last write time, under
windows this is function GetFileTime:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms724320%28v=vs.85%29.aspx
under Linux, stat() should do. Building an md5 hash seems like a slower
version, but both couldn't catch
the case where an application opens a file, writes data, close it, opens
the file again, adds more data,
closes it again.... but neither would the locking mechanism or the file
handle approach help in that case.
So maybe checking the file write time would be best. At first you could
measure how frequently a file
is modified, e.g. you find out every minute. Then if you measure that
the file has not been modified for
10 minutes, it's assumed the file will no longer be modified. It
wouldn't catch the case that an application
is writing data to the file and they still remain in write cache with
the file being kept open but the application
is doing something different for a while. However, it seems it requires
some kind of "guessing" anyway what
the writing application is doing, especially if it would repeatedly
close and re-open the same file to append data.
Cheers,
Werner
On 19.07.2015 16:47, Samer Afach wrote:
Dear Werner:
Thanks for the suggestions. I'll keep these suggestions as last resort
because my program is cross-platform, so I'm trying to avoid such
low-level functions.
I'm thinking now of using an md5 hash of the file instead of the size.
Do you think that would suffice and work? The only problem with that
is that it could be a little expensive, but that's still OK if that's
the only problem because the files aren't really that big (a few
megabytes).
All the best,
Samer
On 19.07.2015 16:37, Werner Benger wrote:
Hi Samer,
if those applications are linking dynamically with HDF5, you could
interface there by providing a "patched" version of HDF5 that is
binary compatible, but still performs some file locking, possibly via
a modified VFD.
Just checking the file sizes won't be a sure check, in particular it
would not work for and application that does not append data but
modifies existing ones. You should be able to check the "last
modified" date on the file as well, and if the HDF5 file cannot be
opened by another application, then it's either corrupted or still
open and data written to it.
It should also be possible to find out which process has file handles
to a specific file open, both Linux and Windows seem to allow that,
just in an OS-specific way:
http://stackoverflow.com/questions/6583158/finding-open-file-descriptors-for-a-process-linux-c-code
https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
It may require your program to run under admin rights though.
Cheers,
Werner
On 19.07.2015 12:32, Samer Afach wrote:
Dear Werner:
Unfortunately I don't have the source of the writing program. It
could be anything actually; labview, C, C++, etc... . I'm working on
a collaborative program for collecting data, and my software is
supposed to upload the data to the main storage server for backup
and data analysis. Everyone is free to use any tools they want for
data acquisition.
Is there any solution to my problem that doesn't involve me having
authority on the writing program? Currently I'm using a method that
I invented, where I measure the size of the file multiple times, and
if the size doesn't change for some time, then that decide that the
file isn't being written... that's the only way I could come up
with, but this has nothing to do with HDF5.
All the best,
Samer
On 19.07.2015 12:23, Werner Benger wrote:
Hi Samer,
do you have full source code access to both programs, the one
writing the file and the one uploading it?
If so, would adding some file locking mechanism (
https://en.wikipedia.org/wiki/File_locking ) help your case?
Cheers,
Werner
On 19.07.2015 02:50, Samer Afach wrote:
Dear Werner:
Thank you for your response and your time.
I tried modifying the source code of h5check, but it's very
complicated and returns strings instead of error codes. Besides, I
tried to compile it by including the three files in the main c
file and it gave errors that don't seem to be easily trackable.
It'll take me lots of time to prepare something reliable out of it
with full testing. I wonder why the hdf group didn't implement
such a functionality in low language, I thought it exists and I
can't find it. :(
About my second question, apparently I wasn't clear on what I
really need and I was misunderstood. What I need is to check
whether the file I want to read is being accessed by some other
program. My problem is that the file I wanna deal with is being
written by some other program, and I want to initiate an upload
after it's done writing. So I would like to check whether the
other program is finished writing and closed it before I send it
to my upload queue.
All the best,
Samer
On 18.07.2015 23:11, Werner Benger wrote:
Hi Samer,
On 18.07.2015 16:23, Samer Afach wrote:
Dear pros:
I have two similar issues I would like to ask about, and I'd be
grateful if you could help me:
1- How can I check HDF5 file integrity in C/C++ (I prefer low
level C)? I know there's a software called h5check, but I need a
function to do that, not a software. In my application, it's not
an option to execute h5check from my system.
Is there any objection against you taking the source code of
h5check, rename the main function to h5check_main() and then call
it as C function from your code?
2- How can I do a quick check in C/C++ to see whether the file
is being written/modified/open for write? I looked in this link
<https://www.hdfgroup.org/HDF5/doc/H5.format.html#FileMetaData>
which discusses metadata, but there doesn't seem to be a clear
way through the C/C++ interface for me a to read the first two
bits that give me that information. Could you please help with
that with a simple example?
This sounds as if you want to know what the HDF5 library is
currently doing with a file? Well what you could do is to modify
the virtual file driver that you want to use, and let it keep
track of that happens with the file. It's the VFD that does the
actual writing of a file content, so you can trace its write()
call to keep track what exactly happens when. It's some effort of
course.
Cheers,
Werner
All the best,
Samer
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
