Hi,
I have written an application(in JAVA and C) which reads contents
from Standard input (stdin) and writes to a file in HDFS (filesystem).
In HDFS the contents of the file are not committed(or seen) until the
file is closed explicitly. The problem is my application is run on unix
environment and can be closed by external factors. The external factors
include termination by another process, raising of signals etc..,
The application will be terminated and will not be able to close the
file properly, Hence the contents will not be written to the file in
HDFS filesystem.
When a process terminates due to signal interrupts or other factors,
there is a chance to handle graceful shutdown through signal handlers in
C and shutdown hooks in Java. However since application links to jvm and
libhdfs (in C) the application signal handler is never called.
In Java the shutdown hook is called, however the application is not able
to close the file properly, because the filesystem connection is closed
due to the interrupts, before the file can be closed.
I am not able to resolve the problem, I have tried lots of options. I
would appreciate any help and will be happy to provide more information.
Thanks
Suresh