[EMBOSS] Jemboss unable to delete job directories if results.home is on NFS
Greetings, We had run into the following problem: if Jemboss results directory (results.home in jemboss.properties) is a directory on NFS, Jemboss is unable to delete job directories, either from Saved Results or from File Manager (in the case of the latter I first delete the files in the directory from the File Manager - no problem there). Here're the steps that reproduced the problem in our environment: 1. Run say "makeprotseq" in Jemboss. The resultant job directory contains files makeseq.fasta, .desc, and .finished. 2. Open File Manager, browse to the job directory in the Remote pane, and delete the file "makeseq.fasta". The file disappears, but when checking the contents of the directory on the server, it turns out that even though "makeseq.fasta" is no longer there, there's a file ".nfs000". Applying "lsof" on the file reveals that the it is being kept open by a "java" process corresponding to the Tomcat server. 3. Attempt to delete the directory from Jemboss File Manager. The directory disappears in File Manager view, but a check on the server side reveals that it's still there. Inside the directory, files .desc and .finished are gone but the file .nfs00... is still there. After a "refresh" in Jemboss File Manager the directory reappears. It was our understanding that the appearance of these ".nfs0..." files after the attempt to delete the result files was the standard behavior of NFS when there's an attempt to delete an open file. People smarter than me then were able to resolve the problem by adding calls "in.close()" after the "while" loops in the following methods in JembossServer.java show_acd show_saved_result list_saved_results loadFilesContent (in two places) update_result_status We would really appreciate if you could please review these changes and confirm whether they're indeed correct and necessary. With best regards, Daniel -- Daniel Rozenbaum Biocceleration, Inc. OCIO/ Office of Application Engineering & Development/ Patent System Division 600 Dulany St. Alexandria, VA 22314 ___ EMBOSS mailing list EMBOSS@lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/emboss
Re: [EMBOSS] Jemboss unable to delete job directories if results.home is on NFS
My apologies - the part of the previous email where I described the modifications we made that appear to have helped resolve this problem was completely incorrect. I'm attaching a version of JembossServer.java (originally from the 6.4.0 distribution) with our modifications preceded with comments starting with "// Biocceleration: " . To reiterate, these modifications were mostly guesswork - trying to identify possible places where an explicit close() could have helped resolve the issue. From: emboss-boun...@lists.open-bio.org [emboss-boun...@lists.open-bio.org] On Behalf Of Rozenbaum, Daniel (Biocceleration Inc) Sent: Friday, September 21, 2012 10:55 PM To: emboss@lists.open-bio.org Subject: [EMBOSS] Jemboss unable to delete job directories if results.home is on NFS Greetings, We had run into the following problem: if Jemboss results directory (results.home in jemboss.properties) is a directory on NFS, Jemboss is unable to delete job directories, either from Saved Results or from File Manager (in the case of the latter I first delete the files in the directory from the File Manager - no problem there). Here're the steps that reproduced the problem in our environment: 1. Run say "makeprotseq" in Jemboss. The resultant job directory contains files makeseq.fasta, .desc, and .finished. 2. Open File Manager, browse to the job directory in the Remote pane, and delete the file "makeseq.fasta". The file disappears, but when checking the contents of the directory on the server, it turns out that even though "makeseq.fasta" is no longer there, there's a file ".nfs000". Applying "lsof" on the file reveals that the it is being kept open by a "java" process corresponding to the Tomcat server. 3. Attempt to delete the directory from Jemboss File Manager. The directory disappears in File Manager view, but a check on the server side reveals that it's still there. Inside the directory, files .desc and .finished are gone but the file .nfs00... is still there. After a "refresh" in Jemboss File Manager the directory reappears. It was our understanding that the appearance of these ".nfs0..." files after the attempt to delete the result files was the standard behavior of NFS when there's an attempt to delete an open file. People smarter than me then were able to resolve the problem by adding calls "in.close()" after the "while" loops in the following methods in JembossServer.java show_acd show_saved_result list_saved_results loadFilesContent (in two places) update_result_status We would really appreciate if you could please review these changes and confirm whether they're indeed correct and necessary. With best regards, Daniel -- Daniel Rozenbaum Biocceleration, Inc. OCIO/ Office of Application Engineering & Development/ Patent System Division 600 Dulany St. Alexandria, VA 22314 ___ EMBOSS mailing list EMBOSS@lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/emboss JembossServer.java_added_close_attempts Description: JembossServer.java_added_close_attempts ___ EMBOSS mailing list EMBOSS@lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/emboss
Re: [EMBOSS] Jemboss unable to delete job directories if results.home is on NFS
Hi Daniel, Thanks for the patch. I will apply your changes to the latest CVS version of the file. If I don't misread your email, problem may have stayed because we have more streams not closed in other jemboss classes. The ones in JembossFileServer get/put_file methods would be the main reason for the .nfs files you reported. I remember seeing this problem long time ago but didn't manage to find out its reason, i think i probably blamed nfs and didn't spend time to check whether we might have anything wrong in jemboss. Mahmut the part of the previous email where I described the modifications we made that appear to have helped resolve this problem was completely incorrect. I'm attaching a version of JembossServer.java (originally from the 6.4.0 distribution) with our modifications preceded with comments starting with "// Biocceleration: " . To reiterate, these modifications were mostly guesswork - trying to identify possible places where an explicit close() could have helped resolve the issue. ___ EMBOSS mailing list EMBOSS@lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/emboss
Re: [EMBOSS] Jemboss unable to delete job directories if results.home is on NFS
Hi Daniel, I was able to produce .nfs files by following the steps you described. Closing the open streams in JembossServer class looks fixes the problem. I don't understand why you thought your modifications were completely incorrect? I checked in you patch to CVS together with a fix to allow jemboss file manager to delete remote folders when they have hidden files like .desc and .finished. Mahmut ___ EMBOSS mailing list EMBOSS@lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/emboss
Re: [EMBOSS] Jemboss unable to delete job directories if results.home is on NFS
Hi Mahmut, Sorry about that confusion - the file I attached to my previous email in this thread did contain the correct fixes. After rebuilding with the latest version of JembossServer.java, everything appears to be working properly. -Daniel From: Mahmut Uludag [ulu...@ebi.ac.uk] Sent: Sunday, September 23, 2012 9:33 AM To: Rozenbaum, Daniel (Biocceleration Inc) Cc: emboss@lists.open-bio.org Subject: Re: [EMBOSS] Jemboss unable to delete job directories if results.home is on NFS Hi Daniel, I was able to produce .nfs files by following the steps you described. Closing the open streams in JembossServer class looks fixes the problem. I don't understand why you thought your modifications were completely incorrect? I checked in you patch to CVS together with a fix to allow jemboss file manager to delete remote folders when they have hidden files like .desc and .finished. Mahmut ___ EMBOSS mailing list EMBOSS@lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/emboss