On Tue, 8 May 2012, Thomas Koch wrote:
There's a known issue with some PyLucene tests that fail on Windows - reported/discussed before - see http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/201104.mbox/<00 0d01cbfa7b$c60ca530$5225ef90$@de> While some tests have been fixed, some others still show errors / fail on windows, currently that's ERROR: test_removeDocument (__main__.PythonDirectoryTests) FAIL: testTiming (lia.indexing.CompoundVersusMultiFileIndexTest.CompoundVersusMultiFileIndexT est) I had a look at this and tried to figure out why the PythonDirectoryTests fail. A possible cause is a windows issue with os.unlink where in certain situations (timing issue!) a file that is going to be deleted is still locked by windows - which may happen because of some process being notified about file removal still holds a lock on it (sth. like an indexer or virus checker). Another reason could simply be that some tests keep files open... anyway: "On Windows, attempting to remove a file that is in use causes an exception to be raised." (from the Python doc on os.remove(path) and unlink(path)). The traceback below shows that the test in PythonDirectoryTests fails in deleteFile (where lock is on an index file like '_0.tis' or '_0.fdt'). There's been a similar problem with Python unit tests (on windows) as discussed here: http://bugs.python.org/issue7443 (can't see that they came to a solution though ... it's still open) I've added special hack for windows to the test_PythonDirectory and added a try-delete-wait-retry-loop there - this shows that it's probably not a timing issue (after 10 secs/retries, the file still cannot be deleted) - so it rather looks like the index files are still open! (couldn't find a bug in the python code though) Thus I tried to simply ignore the exception - the files are then *not* removed, but the test succeeds: ... indexing 98 indexing 99 failed to delete: testpyrepo\_0.tis failed to delete: testpyrepo\_0.nrm failed to delete: testpyrepo\_0.frq failed to delete: testpyrepo\_0.fdx failed to delete: testpyrepo\_0.prx failed to delete: testpyrepo\_0.fdt failed to delete: testpyrepo\_0.tis failed to delete: testpyrepo\_0.nrm failed to delete: testpyrepo\_0.frq failed to delete: testpyrepo\_0.fdx failed to delete: testpyrepo\_0.prx failed to delete: testpyrepo\_0.fdt ... ---------------------------------------------------- Ran 10 tests in 3.659s (The "failed to delete:" is a debug statement - to be removed) I could provide a "patch" that allows the test_PythonDirectory to pass on windows - if that's an accepted solution. Still I see possibility that the PythonDirectory has a bug (i.e. index files are not closed always) - on the other hand it could be timing issue or sth related to my environment (?)
I don't think that hiding failures "under the rug" is the right thing to do.
I wonder if anyone has used PythonDirectory yet (in production) or if someone else came across these issues on windows? (I'm not using PythonDirectory and haven't had similar lock issues except of when running the tests...) If someone could confirm that the test_PythonDirectory passes on his/her windows environment that would be good to know ,-)
PythonDirectory is meant as an example of "python extension" and tests that the extension is functional. It's not intended as a production-level
implementation. Andi..
Haven't had a look at testTiming yet. Regards, Thomas -- Details : ====================================================================== ERROR: test_removeDocument (__main__.PythonDirectoryTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "F:\Devel\workspaces\workspace.pylucene\pylucene-3.6.0-2\test\test_PyLucene. py", line 178, in test_removeDocument self.closeStore(store, searcher, reader) File "test_PythonDirectory.py", line 241, in closeStore arg.close() JavaError: org.apache.jcc.PythonException: (32, 'Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird', u'testpyrepo_0.tis') Traceback (most recent call last): File "test_PythonDirectory.py", line 181, in deleteFile os.unlink(os.path.join(self.path, name)) WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: u'testpyrepo\\_0.tis' Java stacktrace: ... at org.apache.pylucene.store.PythonDirectory.deleteFile(Native Method) at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:57 8) at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:504) at org.apache.lucene.index.IndexFileDeleter.close(IndexFileDeleter.java:377) at org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java:854) at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1520) at org.apache.lucene.index.IndexReader.commit(IndexReader.java:1504) at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:221) at org.apache.lucene.index.IndexReader.close(IndexReader.java:1538) ====================================================================== ERROR: test_removeDocument (__main__.PythonDirectoryTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_PythonDirectory.py", line 233, in tearDown shutil.rmtree(self.STORE_DIR) File "C:\Python27\lib\shutil.py", line 249, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "C:\Python27\lib\shutil.py", line 247, in rmtree os.remove(fullname) WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird: 'testpyrepo\\_0.fdt' ====================================================================== ... /cygdrive/c/Python27/python.exe samples/LuceneInAction/CompoundVersusMultiFileIndexTest.py F ====================================================================== FAIL: testTiming (lia.indexing.CompoundVersusMultiFileIndexTest.CompoundVersusMultiFileIndexT est) ---------------------------------------------------------------------- Traceback (most recent call last): File "F:\devel\workspaces\workspace.pylucene\pylucene-3.6.0-2\samples\LuceneInAct ion\lia\indexing\CompoundVersusMultiFileIndexTest.py", line 62, in testTiming self.assert_(cTiming > mTiming) AssertionError: False is not true ---------------------------------------------------------------------- Ran 1 test in 3.688s FAILED (failures=1)