I¹m not seeing all the files I¹m writing out to webdav from a bundle thread and I¹m losing files. It does a simple zip unpack. If I redirect the output to a non-webdav folder it¹s fine.
I am more invested in using Oliver¹s work here https://issues.apache.org/jira/browse/SLING-4223 as a long term solution, but I¹m still concerned that this doesn¹t work as I will be needing some WebDAV functionality. The results are also inconsistent. The output file set is never complete. It always fails in one of 3 ways: 1) loses files with no error 2) fails to update a timestamp 3) fails to create a directory Even if I use the same input over and over, the output is different every time. I¹m using a simple unzip loop. public void extractAll(File inputFile, File outputDir) throws IOException { FileInputStream ifs = new FileInputStream(inputFile.getAbsolutePath()); try { if(outputDir.canWrite() == false) { throw new IOException("No write permissions for: " + outputDir.getAbsolutePath()); } ZipInputStream zipInputStream = new ZipInputStream(ifs); ZipEntry zipEntry = zipInputStream.getNextEntry(); while (zipEntry != null) { if (!zipEntry.isDirectory()) { String entryName = zipEntry.getName(); String extractName = entryName; File newFile = new File(outputDir.getAbsolutePath() + "/" + extractName); File parentFile = newFile.getParentFile(); if (!parentFile.isDirectory()) { if(parentFile.mkdirs() == false) { throw new IOException("Failed to create parentFile folder: " + parentFile.getAbsolutePath()); } } FileOutputStream outputStream = new FileOutputStream(newFile); try { IOUtils.copy(zipInputStream, outputStream); if( ! newFile.exists()) { throw new IOException("Failed to extract file: " + newFile.getAbsolutePath()); } } finally { outputStream.close(); } if (zipEntry.getTime() > 0) { if( ! newFile.setLastModified(zipEntry.getTime()) ) { throw new IOException("Failed to update timestamp for file: " + newFile.getAbsolutePath()); } } } zipInputStream.closeEntry(); zipEntry = zipInputStream.getNextEntry(); } } finally { ifs.close(); } } } If I step through it, right after the: IOUtils.copy(zipInputStream, outputStream) ... outputStream.close(); the file is missing on the WebDAV filesystem. I filed a jira incident a few days ago showing the lock errors I see with DEBUG logging enabled. https://issues.apache.org/jira/browse/SLING-4222 With default logging there are no errors other than the SlingRequestProcessorImpl service: Resource /content/ACT/2014/2014_11_15/IPHONE/b_05_courrier.folio not found messages when the first write is attempted. Here¹s a log sample of some of the extraction. The final exception is from a later stage trying to read a file that was supposedly unpacked successfully. This is case 1) above, no errors thrown even though not all files were written. 09.12.2014 22:50:59.303 *INFO* [qtp1099493755-350] logs/request.log 09/Dec/2014:22:50:59 -0800 [1273] -> HEAD /content/ACT/2014/2014_11_15/IPHONE/b_15_fowler.folio HTTP/1.1 09.12.2014 22:50:59.307 *INFO* [127.0.0.1 [1418194259304] HEAD /content/ACT/2014/2014_11_15/IPHONE/b_15_fowler.folio HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/ACT/2014/2014_11_15/IPHONE/b_15_fowler.folio not found 09.12.2014 22:50:59.308 *INFO* [qtp1099493755-350] logs/request.log 09/Dec/2014:22:50:59 -0800 [1273] <- 404 text/html 5ms 09.12.2014 22:50:59.308 *INFO* [qtp1099493755-350] logs/access.log 127.0.0.1 - admin 09/Dec/2014:22:50:59 -0800 "HEAD /content/ACT/2014/2014_11_15/IPHONE/b_15_fowler.folio HTTP/1.1" 404 3079 "-" "davfs2/1.4.6 neon/0.29.6" 09.12.2014 22:50:59.618 *INFO* [qtp1099493755-350] logs/request.log 09/Dec/2014:22:50:59 -0800 [1274] -> LOCK /content/ACT/2014/2014_11_15/IPHONE/b_03_edito.folio HTTP/1.1 09.12.2014 22:50:59.625 *INFO* [qtp1099493755-350] logs/request.log 09/Dec/2014:22:50:59 -0800 [1274] <- 201 text/xml; charset=UTF-8 6ms 09.12.2014 22:50:59.625 *INFO* [qtp1099493755-350] logs/access.log 127.0.0.1 - admin 09/Dec/2014:22:50:59 -0800 "LOCK /content/ACT/2014/2014_11_15/IPHONE/b_03_edito.folio HTTP/1.1" 201 400 "-" "davfs2/1.4.6 neon/0.29.6" 09.12.2014 22:50:59.625 *INFO* [qtp1099493755-350] logs/request.log 09/Dec/2014:22:50:59 -0800 [1275] -> HEAD /content/ACT/2014/2014_11_15/IPHONE/b_03_edito.folio HTTP/1.1 09.12.2014 22:50:59.629 *INFO* [127.0.0.1 [1418194259627] HEAD /content/ACT/2014/2014_11_15/IPHONE/b_03_edito.folio HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/ACT/2014/2014_11_15/IPHONE/b_03_edito.folio not found 09.12.2014 22:50:59.631 *INFO* [qtp1099493755-350] logs/request.log 09/Dec/2014:22:50:59 -0800 [1275] <- 404 text/html 6ms 09.12.2014 22:50:59.631 *INFO* [qtp1099493755-350] logs/access.log 127.0.0.1 - admin 09/Dec/2014:22:50:59 -0800 "HEAD /content/ACT/2014/2014_11_15/IPHONE/b_03_edito.folio HTTP/1.1" 404 3067 "-" "davfs2/1.4.6 neon/0.29.6" 09.12.2014 22:50:59.951 *INFO* [qtp1099493755-339] logs/request.log 09/Dec/2014:22:50:59 -0800 [1276] -> LOCK /content/ACT/2014/2014_11_15/IPHONE/b_05_courrier.folio HTTP/1.1 09.12.2014 22:50:59.958 *INFO* [qtp1099493755-339] logs/request.log 09/Dec/2014:22:50:59 -0800 [1276] <- 201 text/xml; charset=UTF-8 7ms 09.12.2014 22:50:59.958 *INFO* [qtp1099493755-339] logs/access.log 127.0.0.1 - admin 09/Dec/2014:22:50:59 -0800 "LOCK /content/ACT/2014/2014_11_15/IPHONE/b_05_courrier.folio HTTP/1.1" 201 400 "-" "davfs2/1.4.6 neon/0.29.6" 09.12.2014 22:50:59.959 *INFO* [qtp1099493755-339] logs/request.log 09/Dec/2014:22:50:59 -0800 [1277] -> HEAD /content/ACT/2014/2014_11_15/IPHONE/b_05_courrier.folio HTTP/1.1 09.12.2014 22:50:59.962 *INFO* [127.0.0.1 [1418194259961] HEAD /content/ACT/2014/2014_11_15/IPHONE/b_05_courrier.folio HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/ACT/2014/2014_11_15/IPHONE/b_05_courrier.folio not found 09.12.2014 22:50:59.964 *INFO* [qtp1099493755-339] logs/request.log 09/Dec/2014:22:50:59 -0800 [1277] <- 404 text/html 5ms 09.12.2014 22:50:59.965 *INFO* [qtp1099493755-339] logs/access.log 127.0.0.1 - admin 09/Dec/2014:22:50:59 -0800 "HEAD /content/ACT/2014/2014_11_15/IPHONE/b_05_courrier.folio HTTP/1.1" 404 3103 "-" "davfs2/1.4.6 neon/0.29.6" 09.12.2014 22:51:00.266 *ERROR* [pool-7-thread-13-<main queue>(incoming/file)] com.nim.ct.dam.ingest.jobs.ImportFileJobConsumer Exception: java.io.FileNotFoundException: /mnt/jcr/content/ACT/2014/2014_11_15/IPHONE/Folio.xml (No such file or directory) java.io.FileNotFoundException: /mnt/jcr/content/ACT/2014/2014_11_15/IPHONE/Folio.xml (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at com.nim.content.formats.folio.FolioArchiveReader.readFolio(FolioArchiveRead er.java:203) at com.nim.content.formats.folio.FolioArchiveReader.unpack(FolioArchiveReader. java:76) at com.nim.ct.dam.ingest.formats.InputFileInterchangeFolio.ingest(InputFileInt erchangeFolio.java:74) at com.nim.ct.dam.ingest.jobs.ImportFileJobConsumer.process(ImportFileJobConsu mer.java:140) at org.apache.sling.event.impl.jobs.JobConsumerManager$JobConsumerWrapper.proc ess(JobConsumerManager.java:512) at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$2.run(AbstractJobQ ueue.java:591) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) at java.lang.Thread.run(Thread.java:745) I¹m using the trunk head as of today. -Bruce
