vibhatha commented on PR #42038: URL: https://github.com/apache/arrow/pull/42038#issuecomment-2156875072
> Ummm.... > > There is a JUnit 5 bug on Windows (Server 2022) with JDK 11, specifically in the CI used by Arrow. > > * [@TempDir directory cannot be deleted on Windows with Java 11 junit-team/junit5#2811](https://github.com/junit-team/junit5/issues/2811) > > * https://github.com/apache/arrow/actions/runs/9437309980/job/25993017637?pr=42038#step:5:2647 > > > Should we consider a rollback because of this, or is there another solution we could explore? > error message > > ``` > Error: Tests run: 5, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 3.553 s <<< FAILURE! -- in org.apache.arrow.adapter.avro.AvroToArrowIteratorTest > [INFO] --- maven-surefire-plugin:3.2.5:test (default-test) @ arrow-jdbc --- > Error: org.apache.arrow.adapter.avro.AvroToArrowIteratorTest.testArrayType -- Time elapsed: 0.125 s <<< ERROR! > java.io.IOException: Failed to delete temp directory D:\a\arrow\arrow\java\adapter\avro\target\junit4444716940079148427. The following paths could not be deleted (see suppressed exceptions for details): <root>, test.avro > at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) > at java.base/java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:395) > at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258) > at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258) > at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) > at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) > at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) > at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) > at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) > Suppressed: java.nio.file.DirectoryNotEmptyException: D:\a\arrow\arrow\java\adapter\avro\target\junit4444716940079148427 > at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:271) > at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105) > at java.base/java.nio.file.Files.delete(Files.java:1142) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2743) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2797) > ... 13 more > Suppressed: java.nio.file.FileSystemException: D:\a\arrow\arrow\java\adapter\avro\target\junit4444716940079148427\test.avro: The process cannot access the file because it is being used by another process. > > at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) > at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) > at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) > at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:274) > at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105) > at java.base/java.nio.file.Files.delete(Files.java:1142) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2725) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2797) > ... 13 more > Suppressed: java.nio.file.FileSystemException: D:\a\arrow\arrow\java\adapter\avro\target\junit4444716940079148427\test.avro: The process cannot access the file because it is being used by another process. > > ... 21 more > ``` > > UPDATE: > > I tried two kinds of approaches > #### Using `try-with-resources` Statements for `FileOutputStream` and `FileInputStream` > > ```java > try (FileOutputStream outStream = new FileOutputStream(dataFile); > FileInputStream inStream = new FileInputStream(dataFile)) { > BinaryEncoder encoder = new EncoderFactory().directBinaryEncoder(outStream, null); > DatumWriter<Object> writer = new GenericDatumWriter<>(schema); > for (Object value : data) { > writer.write(value, encoder); > } > outStream.flush(); > > BinaryDecoder decoder = new DecoderFactory().directBinaryDecoder(inStream, null); > return AvroToArrow.avroToArrow(schema, decoder, config); > } > ``` > > #### Separating `FileOutputStream` and `FileInputStream` Usage to Avoid Concurrent Access > > ```java > protected VectorSchemaRoot writeAndRead(Schema schema, List data) throws Exception { > File dataFile = new File(TMP, "test.avro"); > > try (FileOutputStream out = new FileOutputStream(dataFile)) { > BinaryEncoder encoder = new EncoderFactory().directBinaryEncoder(out, null); > DatumWriter<Object> writer = new GenericDatumWriter<>(schema); > for (Object value : data) { > writer.write(value, encoder); > } > out.flush(); > } > > try (FileInputStream in = new FileInputStream(dataFile)) { > BinaryDecoder decoder = new DecoderFactory().directBinaryDecoder(in, null); > return AvroToArrow.avroToArrow(schema, decoder, config); > } > } > ``` @llama90 thanks for taking a very thorough assessment on this. I think the [issue](https://github.com/apache/arrow/actions/runs/9437483721/job/25993406708?pr=42038#step:5:10715) still persists. Furthermore looking into the JUNIT5 bug report, it seems that there isn't a solid fix yet. So my suggestion would be to fix it once that's resolved. Although, we can create two sub-issues for this ticket, one for functional conversion (orc) and other for non-functional bits. So that we can track the failing one later. But I have doubts about that approach. cc @lidavidm Appreciate your feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
