Yingyi Bu created ASTERIXDB-1987: ------------------------------------ Summary: Sporadic FileNotFound issue in tests Key: ASTERIXDB-1987 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1987 Project: Apache AsterixDB Issue Type: Bug Components: STO - Storage Reporter: Yingyi Bu Assignee: Abdullah Alamoudi
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/org.apache.asterix$asterix-app/1043/testReport/junit/org.apache.asterix.test.runtime/AqlExecutionLessParallelismIT/test_AqlExecutionLessParallelismIT_1385__temp_dataset__temp_primary_plus_ngram_flush_/ {noformat} Regression org.apache.asterix.test.runtime.AqlExecutionLessParallelismIT.test[AqlExecutionLessParallelismIT 1385: temp-dataset: temp_primary_plus_ngram_flush] Failing for the past 1 build (Since Unstable#1043 ) Took 9.2 sec. Error Message FileNotFoundException: File does not exist: target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix/2017-07-13-20-59-02-190_2017-07-13-20-59-02-190_f Stacktrace org.apache.asterix.common.exceptions.AsterixException: FileNotFoundException: File does not exist: target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix/2017-07-13-20-59-02-190_2017-07-13-20-59-02-190_f at org.apache.asterix.test.runtime.AqlExecutionLessParallelismIT.test(AqlExecutionLessParallelismIT.java:70) Standard Error Jul 13, 2017 8:58:53 PM org.apache.hyracks.control.common.config.ConfigManager get WARNING: NC option [nc] storage.lsm.bloomfilter.falsepositiverate being accessed outside of NC-scoped configuration. Jul 13, 2017 8:58:53 PM org.apache.hyracks.control.common.config.ConfigManager get WARNING: NC option [nc] storage.lsm.bloomfilter.falsepositiverate being accessed outside of NC-scoped configuration. Jul 13, 2017 8:58:53 PM org.apache.hyracks.control.common.config.ConfigManager get WARNING: NC option [nc] storage.lsm.bloomfilter.falsepositiverate being accessed outside of NC-scoped configuration. Jul 13, 2017 8:58:53 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 393216 bytes Jul 13, 2017 8:58:53 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 393216 bytes Jul 13, 2017 8:58:53 PM org.apache.hyracks.control.common.work.WorkQueue$WorkerThread auditWaitsAndBlocks WARNING: Work CleanupJoblet waited 0 times (~0ms), blocked 1 times (~0ms) Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 1507328 bytes Jul 13, 2017 8:59:01 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 1736704 bytes Jul 13, 2017 8:59:01 PM org.apache.hyracks.control.common.work.WorkQueue$WorkerThread auditWaitsAndBlocks WARNING: Work CleanupJoblet waited 0 times (~0ms), blocked 1 times (~0ms) Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:01 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 655360 bytes Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 655360 bytes Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.common.work.WorkQueue$WorkerThread auditWaitsAndBlocks WARNING: Work CleanupJoblet waited 1 times (~0ms), blocked 1 times (~0ms) Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.common.work.WorkQueue$WorkerThread auditWaitsAndBlocks WARNING: Work CleanupJoblet waited 1 times (~0ms), blocked 1 times (~0ms) Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.asterix.common.ioopcallbacks.AbstractLSMIOOperationCallback getComponentId WARNING: Flushing a memory component without setting the LSN Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 688128 bytes Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 589824 bytes Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.common.work.WorkQueue$WorkerThread auditWaitsAndBlocks WARNING: Work CleanupJoblet waited 0 times (~0ms), blocked 1 times (~0ms) Expected results file: src/test/resources/runtimets/results/temp-dataset/temp_primary_plus_ngram_flush/temp_primary_plus_ngram_flush.3.adm Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Joblet close WARNING: Freeing leaked 196608 bytes Jul 13, 2017 8:59:02 PM org.apache.asterix.test.common.TestExecutor cleanup WARNING: Last test left some garbage. Dropping dataverses: recovery org.apache.hyracks.api.exceptions.HyracksDataException: HYR0019: Cannot delete the file: /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix/2017-07-13-20-59-02-131_2017-07-13-20-59-02-131_i Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Task run WARNING: Task TAID:TID:ANID:ODID:0:0:1:0 failed with exception org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0019: Cannot delete the file: /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:45) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:84) at org.apache.hyracks.control.nc.Task.run(Task.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0019: Cannot delete the file: /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:210) ... 5 more Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0019: Cannot delete the file: /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:53) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:67) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:48) at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndexFileManager.deleteDirs(AbstractLSMIndexFileManager.java:195) at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.destroy(AbstractLSMIndex.java:233) at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.destroy(IndexDataflowHelper.java:99) at org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.initialize(IndexDropOperatorNodePushable.java:56) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:202) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Caused by: java.io.FileNotFoundException: File does not exist: target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix/2017-07-13-20-59-02-190_2017-07-13-20-59-02-190_f at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2377) at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1679) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1575) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:62) ... 10 more at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:53) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:67) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:48) at org.apache.hyracks.storage.common.buffercache.BufferCache.deleteFile(BufferCache.java:1008) at org.apache.hyracks.storage.common.buffercache.BufferCache.deleteFile(BufferCache.java:970) at org.apache.hyracks.storage.am.lsm.invertedindex.ondisk.OnDiskInvertedIndex.destroy(OnDiskInvertedIndex.java:172) at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndexDiskComponent.destroy(LSMInvertedIndexDiskComponent.java:47) at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.exitComponents(LSMHarness.java:344) at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:556) at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndexAccessor.merge(LSMInvertedIndexAccessor.java:122) at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:48) at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:30) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.file.NoSuchFileException: target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix/2017-07-13-20-59-02-131_2017-07-13-20-59-02-131_i at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.common.work.WorkQueue$WorkerThread auditWaitsAndBlocks WARNING: Work JobletCleanupNotification waited 0 times (~0ms), blocked 1 times (~0ms) at java.nio.file.Files.delete(Files.java:1126) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:64) ... 14 more Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Task run WARNING: Task TAID:TID:ANID:ODID:0:0:3:0 failed with exception org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:45) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:84) at org.apache.hyracks.control.nc.Task.run(Task.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:210) ... 5 more Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.readIndex(IndexDataflowHelper.java:71) at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.destroy(IndexDataflowHelper.java:93) at org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.initialize(IndexDropOperatorNodePushable.java:56) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:202) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Task run WARNING: Task TAID:TID:ANID:ODID:0:0:1:0 failed with exception org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:45) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:84) at org.apache.hyracks.control.nc.Task.run(Task.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:210) ... 5 more Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.readIndex(IndexDataflowHelper.java:71) at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.destroy(IndexDataflowHelper.java:93) at org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.initialize(IndexDropOperatorNodePushable.java:56) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:202) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Task run WARNING: Task TAID:TID:ANID:ODID:0:0:2:0 failed with exception org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:45) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:84) at org.apache.hyracks.control.nc.Task.run(Task.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:210) ... 5 more Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.readIndex(IndexDataflowHelper.java:71) at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.destroy(IndexDataflowHelper.java:93) at org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.initialize(IndexDropOperatorNodePushable.java:56) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:202) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.nc.Task run WARNING: Task TAID:TID:ANID:ODID:0:0:0:0 failed with exception org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:45) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:84) at org.apache.hyracks.control.nc.Task.run(Task.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:210) ... 5 more Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.readIndex(IndexDataflowHelper.java:71) at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.destroy(IndexDataflowHelper.java:93) at org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.initialize(IndexDropOperatorNodePushable.java:56) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:202) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.cc.job.JobManager prepareComplete WARNING: Ignoring duplicate cleanup for JobRun with id: JID:11181 Jul 13, 2017 8:59:02 PM org.apache.hyracks.control.cc.job.JobManager prepareComplete WARNING: Ignoring duplicate cleanup for JobRun with id: JID:11181 Jul 13, 2017 8:59:02 PM org.apache.asterix.api.http.server.QueryServiceServlet handleExecuteStatementException WARNING: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0019: Cannot delete the file: /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0019: Cannot delete the file: /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:58) at org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:62) at org.apache.hyracks.control.nc.Task.run(Task.java:361) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Suppressed: org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? ... 6 more Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:210) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:84) at org.apache.hyracks.control.nc.Task.run(Task.java:285) ... 3 more Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Index resource couldn't be found. Has it been created yet? Was it deleted? at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.readIndex(IndexDataflowHelper.java:71) at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.destroy(IndexDataflowHelper.java:93) at org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.initialize(IndexDropOperatorNodePushable.java:56) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:202) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0019: Cannot delete the file: /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:210) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:84) at org.apache.hyracks.control.nc.Task.run(Task.java:285) ... 3 more Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0019: Cannot delete the file: /home/jenkins/jenkins/workspace/asterix-gerrit-verify-asterix-app/asterixdb/asterix-app/target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:53) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:67) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:48) at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndexFileManager.deleteDirs(AbstractLSMIndexFileManager.java:195) at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.destroy(AbstractLSMIndex.java:233) at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.destroy(IndexDataflowHelper.java:99) at org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.initialize(IndexDropOperatorNodePushable.java:56) at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:202) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Caused by: java.io.FileNotFoundException: File does not exist: target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix/2017-07-13-20-59-02-190_2017-07-13-20-59-02-190_f at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2377) at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1679) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1575) at org.apache.hyracks.api.util.IoUtil.delete(IoUtil.java:62) ... 10 more org.apache.asterix.common.exceptions.AsterixException: FileNotFoundException: File does not exist: target/io/dir/asterix_nc1/iodevice1/storage/partition_1/temp/recovery/Fragile_idx_cfText2Ix/2017-07-13-20-59-02-190_2017-07-13-20-59-02-190_f at org.apache.asterix.test.common.ResultExtractor.extract(ResultExtractor.java:80) at org.apache.asterix.test.common.TestExecutor.cleanup(TestExecutor.java:1421) at org.apache.asterix.test.runtime.LangExecutionUtil.test(LangExecutionUtil.java:130) at org.apache.asterix.test.runtime.LangExecutionUtil.test(LangExecutionUtil.java:112) at org.apache.asterix.test.runtime.AqlExecutionLessParallelismIT.test(AqlExecutionLessParallelismIT.java:70) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)