Drill is not seeing the file in the location you pointed it. What did you name the storage plugin? The default is normally dfs for the distributed filesystem.
Also did you place the file in the root directory of the dfs? What do you get back if you run Hadoop fs –ls / For your testing you can just use the default root volume, but with MapR-FS it is a good idea to create volumes for different data/use cases and then mount these volumes on MapR-FS. This allows for benefits like topology, quota & security management; also ease of use for enterprise features like mirroring, snapshots, etc in the future to name a few. https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html --Andries On 9/13/17, 12:38 AM, "Takeo Ogawara" <[email protected]> wrote: Hi, I modified storage config like this. "type": "file", "enabled": true, "connection": "maprfs:///", "config": null, "workspaces": { "root": { "location": "/", "writable": false, "defaultInputFormat": null } } But query like “select * from mfs.`x.pcap`” failed. Is there any other configuration I should modify? This is drillbit.log and it seems java.io.FileInputStream is going to open MapR FS file path. Thank you. 2017-09-13 16:20:06,123 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.c.s.persistence.ScanResult - loading 9 classes for org.apache.drill.exec.store.dfs.FormatPlugin took 0ms 2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for org.apache.drill.common.logical.FormatPluginConfig took 0ms 2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for org.apache.drill.common.logical.FormatPluginConfig took 0ms 2017-09-13 16:20:06,125 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for org.apache.drill.common.logical.FormatPluginConfig took 0ms 2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.c.s.persistence.ScanResult - loading 9 classes for org.apache.drill.exec.store.dfs.FormatPlugin took 0ms 2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for org.apache.drill.common.logical.FormatPluginConfig took 0ms 2017-09-13 16:20:06,146 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for org.apache.drill.common.logical.FormatPluginConfig took 0ms 2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-09-13 16:20:06,178 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Time: 0ms total, 0.847323ms avg, 0ms max. 2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 1.522000 μs, Latest start: 1.522000 μs, Average start: 1.522000 μs . 2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] INFO o.a.d.e.store.pcap.PcapRecordReader - User Error Occurred: /x.pcap (No such file or directory) (/x.pcap (No such file or directory)) org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: /x.pcap (No such file or directory) File name: /x.pcap [Error Id: 48be766a-8706-407f-8dff-eb563271a4a3 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550) ~[drill-common-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:109) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:104) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin.getReaderBatch(EasyFormatPlugin.java:166) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:35) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:28) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:156) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:136) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:109) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:207) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.11.0.jar:1.11.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_141] at java.lang.Thread.run(Thread.java:748) [na:1.7.0_141] Caused by: java.io.FileNotFoundException: /x.pcap (No such file or directory) at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_141] at java.io.FileInputStream.<init>(FileInputStream.java:146) ~[na:1.7.0_141] at java.io.FileInputStream.<init>(FileInputStream.java:101) ~[na:1.7.0_141] at org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:103) [drill-java-exec-1.11.0.jar:1.11.0] ... 15 common frames omitted 2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0: State change requested AWAITING_ALLOCATION --> FAILED 2017-09-13 16:20:06,200 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0: State change requested FAILED --> FINISHED 2017-09-13 16:20:06,213 [BitServer-4] WARN o.a.drill.exec.work.foreman.Foreman - Dropping request to move to COMPLETED state as query is already at FAILED state (which is terminal). 2017-09-13 16:20:06,214 [BitServer-4] WARN o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0 does not exist. — 97 @Override 98 public void setup(final OperatorContext context, final OutputMutator output) throws ExecutionSetupException { 99 try { 100 101 this.output = output; 102 this.buffer = new byte[100000]; 103 this.in = new FileInputStream(inputPath); 104 this.decoder = new PacketDecoder(in); 105 this.validBytes = in.read(buffer); 106 this.projectedCols = getProjectedColsIfItNull(); 107 setColumns(projectedColumns); 108 } catch (IOException io) { 109 throw UserException.dataReadError(io) 110 .addContext("File name:", inputPath) 111 .build(logger); 112 } 113 } > 2017/09/11 23:16、Andries Engelbrecht <[email protected]>のメール: > > Typically when you use the MapR-FS plugin you don’t need to specify the cluster root path in the dfs workspace. > > Instead of "location": "/mapr/cluster3", use "location": "/", > > "connection": "maprfs:///", already points to the default MapR cluster root. > > --Andries > > > > On 9/11/17, 2:23 AM, "Takeo Ogawara" <[email protected]> wrote: > > Dear all, > > I’m using PCAP storage plugin over MapR FS(5.2.0) with Drill(1.11.0) compiled as follows. > $ mvn clean install -DskipTests -Pmapr > > Some queries caused errors as following. > Does anyone know how to solve these errors? > > 1. Query error when cluster-name is not specified > Storage “mfs” setting is this. > >> "type": "file", >> "enabled": true, >> "connection": "maprfs:///", >> "config": null, >> "workspaces": { >> "root": { >> "location": "/mapr/cluster3", >> "writable": false, >> "defaultInputFormat": null >> } >> } > > > With this setting, the following query failed. >> select * from mfs.`x.pcap` ; >> Error: DATA_READ ERROR: /x.pcap (No such file or directory) >> >> File name: /x.pcap >> Fragment 0:0 >> >> [Error Id: 70b73062-c3ed-4a10-9a88-034b4e6d039a on node21:31010] (state=,code=0) > > But these queries passed. >> select * from mfs.root.`x.pcap` ; >> select * from mfs.`x.csv`; >> select * from mfs.root.`x.csv`; > > 2. Large PCAP file > Query on very large PCAP file (larger than 100GB) failed with following error message. >> Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a >> >> Fragment 1:169 >> >> [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] (state=,code=0) > > This happens even on Linux FS not MapR FS > > Thank you. > > > > > > ——————————————————————— <KDDI総合研究所 ビジョン> Challenge for the future 豊かな未来への挑戦 ——————————————————————— 英雄だけの夏。 https://www.au.com/pr/cm/3taro/ ——————————————————————— 小河原 健生（Takeo Ogawara） (株)KDDI総合研究所 コネクティッドカー1G TEL:049-278-7495 / 070-3623-9914
