I have not used pcap storage format before. Doesn't it require specific format defined in storage plugin ( as psv format given below)?
"formats": { "psv": { "type": "text", "extensions": [ "psv" ], "delimiter": "|" }, Thanks, Arjun ________________________________ From: Takeo Ogawara <ta-ogaw...@kddi-research.jp> Sent: Thursday, September 14, 2017 9:47 AM To: user Subject: Re: Query Error on PCAP over MapR FS I used storage plugin named “mfs” with “maprfs:///“. I modified plugin name from “mfs” to “dfs” and tested a query, but the result was the same (No such file). "Hadoop fs -ls / " can find x.pcap [drill@node21 log]$ hadoop fs -ls / | grep x.pcap -rw-r--r-- 3 root root 6083 2017-09-13 16:14 /x.pcap Show files in drill 0: jdbc:drill:drillbit=localhost> show files in dfs.`x.pcap`; +---------+--------------+---------+---------+--------+--------+--------------+------------------------+-------------------------+ | name | isDirectory | isFile | length | owner | group | permissions | accessTime | modificationTime | +---------+--------------+---------+---------+--------+--------+--------------+------------------------+-------------------------+ | x.pcap | false | true | 6083 | root | root | rw-r--r-- | 2017-09-13 16:14:52.0 | 2017-09-13 16:14:52.24 | +---------+--------------+---------+---------+--------+--------+--------------+------------------------+-------------------------+ 1 row selected (0.328 seconds) I’m running drill as user “drill”. Is there something wrong with file permissions? > For your testing you can just use the default root volume, but with MapR-FS > it is a good idea to create volumes for different data/use cases and then > mount these volumes on MapR-FS. > This allows for benefits like topology, quota & security management; also > ease of use for enterprise features like mirroring, snapshots, etc in the > future to name a few. > https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html Managing Data with Volumes<https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html> maprdocs.mapr.com MapR provides volumes as a way to organize data and manage cluster performance. A volume is a logical unit that allows you to apply policies to a set of files, directories, and sub-volumes. A ... Thank you for the information. I’ll separate the volume for PCAP from other services. Thank you. > 2017/09/13 23:48、Andries Engelbrecht <aengelbre...@mapr.com>のメ�`ル: > > Drill is not seeing the file in the location you pointed it. > > What did you name the storage plugin? > The default is normally dfs for the distributed filesystem. > > Also did you place the file in the root directory of the dfs? > What do you get back if you run Hadoop fs �Cls / > > For your testing you can just use the default root volume, but with MapR-FS > it is a good idea to create volumes for different data/use cases and then > mount these volumes on MapR-FS. > This allows for benefits like topology, quota & security management; also > ease of use for enterprise features like mirroring, snapshots, etc in the > future to name a few. > https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html Managing Data with Volumes<https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html> maprdocs.mapr.com MapR provides volumes as a way to organize data and manage cluster performance. A volume is a logical unit that allows you to apply policies to a set of files, directories, and sub-volumes. A ... > > > --Andries > > > On 9/13/17, 12:38 AM, "Takeo Ogawara" <ta-ogaw...@kddi-research.jp> wrote: > > Hi, > > I modified storage config like this. > > "type": "file", > "enabled": true, > "connection": "maprfs:///", > "config": null, > "workspaces": { > "root": { > "location": "/", > "writable": false, > "defaultInputFormat": null > } > } > > But query like “select * from mfs.`x.pcap`” failed. > Is there any other configuration I should modify? > > This is drillbit.log and it seems java.io.FileInputStream is going to open > MapR FS file path. > > Thank you. > > 2017-09-13 16:20:06,123 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.c.s.persistence.ScanResult - loading 9 classes for > org.apache.drill.exec.store.dfs.FormatPlugin took 0ms > 2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for > org.apache.drill.common.logical.FormatPluginConfig took 0ms > 2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for > org.apache.drill.common.logical.FormatPluginConfig took 0ms > 2017-09-13 16:20:06,125 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for > org.apache.drill.common.logical.FormatPluginConfig took 0ms > 2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.c.s.persistence.ScanResult - loading 9 classes for > org.apache.drill.exec.store.dfs.FormatPlugin took 0ms > 2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for > org.apache.drill.common.logical.FormatPluginConfig took 0ms > 2017-09-13 16:20:06,146 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for > org.apache.drill.common.logical.FormatPluginConfig took 0ms > 2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 > ms, numFiles: 1 > 2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 > ms, numFiles: 1 > 2017-09-13 16:20:06,178 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 > ms, numFiles: 1 > 2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of > 1 using 1 threads. Time: 0ms total, 0.847323ms avg, 0ms max. > 2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] > INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of > 1 using 1 threads. Earliest start: 1.522000 μs, Latest start: 1.522000 μs, > Average start: 1.522000 μs . > 2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] > INFO o.a.d.e.store.pcap.PcapRecordReader - User Error Occurred: /x.pcap (No > such file or directory) (/x.pcap (No such file or directory)) > org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: /x.pcap > (No such file or directory) > > File name: /x.pcap > > [Error Id: 48be766a-8706-407f-8dff-eb563271a4a3 ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550) > ~[drill-common-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:109) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:104) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin.getReaderBatch(EasyFormatPlugin.java:166) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:35) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:28) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:156) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:136) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:109) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:207) > [drill-java-exec-1.11.0.jar:1.11.0] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.11.0.jar:1.11.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_141] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_141] > at java.lang.Thread.run(Thread.java:748) [na:1.7.0_141] > Caused by: java.io.FileNotFoundException: /x.pcap (No such file or > directory) > at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_141] > at java.io.FileInputStream.<init>(FileInputStream.java:146) > ~[na:1.7.0_141] > at java.io.FileInputStream.<init>(FileInputStream.java:101) > ~[na:1.7.0_141] > at > org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:103) > [drill-java-exec-1.11.0.jar:1.11.0] > ... 15 common frames omitted > 2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] > INFO o.a.d.e.w.fragment.FragmentExecutor - > 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0: State change requested > AWAITING_ALLOCATION --> FAILED > 2017-09-13 16:20:06,200 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] > INFO o.a.d.e.w.fragment.FragmentExecutor - > 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0: State change requested FAILED --> > FINISHED > 2017-09-13 16:20:06,213 [BitServer-4] WARN > o.a.drill.exec.work.foreman.Foreman - Dropping request to move to COMPLETED > state as query is already at FAILED state (which is terminal). > 2017-09-13 16:20:06,214 [BitServer-4] WARN > o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. > 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0 does not exist. > > ― > > 97 @Override > 98 public void setup(final OperatorContext context, final OutputMutator > output) throws ExecutionSetupException { > 99 try { > 100 > 101 this.output = output; > 102 this.buffer = new byte[100000]; > 103 this.in = new FileInputStream(inputPath); > 104 this.decoder = new PacketDecoder(in); > 105 this.validBytes = in.read(buffer); > 106 this.projectedCols = getProjectedColsIfItNull(); > 107 setColumns(projectedColumns); > 108 } catch (IOException io) { > 109 throw UserException.dataReadError(io) > 110 .addContext("File name:", inputPath) > 111 .build(logger); > 112 } > 113 } > > >> 2017/09/11 23:16、Andries Engelbrecht <aengelbre...@mapr.com>のメ�`ル: >> >> Typically when you use the MapR-FS plugin you don’t need to specify the >> cluster root path in the dfs workspace. >> >> Instead of "location": "/mapr/cluster3", use "location": "/", >> >> "connection": "maprfs:///", already points to the default MapR cluster root. >> >> --Andries >> >> >> >> On 9/11/17, 2:23 AM, "Takeo Ogawara" <ta-ogaw...@kddi-research.jp> wrote: >> >> Dear all, >> >> I’m using PCAP storage plugin over MapR FS(5.2.0) with Drill(1.11.0) >> compiled as follows. >> $ mvn clean install -DskipTests -Pmapr >> >> Some queries caused errors as following. >> Does anyone know how to solve these errors? >> >> 1. Query error when cluster-name is not specified >> Storage “mfs” setting is this. >> >>> "type": "file", >>> "enabled": true, >>> "connection": "maprfs:///", >>> "config": null, >>> "workspaces": { >>> "root": { >>> "location": "/mapr/cluster3", >>> "writable": false, >>> "defaultInputFormat": null >>> } >>> } >> >> >> With this setting, the following query failed. >>> select * from mfs.`x.pcap` ; >>> Error: DATA_READ ERROR: /x.pcap (No such file or directory) >>> >>> File name: /x.pcap >>> Fragment 0:0 >>> >>> [Error Id: 70b73062-c3ed-4a10-9a88-034b4e6d039a on node21:31010] >>> (state=,code=0) >> >> But these queries passed. >>> select * from mfs.root.`x.pcap` ; >>> select * from mfs.`x.csv`; >>> select * from mfs.root.`x.csv`; >> >> 2. Large PCAP file >> Query on very large PCAP file (larger than 100GB) failed with following >> error message. >>> Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a >>> >>> Fragment 1:169 >>> >>> [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] >>> (state=,code=0) >> >> This happens even on Linux FS not MapR FS >> >> Thank you. >> >> >> >> >> >> > > ――――――――――――――――――――――― > <KDDI�t合研究所 ビジョン> > Challenge for the future �Nかな未来への挑�� > ――――――――――――――――――――――― > 英雄だけの夏。 > https://www.au.com/pr/cm/3taro/ > ――――――――――――――――――――――― > 小河原 健生(Takeo Ogawara) > (株)KDDI�t合研究所 > コネクティッドカ�`1G > > TEL:049-278-7495 / 070-3623-9914 > > >