So then there should be one fragment reading the data. Takeo, if you file a Jira, perhaps you can include the Drill profile, and we can look at how many fragments are used by the query. The profiles can be found via a web URL
http://<your_IP_address>:8047 Thanks. --Robert ________________________________ From: Ted Dunning <[email protected]> Sent: Tuesday, September 12, 2017 4:30 PM To: user Cc: [email protected] Subject: Re: ***UNCHECKED*** Re: Query Error on PCAP over MapR FS PCAP is a binary format that cannot easily be split. On Wed, Sep 13, 2017 at 1:15 AM, Robert Hou <[email protected]> wrote: > Hi Ted, > > > My understanding is, by default, Drill should not have multiple threads > reading the PCAP file in parallel. But if the records in a PCAP file can > be read in parallel (e.g. rows are restricted to one line each), then the > plugin can be designed to read the file in parallel. > > > Are PCAP records single-line records? > > > Thanks. > > > --Robert > > ________________________________ > From: Ted Dunning <[email protected]> > Sent: Tuesday, September 12, 2017 3:54 PM > To: user > Cc: [email protected] > Subject: Re: ***UNCHECKED*** Re: Query Error on PCAP over MapR FS > > Robert, > > Thanks for looking at this. I think that this is a bug in the way that we > wrote the format plugin in that it isn't telling Drill to not split the > file. > > > > On Wed, Sep 13, 2017 at 12:52 AM, Robert Hou <[email protected]> wrote: > > > I asked a couple of Drill developers. We don't have much experience with > > PCAP yet. Takeo, can you file a Jira for this, and include the > information > > below? The error message mentions a bad magic number, which Drill > > sometimes uses to help determine the file format. > > > > > > Also, it appears that you have tried to query your data across many small > > files rather than one large file. This is the preferred approach, and it > > seems that this approach works for you. Please let me know if you think > > otherwise, that you need to access your data in one large PCAP file. > > > > > > Thanks. > > > > > > --Robert > > > > > > ________________________________ > > From: Ted Dunning <[email protected]> > > Sent: Monday, September 11, 2017 8:15 PM > > To: user; [email protected] > > Subject: Re: ***UNCHECKED*** Re: Query Error on PCAP over MapR FS > > > > This stack trace makes it clear that this is a bug in the PCAP decoder > > caused by a misunderstanding of how to force large files to be read in > one > > batch on a single drillBit. > > > > Are there some real Drill experts out there who can provide hints about > how > > to avoid this? > > > > > > > > On Tue, Sep 12, 2017 at 5:03 AM, Takeo Ogawara < > > [email protected]> > > wrote: > > > > > Sorry > > > > > > I paste plain texts. > > > > > > > 2017-09-11 15:06:52,390 [BitServer-2] WARN o.a.d.exec.rpc.control. > > WorkEventBus > > > - A fragment message arrived but there was no registered listener for > > that > > > message: profile { > > > > state: FAILED > > > > error { > > > > error_id: "bbf284b6-9da4-4869-ac20-fa100eed11b9" > > > > endpoint { > > > > address: "node22" > > > > user_port: 31010 > > > > control_port: 31011 > > > > data_port: 31012 > > > > version: "1.11.0" > > > > } > > > > error_type: SYSTEM > > > > message: "SYSTEM ERROR: IllegalStateException: Bad magic number = > > > 0a0d0d0a\n\nFragment 1:200\n\n[Error Id: bbf284b6-9da4-4869-ac20- > > fa100eed11b9 > > > on node22:31010]" > > > > exception { > > > > exception_class: "java.lang.IllegalStateException" > > > > message: "Bad magic number = 0a0d0d0a" > > > > stack_trace { > > > > class_name: "com.google.common.base.Preconditions" > > > > file_name: "Preconditions.java" > > > > line_number: 173 > > > > method_name: "checkState" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec.store. > > > pcap.decoder.PacketDecoder" > > > > file_name: "PacketDecoder.java" > > > > line_number: 84 > > > > method_name: "<init>" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec.store. > > pcap.PcapRecordReader" > > > > file_name: "PcapRecordReader.java" > > > > line_number: 104 > > > > method_name: "setup" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec.physical.impl.ScanBatch" > > > > file_name: "ScanBatch.java" > > > > line_number: 104 > > > > method_name: "<init>" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec.store. > > > dfs.easy.EasyFormatPlugin" > > > > file_name: "EasyFormatPlugin.java" > > > > line_number: 166 > > > > method_name: "getReaderBatch" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec.store.dfs.easy. > > > EasyReaderBatchCreator" > > > > file_name: "EasyReaderBatchCreator.java" > > > > line_number: 35 > > > > method_name: "getBatch" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec.store.dfs.easy. > > > EasyReaderBatchCreator" > > > > file_name: "EasyReaderBatchCreator.java" > > > > line_number: 28 > > > > method_name: "getBatch" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec. > physical.impl.ImplCreator" > > > > file_name: "ImplCreator.java" > > > > line_number: 156 > > > > method_name: "getRecordBatch" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec. > physical.impl.ImplCreator" > > > > file_name: "ImplCreator.java" > > > > line_number: 179 > > > > method_name: "getChildren" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec. > physical.impl.ImplCreator" > > > > file_name: "ImplCreator.java" > > > > line_number: 136 > > > > method_name: "getRecordBatch" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec. > physical.impl.ImplCreator" > > > > file_name: "ImplCreator.java" > > > > line_number: 179 > > > > method_name: "getChildren" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec. > physical.impl.ImplCreator" > > > > file_name: "ImplCreator.java" > > > > line_number: 136 > > > > method_name: "getRecordBatch" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec. > physical.impl.ImplCreator" > > > > file_name: "ImplCreator.java" > > > > line_number: 179 > > > > method_name: "getChildren" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec. > physical.impl.ImplCreator" > > > > file_name: "ImplCreator.java" > > > > line_number: 109 > > > > method_name: "getRootExec" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec. > physical.impl.ImplCreator" > > > > file_name: "ImplCreator.java" > > > > line_number: 87 > > > > method_name: "getExec" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.exec.work. > > > fragment.FragmentExecutor" > > > > file_name: "FragmentExecutor.java" > > > > line_number: 207 > > > > method_name: "run" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "org.apache.drill.common.SelfCleaningRunnable" > > > > file_name: "SelfCleaningRunnable.java" > > > > line_number: 38 > > > > method_name: "run" > > > > is_native_method: false > > > > } > > > > stack_trace { > > > > class_name: "..." > > > > line_number: 0 > > > > method_name: "..." > > > > is_native_method: false > > > > } > > > > } > > > > } > > > > minor_fragment_id: 200 > > > > operator_profile { > > > > input_profile { > > > > records: 0 > > > > batches: 0 > > > > schemas: 0 > > > > } > > > > operator_id: 0 > > > > operator_type: 37 > > > > setup_nanos: 0 > > > > process_nanos: 29498572 > > > > peak_local_memory_allocated: 0 > > > > wait_nanos: 0 > > > > } > > > > start_time: 1505110011975 > > > > end_time: 1505110012320 > > > > memory_used: 0 > > > > max_memory_used: 1000000 > > > > endpoint { > > > > address: "node22" > > > > user_port: 31010 > > > > control_port: 31011 > > > > data_port: 31012 > > > > version: "1.11.0" > > > > } > > > > } > > > > handle { > > > > query_id { > > > > part1: 2758973773160297386 > > > > part2: -412723615757922113 > > > > } > > > > major_fragment_id: 1 > > > > minor_fragment_id: 200 > > > > } > > > > . > > > > > > > > > > [Error Id: c737dd8b-78e4-40c6-89b0-d53260770b11 on node21:31010] > > > > at org.apache.drill.exec.rpc.user.QueryResultHandler. > > > resultArrived(QueryResultHandler.java:123) > > [drill-java-exec-1.11.0.jar:1. > > > 11.0] > > > > at org.apache.drill.exec.rpc.user.UserClient.handle( > > UserClient.java:368) > > > [drill-java-exec-1.11.0.jar:1.11.0] > > > > at org.apache.drill.exec.rpc.user.UserClient.handle( > > UserClient.java:90) > > > [drill-java-exec-1.11.0.jar:1.11.0] > > > > at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode( > > RpcBus.java:274) > > > [drill-rpc-1.11.0.jar:1.11.0] > > > > at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode( > > RpcBus.java:244) > > > [drill-rpc-1.11.0.jar:1.11.0] > > > > at io.netty.handler.codec.MessageToMessageDecoder. > channelRead( > > > MessageToMessageDecoder.java:89) [netty-codec-4.0.27.Final.jar: > > > 4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > invokeChannelRead(AbstractChannelHandlerContext.java:339) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > fireChannelRead(AbstractChannelHandlerContext.java:324) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.handler.timeout.IdleStateHandler.channelRead( > > IdleStateHandler.java:254) > > > [netty-handler-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > invokeChannelRead(AbstractChannelHandlerContext.java:339) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > fireChannelRead(AbstractChannelHandlerContext.java:324) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.handler.codec.MessageToMessageDecoder. > channelRead( > > > MessageToMessageDecoder.java:103) [netty-codec-4.0.27.Final.jar: > > > 4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > invokeChannelRead(AbstractChannelHandlerContext.java:339) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > fireChannelRead(AbstractChannelHandlerContext.java:324) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.handler.codec.ByteToMessageDecoder.channelRead( > > ByteToMessageDecoder.java:242) > > > [netty-codec-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > invokeChannelRead(AbstractChannelHandlerContext.java:339) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > fireChannelRead(AbstractChannelHandlerContext.java:324) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.ChannelInboundHandlerAdapter. > channelRead( > > > ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.27.Final. > > > jar:4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > invokeChannelRead(AbstractChannelHandlerContext.java:339) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.AbstractChannelHandlerContext. > > > fireChannelRead(AbstractChannelHandlerContext.java:324) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.DefaultChannelPipeline.fireChannelRead( > > > DefaultChannelPipeline.java:847) [netty-transport-4.0.27.Final. > > > jar:4.0.27.Final] > > > > at io.netty.channel.nio.AbstractNioByteChannel$ > > > NioByteUnsafe.read(AbstractNioByteChannel.java:131) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.nio.NioEventLoop.processSelectedKey( > > NioEventLoop.java:511) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.nio.NioEventLoop. > > > processSelectedKeysOptimized(NioEventLoop.java:468) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > > NioEventLoop.java:382) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop. > > java:354) > > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > > > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > > > run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final. > > > jar:4.0.27.Final] > > > > at java.lang.Thread.run(Thread.java:748) [na:1.7.0_141] > > > > 2017-09-11 15:32:36,406 [Client-1] INFO o.a.d.j.i.DrillCursor$ > > ResultsListener > > > - [#5] Query failed: > > > > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: > > > IllegalStateException: Bad magic number = 0a0d0d0a > > > > > > > > > > > > > > > > > > > > > > > > > 2017/09/12 11:53、Takeo Ogawara <[email protected]>のメ�`ル: > > > > > > > > Thank you for replies. > > > > > > > >> Instead of "location": "/mapr/cluster3", use "location": "/", > > > > I’ll use this config. > > > > > > > > > > > >> Can you provide the stack trace from the Drillbit that hit the > > problem? > > > > > > > > You can find logs in attached files. > > > > > > > >> Is it absolutely required to query large files like this? Would it > be > > > >> acceptable to split the file first by making a quick scan over it? > > > > No,loading large file isn’t necessarily required. > > > > In fact, this large PCAP file is created by concatenating small PCAP > > > files with mergecap command. > > > > So there is no problem with input small PCAP files into Drill. > > > > > > > > How can I analyze numbers of PCAP files together? > > > > Can I concatenate parsed packet records of small PCAP files inside > > Drill > > > query? > > > > Or should I export parsed records into a database and then merge > them? > > > > > > > > > > > >> 2017/09/12 5:07、Ted Dunning <[email protected]>のメ�`ル: > > > >> > > > >> On Mon, Sep 11, 2017 at 11:23 AM, Takeo Ogawara < > > > [email protected] > > > >>> wrote: > > > >> > > > >>> ... > > > >>> > > > >>> 1. Query error when cluster-name is not specified > > > >>> ... > > > >>> > > > >>> With this setting, the following query failed. > > > >>>> select * from mfs.`x.pcap` ; > > > >>>> Error: DATA_READ ERROR: /x.pcap (No such file or directory) > > > >>>> > > > >>>> File name: /x.pcap > > > >>>> Fragment 0:0 > > > >>>> > > > >>>> [Error Id: 70b73062-c3ed-4a10-9a88-034b4e6d039a on node21:31010] > > > >>> (state=,code=0) > > > >>> > > > >>> But these queries passed. > > > >>>> select * from mfs.root.`x.pcap` ; > > > >>>> select * from mfs.`x.csv`; > > > >>>> select * from mfs.root.`x.csv`; > > > >>> > > > >> > > > >> As Andries mentioned, the problem here has to do with understanding > > what > > > >> Drill is thinking about how paths are manipulated. Nothing to do > with > > > the > > > >> PCAP capabilities. > > > >> > > > >> Usually, what I do is put entries into the configuration which > > directly > > > >> point to the directory above my data, but I can't add anything > Andries > > > >> comment. > > > >> > > > >> > > > >>> 2. Large PCAP file > > > >>> Query on very large PCAP file (larger than 100GB) failed with > > following > > > >>> error message. > > > >>>> Error: SYSTEM ERROR: IllegalStateException: Bad magic number = > > > 0a0d0d0a > > > >>>> > > > >>>> Fragment 1:169 > > > >>>> > > > >>>> [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > > >>> (state=,code=0) > > > >>> > > > >>> This happens even on Linux FS not MapR FS > > > >>> > > > >> > > > >> Can you provide the stack trace from the Drillbit that hit the > > problem? > > > >> > > > >> I suspect that this has to do with splitting of the PCAP file. > > > Normally, it > > > >> is assumed that parallelism will be achieved by having lots of > smaller > > > >> files since it is difficult to jump into the middle of a PCAP file > and > > > get > > > >> good results. > > > >> > > > >> Even if we disable splitting to avoid this error, you will have the > > > >> complementary problem of slow queries due to single-threading. That > > > doesn't > > > >> seem very satisfactory either. > > > >> > > > >> A similar problem is that splitting a PCAP file pretty much > requires a > > > >> single-threaded read of the file in question. The read doesn't need > to > > > >> process very much data, but it does need to touch the whole file. > > > >> > > > >> Is it absolutely required to query large files like this? Would it > be > > > >> acceptable to split the file first by making a quick scan over it? > > > > > > > > > > > > <sa1153582.zip> > > > > > > ――――――――――――――――――――――― > > > <KDDI�t合研究所 ビジョン> > > > Challenge for the future �Nかな未来への挑�� > > > ――――――――――――――――――――――― > > > 英雄だけの夏。 > > > https://www.au.com/pr/cm/3taro/ [https://kddi-h.assetsadobe3.com/is/image/content/dam/au-com/pr/cm/3taro/images/og-3t.jpg?scl=1]<https://www.au.com/pr/cm/3taro/> 【au】三太郎スペシャルペ�`ジ<https://www.au.com/pr/cm/3taro/> www.au.com 桃太郎、浦�u太郎、金太郎や、かぐや��、乙��、鬼ちゃん、����が�Rり�冥菠搿�au三太郎」のCMギャラリ�`です。最新�踊�や ... > [https://kddi-h.assetsadobe3.com/is/image/content/dam/au- > com/pr/cm/3taro/images/og-3t.jpg?scl=1]<https://www.au.com/pr/cm/3taro/> > > 【au】三太郎スペシャルペ�`ジ<https://www.au.com/pr/cm/3taro/> > www.au.com > 桃太郎、浦�u太郎、金太郎や、かぐや��、乙��、鬼ちゃん、����が�Rり�冥菠搿�au三太郎」のCMギャラリ�`です。最新�踊�や ... > > > > > [https://kddi-h.assetsadobe3.com/is/image/content/dam/au- > > com/pr/cm/3taro/images/og-3t.jpg?scl=1]<https://www.au.com/pr/cm/3taro/> > > > > 【au】三太郎スペシャルペ�`ジ<https://www.au.com/pr/cm/3taro/> > > www.au.com > > 桃太郎、浦�u太郎、金太郎や、かぐや��、乙��、鬼ちゃん、����が�Rり�冥菠搿�au三太郎」のCMギャラリ�`です。最新�踊�や ... > > > > > > > > > ――――――――――――――――――――――― > > > 小河原 健生（Takeo Ogawara） > > > (株)KDDI�t合研究所 > > > コネクティッドカ�`1G > > > > > > TEL:049-278-7495 / 070-3623-9914 > > > > > > > > >
