[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: 2647724c-d367-c8cc-0b6d-579228ffa31e.sys.drill sample2.pcap Sorry, I attached html file. Profile file is here. You can create dummy large PCAP file with mergecap command using sample pcap file. > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: 2647724c-d367-c8cc-0b6d-579228ffa31e.sys.drill, Apache > Drill_files.zip, Apache Drill.html, sample2.pcap > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: Apache Drill_files.zip Apache Drill.html Query profiles. > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: (was: Apache Drill_files.zip) > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: (was: Apache Drill.html) > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file
[ https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeo Ogawara updated DRILL-5785: - Attachment: Apache Drill_files.zip Apache Drill.html > Query Error on a large PCAP file > > > Key: DRILL-5785 > URL: https://issues.apache.org/jira/browse/DRILL-5785 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: Takeo Ogawara >Priority: Minor > Attachments: Apache Drill_files.zip, Apache Drill.html > > > Query on a very large PCAP file (larger than 100GB) failed with following > error message. > > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > > > Fragment 1:169 > > > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5785) Query Error on a large PCAP file
[
https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163984#comment-16163984
]
Takeo Ogawara commented on DRILL-5785:
--
Here are the messages in Drill log files.
drillbit.log
---
2017-09-11 15:06:52,390 [BitServer-2] WARN o.a.d.exec.rpc.control.WorkEventBus
- A fragment message arrived but there was no registered listener for that
message: profile {
state: FAILED
error {
error_id: "bbf284b6-9da4-4869-ac20-fa100eed11b9"
endpoint {
address: "node22"
user_port: 31010
control_port: 31011
data_port: 31012
version: "1.11.0"
}
error_type: SYSTEM
message: "SYSTEM ERROR: IllegalStateException: Bad magic number =
0a0d0d0a\n\nFragment 1:200\n\n[Error Id: bbf284b6-9da4-4869-ac20-fa100eed11b9
on node22:31010]"
exception {
exception_class: "java.lang.IllegalStateException"
message: "Bad magic number = 0a0d0d0a"
stack_trace {
class_name: "com.google.common.base.Preconditions"
file_name: "Preconditions.java"
line_number: 173
method_name: "checkState"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.store.pcap.decoder.PacketDecoder"
file_name: "PacketDecoder.java"
line_number: 84
method_name: ""
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.store.pcap.PcapRecordReader"
file_name: "PcapRecordReader.java"
line_number: 104
method_name: "setup"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ScanBatch"
file_name: "ScanBatch.java"
line_number: 104
method_name: ""
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin"
file_name: "EasyFormatPlugin.java"
line_number: 166
method_name: "getReaderBatch"
is_native_method: false
}
stack_trace {
class_name:
"org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator"
file_name: "EasyReaderBatchCreator.java"
line_number: 35
method_name: "getBatch"
is_native_method: false
}
stack_trace {
class_name:
"org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator"
file_name: "EasyReaderBatchCreator.java"
line_number: 28
method_name: "getBatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 156
method_name: "getRecordBatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 179
method_name: "getChildren"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 136
method_name: "getRecordBatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 179
method_name: "getChildren"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 136
method_name: "getRecordBatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 179
method_name: "getChildren"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 109
method_name: "getRootExec"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 87
method_name: "getExec"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.work.fragment.FragmentExecutor"
file_name: "FragmentExecutor.java"
line_number: 207
method_name: "run"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.common.SelfCleaningRunnable"
file_name: "SelfCleaningRunnable.java"
line_number: 38
method_name: "run"
is_native_method: false
}
stack_trace {
class_name: "..."
line_number: 0
method_name:
[jira] [Commented] (DRILL-5708) Add DNS decode function for PCAP storage
[ https://issues.apache.org/jira/browse/DRILL-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119286#comment-16119286 ] Takeo Ogawara commented on DRILL-5708: -- Hello Givre Thank you for the comment. Main outputs are following in my mind. 1. Domain name, queried by user 2. Canonical names in response sequences 3. Resolved IP Address 4. TTL > Add DNS decode function for PCAP storage > > > Key: DRILL-5708 > URL: https://issues.apache.org/jira/browse/DRILL-5708 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Reporter: Takeo Ogawara >Priority: Minor > > As described in DRILL-5432, it is very useful to analyze packet contents and > application layer protocols. To improve the PCAP analysis function, it's > better to add a function to decode DNS queries and responses. This enables to > classify packets by FQDN and display user access trends. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5708) Add DNS decode function for PCAP storage
Takeo Ogawara created DRILL-5708: Summary: Add DNS decode function for PCAP storage Key: DRILL-5708 URL: https://issues.apache.org/jira/browse/DRILL-5708 Project: Apache Drill Issue Type: Improvement Components: Storage - Other Reporter: Takeo Ogawara Priority: Minor As described in DRILL-5432, it is very useful to analyze packet contents and application layer protocols. To improve the PCAP analysis function, it's better to add a function to decode DNS queries and responses. This enables to classify packets by FQDN and display user access trends. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
