[GitHub] metron issue #1109: METRON-1674: Create REST endpoint for job status abstrac...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1109 That PR should be up for review soon. ---
[GitHub] metron issue #1109: METRON-1674: Create REST endpoint for job status abstrac...
Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1109 I see, thanks. Is there an endpoint for the PDML output? Or it is out of the scope of this PR? ---
[GitHub] metron issue #1109: METRON-1674: Create REST endpoint for job status abstrac...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1109 @tiborm That is an HDFS path, not a URL. It's of no concern to the UI. ---
[GitHub] metron issue #1109: METRON-1674: Create REST endpoint for job status abstrac...
Github user tiborm commented on the issue:
https://github.com/apache/metron/pull/1109
Is it necessary to have /{user}/{jobType}/ in the output url?
Right now our UI calling this:
/apps/metron/pcap/output/{jobId}/{pageId}
On our side, pageId is a simple int at the moment. Is "page-*" at the end
of the URL pattern means the service expecting something like "page-5"?
---
[GitHub] metron issue #1109: METRON-1674: Create REST endpoint for job status abstrac...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/1109 +1 works great. I was able to put the output of the pcap topology into a directory in HDFS, run a query over it, got a page in the output directory (with status correctly updating while the job was running), was able to open the page in wireshark ---
[GitHub] metron issue #1109: METRON-1674: Create REST endpoint for job status abstrac...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1109 The latest commit addresses feedback received so far. I also changed the structure of the HDFS paths to avoid issues around reading input and also to output data to unique locations. Previously all data was written to `/apps/metron/pcap/output/page-*` which obviously won't work. I will update the PR description with new testing instructions. @james-sirota this should resolve the error you were getting. ---
[GitHub] metron issue #1109: METRON-1674: Create REST endpoint for job status abstrac...
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/1109
Here is what I've done so far:
Capture a random PCAP using tcpdump, place that PCAP into
/apps/metron/pcap/input on HDFS, run the service in Swager under POST
/api/v1/pcap/fixed
I send
{
"endTime":1531892977000,
"startTime":1458240269424
}
which are the right timestamps for my PCAP, and get back an error:
{
"timestamp": "2018-07-18 06:00:45",
"status": 500,
"error": "Internal Server Error",
"message": "-1",
"path": "/api/v1/pcap/fixed"
}
Which in metron-rest.log corresponds to:
Jul 18, 2018 6:00:45 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with
path [] threw exception [Request processing failed; nested exception is
java.lang.ArrayIndexOutOfBoundsException: -1] with root cause
java.lang.ArrayIndexOutOfBoundsException: -1
at
org.apache.metron.pcap.PcapFilenameHelper.getKafkaPartition(PcapFilenameHelper.java:66)
at
org.apache.metron.pcap.utils.FileFilterUtil.getFilesByPartition(FileFilterUtil.java:63)
at
org.apache.metron.pcap.utils.FileFilterUtil.getPathsInTimeRange(FileFilterUtil.java:50)
at org.apache.metron.pcap.mr.PcapJob.createJob(PcapJob.java:382)
at org.apache.metron.pcap.mr.PcapJob.query(PcapJob.java:276)
at org.apache.metron.pcap.mr.PcapJob.submit(PcapJob.java:235)
at
org.apache.metron.rest.config.PcapJobSupplier.get(PcapJobSupplier.java:38)
at
org.apache.metron.rest.config.PcapJobSupplier.get(PcapJobSupplier.java:30)
at
org.apache.metron.job.manager.InMemoryJobManager.submit(InMemoryJobManager.java:48)
at
org.apache.metron.rest.service.impl.PcapServiceImpl.fixed(PcapServiceImpl.java:64)
at
org.apache.metron.rest.controller.PcapController.fixed(PcapController.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
---
