[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/1135 ---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1135#discussion_r206639730 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/PcapService.java --- @@ -43,5 +44,5 @@ InputStream getRawPcap(String username, String jobId, Integer page) throws RestException; - + Map getConfiguration(String username, String jobId) throws RestException; --- End diff -- I added javadocs for this service. ---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206639767
--- Diff: metron-interface/metron-rest/README.md ---
@@ -556,6 +558,14 @@ Request and Response objects are JSON formatted. The
JSON schemas are available
* jobId - Job ID of submitted job
* Returns:
* 200 - Kills passed job.
+
+### `GET /api/v1/pcap/{jobId}/configuration`
--- End diff --
I agree. Done.
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206639667
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/PcapServiceImpl.java
---
@@ -199,6 +208,37 @@ public InputStream getRawPcap(String username, String
jobId, Integer page) throw
return inputStream;
}
+ @Override
+ public Map getConfiguration(String username, String
jobId) throws RestException {
+Map configuration = new HashMap<>();
+try {
+ Map jobConfiguration = jobManager.getJob(username,
jobId).getConfiguration();
+ configuration.put(PcapOptions.BASE_PATH.getKey(),
PcapOptions.BASE_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.FINAL_OUTPUT_PATH.getKey(),
PcapOptions.FINAL_OUTPUT_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.START_TIME_MS.getKey(),
PcapOptions.START_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.END_TIME_MS.getKey(),
PcapOptions.END_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.NUM_REDUCERS.getKey(),
PcapOptions.NUM_REDUCERS.get(jobConfiguration, Integer.class));
+
+ boolean isFixedFilter =
PcapOptions.FILTER_IMPL.get(jobConfiguration, PcapFilterConfigurator.class)
instanceof FixedPcapFilter.Configurator;
+ if (isFixedFilter) {
+configuration.put(FixedPcapOptions.IP_SRC_ADDR.getKey(),
FixedPcapOptions.IP_SRC_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_ADDR.getKey(),
FixedPcapOptions.IP_DST_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_SRC_PORT.getKey(),
FixedPcapOptions.IP_SRC_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_PORT.getKey(),
FixedPcapOptions.IP_DST_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PROTOCOL.getKey(),
FixedPcapOptions.PROTOCOL.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PACKET_FILTER.getKey(),
FixedPcapOptions.PACKET_FILTER.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.INCLUDE_REVERSE.getKey(),
FixedPcapOptions.INCLUDE_REVERSE.get(jobConfiguration, String.class));
+ } else {
+configuration.put(QueryPcapOptions.QUERY.getKey(),
QueryPcapOptions.QUERY.get(jobConfiguration, String.class));
+ }
+} catch (JobNotFoundException e) {
+ // do nothing and return null pcapStatus
--- End diff --
I replaced the empty catch blocks with a log warning. Will that work?
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206639511
--- Diff:
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/PcapControllerIntegrationTest.java
---
@@ -420,5 +420,55 @@ public void testRawDownload() throws Exception {
.andExpect(status().isNotFound());
}
+ @Test
+ public void testGetFixedFilterConfiguration() throws Exception {
+MockPcapJob mockPcapJob = (MockPcapJob) wac.getBean("mockPcapJob");
+
+mockPcapJob.setStatus(new
JobStatus().withJobId("jobId").withState(JobStatus.State.RUNNING));
+
+this.mockMvc.perform(post(pcapUrl + "/fixed").with(httpBasic(user,
password)).with(csrf()).contentType(MediaType.parseMediaType("application/json;charset=UTF-8")).content(fixedJson))
+.andExpect(status().isOk())
+
.andExpect(content().contentType(MediaType.parseMediaType("application/json;charset=UTF-8")))
+.andExpect(jsonPath("$.jobId").value("jobId"))
+.andExpect(jsonPath("$.jobStatus").value("RUNNING"));
+
+this.mockMvc.perform(get(pcapUrl +
"/jobId/configuration").with(httpBasic(user, password)))
+.andExpect(status().isOk())
+
.andExpect(content().contentType(MediaType.parseMediaType("application/json;charset=UTF-8")))
+.andExpect(jsonPath("$.basePath").value("/base/path"))
+
.andExpect(jsonPath("$.finalOutputPath").value("/final/output/path"))
+.andExpect(jsonPath("$.startTimeMs").value(10))
+.andExpect(jsonPath("$.endTimeMs").value(20))
+.andExpect(jsonPath("$.numReducers").value(2))
+.andExpect(jsonPath("$.ipSrcAddr").value("192.168.1.2"))
+.andExpect(jsonPath("$.ipDstAddr").value("192.168.1.1"))
+.andExpect(jsonPath("$.ipSrcPort").value("2000"))
+.andExpect(jsonPath("$.ipDstPort").value("1000"))
+.andExpect(jsonPath("$.protocol").value("TCP"))
+.andExpect(jsonPath("$.packetFilter").value("filter"))
+.andExpect(jsonPath("$.includeReverse").value("true"));
+ }
+ @Test
+ public void testGeQueryFilterConfiguration() throws Exception {
--- End diff --
Done
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206639491
--- Diff:
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/PcapServiceImplTest.java
---
@@ -639,4 +642,78 @@ public void getRawShouldThrowException() throws
Exception {
pcapService.getRawPcap("user", "jobId", 1);
}
+ @Test
+ public void
getConfigurationShouldProperlyReturnFixedFilterConfiguration() throws Exception
{
+FixedPcapRequest fixedPcapRequest = new FixedPcapRequest();
+fixedPcapRequest.setBasePath("basePath");
+fixedPcapRequest.setBaseInterimResultPath("baseOutputPath");
+fixedPcapRequest.setFinalOutputPath("finalOutputPath");
+fixedPcapRequest.setStartTimeMs(1L);
+fixedPcapRequest.setEndTimeMs(2L);
+fixedPcapRequest.setNumReducers(2);
+fixedPcapRequest.setIpSrcAddr("ip_src_addr");
+fixedPcapRequest.setIpDstAddr("ip_dst_addr");
+fixedPcapRequest.setIpSrcPort(1000);
+fixedPcapRequest.setIpDstPort(2000);
+fixedPcapRequest.setProtocol("tcp");
+fixedPcapRequest.setPacketFilter("filter");
+fixedPcapRequest.setIncludeReverse(true);
+MockPcapJob mockPcapJob = new MockPcapJob();
+mockPcapJobSupplier.setMockPcapJob(mockPcapJob);
+JobManager jobManager = new InMemoryJobManager<>();
+
+PcapServiceImpl pcapService = spy(new PcapServiceImpl(environment,
configuration, mockPcapJobSupplier, jobManager, pcapToPdmlScriptWrapper));
+FileSystem fileSystem = mock(FileSystem.class);
+doReturn(fileSystem).when(pcapService).getFileSystem();
+mockPcapJob.setStatus(new JobStatus()
+.withJobId("jobId"));
+
+pcapService.submit("user", fixedPcapRequest);
+
+Map configuration =
pcapService.getConfiguration("user", "jobId");
+Assert.assertEquals("basePath",
PcapOptions.BASE_PATH.get(configuration, String.class));
+Assert.assertEquals("finalOutputPath",
PcapOptions.FINAL_OUTPUT_PATH.get(configuration, String.class));
+Assert.assertEquals(1L, PcapOptions.START_TIME_MS.get(configuration,
Long.class).longValue());
+Assert.assertEquals(2L, PcapOptions.END_TIME_MS.get(configuration,
Long.class).longValue());
+Assert.assertEquals(2, PcapOptions.NUM_REDUCERS.get(configuration,
Integer.class).intValue());
+Assert.assertEquals("ip_src_addr",
FixedPcapOptions.IP_SRC_ADDR.get(configuration, String.class));
+Assert.assertEquals("ip_dst_addr",
FixedPcapOptions.IP_DST_ADDR.get(configuration, String.class));
+Assert.assertEquals(1000,
FixedPcapOptions.IP_SRC_PORT.get(configuration, Integer.class).intValue());
+Assert.assertEquals(2000,
FixedPcapOptions.IP_DST_PORT.get(configuration, Integer.class).intValue());
+Assert.assertEquals("tcp",
FixedPcapOptions.PROTOCOL.get(configuration, String.class));
+Assert.assertEquals("filter",
FixedPcapOptions.PACKET_FILTER.get(configuration, String.class));
+Assert.assertEquals(true,
FixedPcapOptions.INCLUDE_REVERSE.get(configuration, Boolean.class));
+ }
+
+ @Test
+ public void
getConfigurationShouldProperlyReturnQueryFilterConfiguration() throws Exception
{
--- End diff --
I added a couple test cases for exception handling.
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206625671
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/PcapServiceImpl.java
---
@@ -199,6 +208,37 @@ public InputStream getRawPcap(String username, String
jobId, Integer page) throw
return inputStream;
}
+ @Override
+ public Map getConfiguration(String username, String
jobId) throws RestException {
+Map configuration = new HashMap<>();
+try {
+ Map jobConfiguration = jobManager.getJob(username,
jobId).getConfiguration();
+ configuration.put(PcapOptions.BASE_PATH.getKey(),
PcapOptions.BASE_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.FINAL_OUTPUT_PATH.getKey(),
PcapOptions.FINAL_OUTPUT_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.START_TIME_MS.getKey(),
PcapOptions.START_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.END_TIME_MS.getKey(),
PcapOptions.END_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.NUM_REDUCERS.getKey(),
PcapOptions.NUM_REDUCERS.get(jobConfiguration, Integer.class));
+
+ boolean isFixedFilter =
PcapOptions.FILTER_IMPL.get(jobConfiguration, PcapFilterConfigurator.class)
instanceof FixedPcapFilter.Configurator;
+ if (isFixedFilter) {
+configuration.put(FixedPcapOptions.IP_SRC_ADDR.getKey(),
FixedPcapOptions.IP_SRC_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_ADDR.getKey(),
FixedPcapOptions.IP_DST_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_SRC_PORT.getKey(),
FixedPcapOptions.IP_SRC_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_PORT.getKey(),
FixedPcapOptions.IP_DST_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PROTOCOL.getKey(),
FixedPcapOptions.PROTOCOL.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PACKET_FILTER.getKey(),
FixedPcapOptions.PACKET_FILTER.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.INCLUDE_REVERSE.getKey(),
FixedPcapOptions.INCLUDE_REVERSE.get(jobConfiguration, String.class));
+ } else {
+configuration.put(QueryPcapOptions.QUERY.getKey(),
QueryPcapOptions.QUERY.get(jobConfiguration, String.class));
+ }
+} catch (JobNotFoundException e) {
+ // do nothing and return null pcapStatus
--- End diff --
We should log here.
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206623131
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/PcapServiceImpl.java
---
@@ -199,6 +208,37 @@ public InputStream getRawPcap(String username, String
jobId, Integer page) throw
return inputStream;
}
+ @Override
+ public Map getConfiguration(String username, String
jobId) throws RestException {
+Map configuration = new HashMap<>();
+try {
+ Map jobConfiguration = jobManager.getJob(username,
jobId).getConfiguration();
+ configuration.put(PcapOptions.BASE_PATH.getKey(),
PcapOptions.BASE_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.FINAL_OUTPUT_PATH.getKey(),
PcapOptions.FINAL_OUTPUT_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.START_TIME_MS.getKey(),
PcapOptions.START_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.END_TIME_MS.getKey(),
PcapOptions.END_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.NUM_REDUCERS.getKey(),
PcapOptions.NUM_REDUCERS.get(jobConfiguration, Integer.class));
+
+ boolean isFixedFilter =
PcapOptions.FILTER_IMPL.get(jobConfiguration, PcapFilterConfigurator.class)
instanceof FixedPcapFilter.Configurator;
+ if (isFixedFilter) {
+configuration.put(FixedPcapOptions.IP_SRC_ADDR.getKey(),
FixedPcapOptions.IP_SRC_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_ADDR.getKey(),
FixedPcapOptions.IP_DST_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_SRC_PORT.getKey(),
FixedPcapOptions.IP_SRC_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_PORT.getKey(),
FixedPcapOptions.IP_DST_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PROTOCOL.getKey(),
FixedPcapOptions.PROTOCOL.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PACKET_FILTER.getKey(),
FixedPcapOptions.PACKET_FILTER.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.INCLUDE_REVERSE.getKey(),
FixedPcapOptions.INCLUDE_REVERSE.get(jobConfiguration, String.class));
+ } else {
+configuration.put(QueryPcapOptions.QUERY.getKey(),
QueryPcapOptions.QUERY.get(jobConfiguration, String.class));
+ }
+} catch (JobNotFoundException e) {
+ // do nothing and return null pcapStatus
--- End diff --
The pattern is used in other places in the class and the comment was
copied. I updated it to say an empty map is returned.
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206607449
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/PcapServiceImpl.java
---
@@ -199,6 +208,37 @@ public InputStream getRawPcap(String username, String
jobId, Integer page) throw
return inputStream;
}
+ @Override
+ public Map getConfiguration(String username, String
jobId) throws RestException {
+Map configuration = new HashMap<>();
+try {
+ Map jobConfiguration = jobManager.getJob(username,
jobId).getConfiguration();
--- End diff --
Will `jobManager.getJob(username, jobId)` never return null? I see null
checks in other places where similar calls are made.
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1135#discussion_r206609288 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/PcapService.java --- @@ -43,5 +44,5 @@ InputStream getRawPcap(String username, String jobId, Integer page) throws RestException; - + Map getConfiguration(String username, String jobId) throws RestException; --- End diff -- Would be nice to see some javadocs here, but that doesn't seem unique to this PR. Maybe in another PR we can get some javadocs in here. ---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206611012
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/PcapServiceImpl.java
---
@@ -199,6 +208,37 @@ public InputStream getRawPcap(String username, String
jobId, Integer page) throw
return inputStream;
}
+ @Override
+ public Map getConfiguration(String username, String
jobId) throws RestException {
+Map configuration = new HashMap<>();
+try {
+ Map jobConfiguration = jobManager.getJob(username,
jobId).getConfiguration();
+ configuration.put(PcapOptions.BASE_PATH.getKey(),
PcapOptions.BASE_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.FINAL_OUTPUT_PATH.getKey(),
PcapOptions.FINAL_OUTPUT_PATH.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.START_TIME_MS.getKey(),
PcapOptions.START_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.END_TIME_MS.getKey(),
PcapOptions.END_TIME_MS.get(jobConfiguration, String.class));
+ configuration.put(PcapOptions.NUM_REDUCERS.getKey(),
PcapOptions.NUM_REDUCERS.get(jobConfiguration, Integer.class));
+
+ boolean isFixedFilter =
PcapOptions.FILTER_IMPL.get(jobConfiguration, PcapFilterConfigurator.class)
instanceof FixedPcapFilter.Configurator;
+ if (isFixedFilter) {
+configuration.put(FixedPcapOptions.IP_SRC_ADDR.getKey(),
FixedPcapOptions.IP_SRC_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_ADDR.getKey(),
FixedPcapOptions.IP_DST_ADDR.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_SRC_PORT.getKey(),
FixedPcapOptions.IP_SRC_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.IP_DST_PORT.getKey(),
FixedPcapOptions.IP_DST_PORT.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PROTOCOL.getKey(),
FixedPcapOptions.PROTOCOL.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.PACKET_FILTER.getKey(),
FixedPcapOptions.PACKET_FILTER.get(jobConfiguration, String.class));
+configuration.put(FixedPcapOptions.INCLUDE_REVERSE.getKey(),
FixedPcapOptions.INCLUDE_REVERSE.get(jobConfiguration, String.class));
+ } else {
+configuration.put(QueryPcapOptions.QUERY.getKey(),
QueryPcapOptions.QUERY.get(jobConfiguration, String.class));
+ }
+} catch (JobNotFoundException e) {
+ // do nothing and return null pcapStatus
--- End diff --
Is this comment correct? Seems like we return an empty configuration map
(assuming the call to getJob does not return null).
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206611264
--- Diff:
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/PcapControllerIntegrationTest.java
---
@@ -420,5 +420,55 @@ public void testRawDownload() throws Exception {
.andExpect(status().isNotFound());
}
+ @Test
+ public void testGetFixedFilterConfiguration() throws Exception {
+MockPcapJob mockPcapJob = (MockPcapJob) wac.getBean("mockPcapJob");
+
+mockPcapJob.setStatus(new
JobStatus().withJobId("jobId").withState(JobStatus.State.RUNNING));
+
+this.mockMvc.perform(post(pcapUrl + "/fixed").with(httpBasic(user,
password)).with(csrf()).contentType(MediaType.parseMediaType("application/json;charset=UTF-8")).content(fixedJson))
+.andExpect(status().isOk())
+
.andExpect(content().contentType(MediaType.parseMediaType("application/json;charset=UTF-8")))
+.andExpect(jsonPath("$.jobId").value("jobId"))
+.andExpect(jsonPath("$.jobStatus").value("RUNNING"));
+
+this.mockMvc.perform(get(pcapUrl +
"/jobId/configuration").with(httpBasic(user, password)))
+.andExpect(status().isOk())
+
.andExpect(content().contentType(MediaType.parseMediaType("application/json;charset=UTF-8")))
+.andExpect(jsonPath("$.basePath").value("/base/path"))
+
.andExpect(jsonPath("$.finalOutputPath").value("/final/output/path"))
+.andExpect(jsonPath("$.startTimeMs").value(10))
+.andExpect(jsonPath("$.endTimeMs").value(20))
+.andExpect(jsonPath("$.numReducers").value(2))
+.andExpect(jsonPath("$.ipSrcAddr").value("192.168.1.2"))
+.andExpect(jsonPath("$.ipDstAddr").value("192.168.1.1"))
+.andExpect(jsonPath("$.ipSrcPort").value("2000"))
+.andExpect(jsonPath("$.ipDstPort").value("1000"))
+.andExpect(jsonPath("$.protocol").value("TCP"))
+.andExpect(jsonPath("$.packetFilter").value("filter"))
+.andExpect(jsonPath("$.includeReverse").value("true"));
+ }
+ @Test
+ public void testGeQueryFilterConfiguration() throws Exception {
--- End diff --
typo... testGeQueryFilterConfiguration != testGetQueryFilterConfiguration
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206605219
--- Diff: metron-interface/metron-rest/README.md ---
@@ -556,6 +558,14 @@ Request and Response objects are JSON formatted. The
JSON schemas are available
* jobId - Job ID of submitted job
* Returns:
* 200 - Kills passed job.
+
+### `GET /api/v1/pcap/{jobId}/configuration`
--- End diff --
Small nit: In the existing endpoints, we have tended to use "config"
instead of "configuration". Would it make sense to use "config" here for
consistency?
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1135#discussion_r206611838
--- Diff:
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/PcapServiceImplTest.java
---
@@ -639,4 +642,78 @@ public void getRawShouldThrowException() throws
Exception {
pcapService.getRawPcap("user", "jobId", 1);
}
+ @Test
+ public void
getConfigurationShouldProperlyReturnFixedFilterConfiguration() throws Exception
{
+FixedPcapRequest fixedPcapRequest = new FixedPcapRequest();
+fixedPcapRequest.setBasePath("basePath");
+fixedPcapRequest.setBaseInterimResultPath("baseOutputPath");
+fixedPcapRequest.setFinalOutputPath("finalOutputPath");
+fixedPcapRequest.setStartTimeMs(1L);
+fixedPcapRequest.setEndTimeMs(2L);
+fixedPcapRequest.setNumReducers(2);
+fixedPcapRequest.setIpSrcAddr("ip_src_addr");
+fixedPcapRequest.setIpDstAddr("ip_dst_addr");
+fixedPcapRequest.setIpSrcPort(1000);
+fixedPcapRequest.setIpDstPort(2000);
+fixedPcapRequest.setProtocol("tcp");
+fixedPcapRequest.setPacketFilter("filter");
+fixedPcapRequest.setIncludeReverse(true);
+MockPcapJob mockPcapJob = new MockPcapJob();
+mockPcapJobSupplier.setMockPcapJob(mockPcapJob);
+JobManager jobManager = new InMemoryJobManager<>();
+
+PcapServiceImpl pcapService = spy(new PcapServiceImpl(environment,
configuration, mockPcapJobSupplier, jobManager, pcapToPdmlScriptWrapper));
+FileSystem fileSystem = mock(FileSystem.class);
+doReturn(fileSystem).when(pcapService).getFileSystem();
+mockPcapJob.setStatus(new JobStatus()
+.withJobId("jobId"));
+
+pcapService.submit("user", fixedPcapRequest);
+
+Map configuration =
pcapService.getConfiguration("user", "jobId");
+Assert.assertEquals("basePath",
PcapOptions.BASE_PATH.get(configuration, String.class));
+Assert.assertEquals("finalOutputPath",
PcapOptions.FINAL_OUTPUT_PATH.get(configuration, String.class));
+Assert.assertEquals(1L, PcapOptions.START_TIME_MS.get(configuration,
Long.class).longValue());
+Assert.assertEquals(2L, PcapOptions.END_TIME_MS.get(configuration,
Long.class).longValue());
+Assert.assertEquals(2, PcapOptions.NUM_REDUCERS.get(configuration,
Integer.class).intValue());
+Assert.assertEquals("ip_src_addr",
FixedPcapOptions.IP_SRC_ADDR.get(configuration, String.class));
+Assert.assertEquals("ip_dst_addr",
FixedPcapOptions.IP_DST_ADDR.get(configuration, String.class));
+Assert.assertEquals(1000,
FixedPcapOptions.IP_SRC_PORT.get(configuration, Integer.class).intValue());
+Assert.assertEquals(2000,
FixedPcapOptions.IP_DST_PORT.get(configuration, Integer.class).intValue());
+Assert.assertEquals("tcp",
FixedPcapOptions.PROTOCOL.get(configuration, String.class));
+Assert.assertEquals("filter",
FixedPcapOptions.PACKET_FILTER.get(configuration, String.class));
+Assert.assertEquals(true,
FixedPcapOptions.INCLUDE_REVERSE.get(configuration, Boolean.class));
+ }
+
+ @Test
+ public void
getConfigurationShouldProperlyReturnQueryFilterConfiguration() throws Exception
{
--- End diff --
Is there a test case for trying to get the configuration of a job ID that
does not exist?
---
[GitHub] metron pull request #1135: METRON-1700: Create REST endpoint to get job conf...
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1135
METRON-1700: Create REST endpoint to get job configuration
## Contributor Comments
This PR adds a REST endpoint that returns the job configuration for a
submitted job. The endpoint returns a static list of properties that can be
serialized and are useful for frontend applications. Changes outside of REST
were trivial because the PcapJob class already stores the configuration.
### Changes Included
- Added a method to the Statusable interface that returns a configuration
map
- Added a method to the PcapService that filters the Statusable
configuration and returns a subset of properties
- Update unit and integration tests
### Testing
This has been tested in full dev. Testing instructions are as follows:
1. Spin up full dev and put pcap data in the `/apps/metron/pcap/input` HDFS
directory
2. Submit a pcap query:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{}' 'http://node1:8082/api/v1/pcap/fixed'
```
A job status should be returned as usual.
3. Now the configuration can be retrieved with the get configuration
endpoint:
```
curl -X GET --header 'Accept: application/json'
'http://node1:8082/api/v1/pcap/job_1532356159290_0052/configuration'
```
A subset of properties, including those in the pcap query:
```
{
"finalOutputPath": "/apps/metron/pcap/output",
"ipSrcPort": "",
"includeReverse": false,
"startTimeMs": 1449010818000,
"endTimeMs": 1532642418000,
"protocol": "",
"ipDstAddr": "",
"basePath": "/apps/metron/pcap/input",
"packetFilter": "",
"ipSrcAddr": "",
"ipDstPort": "",
"numReducers": 10,
}
```
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-1700
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1135.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in
