[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/156 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67512583
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+
+import java.io.BufferedOutputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.PrintStream;
+import java.nio.charset.StandardCharsets;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.EnumMap;
+import java.util.List;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+import static org.mockito.Matchers.*;
+import static org.mockito.Mockito.when;
+
+public class PcapCliTest {
+
+ @Mock
+ private PcapJob jobRunner;
+ @Mock
+ private ResultsWriter resultsWriter;
+ @Mock
+ private Clock clock;
+
+ @Before
+ public void setup() {
+MockitoAnnotations.initMocks(this);
+ }
+
+ @Test
+ public void runs_fixed_pcap_filter_job_with_default_argument_list()
throws Exception {
+String[] args = {
+"fixed",
+"-start_time", "500",
--- End diff --
I thought these were long args now, how does - work as opposed to --?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67512176
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
--- End diff --
This is addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67424320
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
---
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.*;
+import org.apache.metron.common.Constants;
+
+public class FixedCliParser extends CliParser {
+ private Options fixedOptions;
+
+ public FixedCliParser() {
+fixedOptions = buildFixedOptions();
+ }
+
+ private Options buildFixedOptions() {
+Options options = buildOptions();
+options.addOption(newOption("srcAddr", true, "Source IP address"));
--- End diff --
Can we conform to the naming convention around src/dst addr/port and change:
* `srcAddr` to `ip_src_addr` as the long option
* `dstAddr` to `ip_dst_addr` as the long option
* `srcPort` to `ip_src_port` as the long option
* `dstPort` to `ip_dst_port` as the long option
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67422356
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
--- End diff --
Nope, this is fine by me. I just wanted to make sure it wasn't to skip
over hadoop options because we weren't using GenericOptionsParser. I retract
the comment. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67422278
--- Diff:
metron-platform/metron-pcap-backend/src/main/scripts/pcap_query.sh ---
@@ -0,0 +1,34 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default}
+[ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && .
${BIGTOP_DEFAULTS_DIR}/hbase
+
+# Autodetect JAVA_HOME if not defined
+if [ -e /usr/libexec/bigtop-detect-javahome ]; then
+ . /usr/libexec/bigtop-detect-javahome
+elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
+ . /usr/lib/bigtop-utils/bigtop-detect-javahome
+fi
+
+export METRON_VERSION=${project.version}
+export METRON_HOME=/usr/metron/$METRON_VERSION
+export API_JAR=${project.artifactId}-$METRON_VERSION.jar
--- End diff --
Good catch - originally had this in the API project.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67422111
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
--- End diff --
I did that for Context-sensitive arguments, e.g.
$ ./pcap_query.sh fixed -srcAddr 1 -destAddr 2 -srcPort 8080 -destPort
8081...
and
$ ./pcap_query.sh query -query "my query string"...
Is there a better way to handle this in Commons CLI? groupoptions sounds
correct, but it's a set of mutually exclusive options.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67422117
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * Provides commmon required fields for the PCAP filter jobs
+ */
+public class CliParser {
+
+ public Options buildOptions() {
+Options options = new Options();
+options.addOption(newOption("h", false, "Display help"));
+options.addOption(newOption("basePath", true, String.format("Base PCAP
data path. Default is '%s'", CliConfig.BASE_PATH_DEFAULT)));
+options.addOption(newOption("baseOutputPath", true,
String.format("Query result output path. Default is '%s'",
CliConfig.BASE_OUTPUT_PATH_DEFAULT)));
+options.addOption(newOption("startTime", true, "Packet start time
range. Default is '0'"));
--- End diff --
Can we specify a date format to use to parse the start and end times? I
suspect that's a useful piece of sugar as it would require people to not have
to look up the epoch time translation every time they want to run this for a
time range. I'd just pass an arg with the date format to use. If it's not
specified, then assume epoch time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67421802
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * Provides commmon required fields for the PCAP filter jobs
+ */
+public class CliParser {
+
+ public Options buildOptions() {
+Options options = new Options();
+options.addOption(newOption("h", false, "Display help"));
+options.addOption(newOption("basePath", true, String.format("Base PCAP
data path. Default is '%s'", CliConfig.BASE_PATH_DEFAULT)));
+options.addOption(newOption("baseOutputPath", true,
String.format("Query result output path. Default is '%s'",
CliConfig.BASE_OUTPUT_PATH_DEFAULT)));
+options.addOption(newOption("startTime", true, "Packet start time
range. Default is '0'"));
--- End diff --
start time should be required. As it stands, the defaults, if unspecified
is to return everything. That is likely not what we want and will result in a
VERY big MR job in some of the installations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67421199
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
+ } catch (ParseException e) {
+System.out.println(e.getMessage());
+fixedParser.printHelp();
--- End diff --
Also, would be nice for printHelp to take the stream as an arg
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67421145
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
+ } catch (ParseException e) {
+System.out.println(e.getMessage());
--- End diff --
We should probably print to stderr here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67420651
--- Diff:
metron-platform/metron-pcap-backend/src/main/scripts/pcap_query.sh ---
@@ -0,0 +1,34 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default}
+[ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && .
${BIGTOP_DEFAULTS_DIR}/hbase
+
+# Autodetect JAVA_HOME if not defined
+if [ -e /usr/libexec/bigtop-detect-javahome ]; then
+ . /usr/libexec/bigtop-detect-javahome
+elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
+ . /usr/lib/bigtop-utils/bigtop-detect-javahome
+fi
+
+export METRON_VERSION=${project.version}
+export METRON_HOME=/usr/metron/$METRON_VERSION
+export API_JAR=${project.artifactId}-$METRON_VERSION.jar
--- End diff --
Can we call the `API_JAR` variable something else since it's coming from
the metron-pcap-backend project? Perhaps `PCAP_BACKEND_JAR`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67420354
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliParser.java
---
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.*;
+
+public class QueryCliParser extends CliParser {
+ private Options queryOptions;
+
+ public QueryCliParser() {
+queryOptions = setupOptions();
+ }
+
+ private Options setupOptions() {
+Options options = buildOptions();
+options.addOption(newOption("query", true, "Query string to use as a
filter"));
+return options;
+ }
+
+ /**
+ * Parses query pcap filter options and required parameters common to
all filter types.
+ *
+ * @param args command line arguments to parse
+ * @return Configuration tailored to query pcap queries
+ * @throws ParseException
+ */
+ public QueryCliConfig parse(String[] args) throws ParseException {
+CommandLineParser parser = new BasicParser();
--- End diff --
All the existing utilities use PosixParser, can we move to that to conform?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67420179
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
--- End diff --
Why are we starting at 1 here? Is it to skip over some hadoop configs? If
so, see above comment about GenericOptionsParser for a more general approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67420022
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
--- End diff --
Don't we need to use GenericOptionsParser(conf, args).getRemainingArgs()?
Otherwise we're going to get all the hadoop args if we need to specify specific
`-D` stuff for the MR job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67419245
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * Provides commmon required fields for the PCAP filter jobs
+ */
+public class CliParser {
+
+ public Options buildOptions() {
+Options options = new Options();
+options.addOption(newOption("h", false, "Display help"));
+options.addOption(newOption("basePath", true, String.format("Base PCAP
data path. Default is '%s'", CliConfig.BASE_PATH_DEFAULT)));
--- End diff --
Is this a short option as well as a long option? I tend to like having
both at our disposal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
