Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/449
@nickwallen yes, I was
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/463
We actually need it for all of the tests that ensure actual parallelism and
I've strengthened the other test impacted. We had gotten around it by being
too lenient, honestly. The
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/449
Ok, I'm ok with that. The writer should be more adaptable here and that
shouldn't hold your PR up, agreed. Can we make it the enrichment queue, though?
---
If your project
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/463
Yeah, I can get behind that, @justinleet I'd state it slightly differently
though. It's not that we want to test the Stream so much as we want to test
that the Spliter
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/463
@mmiklavc The stream can consider itself parallel without it actually
functioning in parallel. If we have one big split, only one thread will be
used in the threadpool, so it'll
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/463
I reacted to your request, @justinleet . Also, I would like to put out a
request for the community:
The `delegatingSpliterator` pattern that I used uses polymorphism to
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/463
Thanks! I don't quite get why that works and mine returns one fewer split
though. Anyone have an insight on that?
---
If your project is set up for it, you can reply to this emai
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/463
@justinleet Thanks! I replaced the polymorphic delegate with the cleaner
mock version *and* I learned a bit about how mocking works.
---
If your project is set up for it, you can reply
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/464
METRON-740: Normalizing and adding log4j properties override where possible.
Add the ability to pass a log4j properties to MaaS CLI.
In order to streamline the review of the
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/465
METRON-741: Stellar Field Transformations should execute all of the
transformations, not just the output
Stellar Field Transformations should execute all of the transformations,
not
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/461
+1 by inspection, looks good! Thanks @justinleet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/466
METRON-742: Generated code for profile window selector DSL did not get
committed as part of METRON-690
Lest we be concerned this implies that the testing done on 690 is voided,
this was
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/466
Yep
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/466
It shouldn't because the tests are already run with the code submitted in
this PR. This is just about getting git in sync with what's generated.
---
If your project is set
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/467
METRON-743: Sort the files when reading results from Pcap
The FileSystem.listFiles() call does not return the files in sorted order,
which we assume for all FileSystem implementations
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/467
The performance penalties are minimal. The number of files will equal the
number of reducers, which does not scale with the data, and user specifiable.
Also we are just sorting the file
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/438
Ok, I'm +1 on this pending successful travis.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/467
Ok, I ran this up and tested it and I got the results I expect, but I'd
like some independent confirmation by @kylerichardson
---
If your project is set up for it, you can reply to
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/468
METRON-744: Allow Stellar functions to be loaded from HDFS
## Description
The benefit of Stellar is that adding new functionality is as simple as
providing a Jar. This enables people
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/449
Looks great, quick question. If I submit a profile that looks like:
```
{
"profiles": [
{
"profile": "test",
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/449#discussion_r103495230
--- Diff:
metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/KafkaDestinationHandler.java
---
@@ -0,0 +1,106
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/449#discussion_r103496490
--- Diff:
metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/KafkaDestinationHandler.java
---
@@ -0,0 +1,106
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Correct, though that is entirely hidden from us by the ClassIndex library.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
By the way, the following gist shows the pom and stellar function used in
this acceptance test script. Notice nothing is going on in the gist that would
make the author of the function
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103552286
--- Diff: metron-analytics/metron-profiler/pom.xml ---
@@ -339,6 +339,7
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103552339
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103560960
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103561250
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103561312
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Did you let it run for a bit? Also, there's a problem with enrichments
currently, I'm about to submit a new commit to fix a couple bugs.
---
If your project is set up for i
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
I fixed a ton of bugs here in the last commit, so if things were failing
for you before now, it's expected.
---
If your project is set up for it, you can reply to this email and
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Well, yeah, I'm going to add some docs today around the new stellar config.
Not sure what kind of diagram I could create, but I'm going to describe:
* The stellar conf
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103722138
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103724085
--- Diff: metron-analytics/metron-profiler/pom.xml ---
@@ -339,6 +339,7
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103722745
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
@nickwallen Yes, that's correct. I'd add one further modification to the
`ClasspathFunctionResolver` modification, it enables you to specify a
classloader and it replaces R
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Ok, I finished the docs and fixed some unit tests that were grumpy. I
think that a default hdfs directory really should wait for a followon JIRA and
done in the MPack. I'd a
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/316
Ok, this is a big one and has been around for a long time now in
Metron-years. Before we go ahead and pull the trigger on commit, I want to
make sure there are no outstanding
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/316
+1 by the way ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
I ran this through from the beginning to the end without issue to make sure
the stuff I did in the last few hours didn't affect the test.
The one issue, and this is off-
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/316
@merrimanr @jjmeyer0 I think we should make sure JJ gets credit for his
contribution on this. We should squash the commits into the minimal possible
each named "METRON-503: Metron
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/449
Yep, looks good, got my +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/316
For convenience, I staged the commit in a remote branch:
https://github.com/cestella/incubator-metron/tree/METRON-503_sandbox . I'd ask
that @merrimanr take it for a test drive.
-
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
@dlyle65535 Yeah, in this case, the file was physically not in HDFS though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Ok, in the aftermath of the API PR, I added support for properly setting up
stellar to read dependencies from HDFS for the API. As such, I validated the
API continues to work by
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/470
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/471#discussion_r104254466
--- Diff: .github/PULL_REQUEST_TEMPLATE.md ---
@@ -1,10 +1,9 @@
-Thank you for submitting a contribution to Apache Metron (Incubating
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/471
The benefit of having this in git is that when you go to
https://github.com/apache/incubator-metron/pulls, committers can see in one
screen which PRs have passing travis and how far along
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/473
METRON-712: Separate evaluation from parsing in Stellar
# Description
With the current implementation of Stellar, we cannot cache the parse tree
and then apply it after the fact
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/473
TODO: I still have yet to add some documentation around the
microbenchmarking suite.
TODO: Run this up in vagrant. I can't do so now as I am on hotel wifi, so
it'll be thi
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/475
+1 by inspection, great job @justinleet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/476
+1, great catch @mattf-horton !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/477
METRON-766: Release 0.3.1
* Remove the 0.3.0 bits in svn from release
* Add the 0.3.1 bits from RC5 to release
* Update the webpage with the 0.3.1 release
You can merge
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/477
You can try this out by running `bundle exec jekyll serve` from the site
directory, navigating to [http://localhost:4000](http://localhost:4000) and
clicking on the "Downloads"
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/478
METRON-767: Clean up license from METRON-622
Metron-622 put some verbiage at the end of the license that isn't necessary
and should be removed. This was pointed out during the
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/478
Nah, I slightly broadened the scope of this one, renamed the JIRA and will
adjust the title accordingly
---
If your project is set up for it, you can reply to this email and have your
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/477
METRON-766: Release 0.3.1
* Remove the 0.3.0 bits in svn from release
* Add the 0.3.1 bits from RC5 to release
* Update the webpage with the 0.3.1 release
You can merge
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/477
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/477
I'm kicking travis, but it appears that there are some intermittent test
failures in the REST API tests. In particular around deleting topics for the
KafkaControllerIntegratio
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/477
I'm going to have to kick travis again. It looks like that Kafka test in
the REST API has a number of intermittent issues. Seeing this exception:
```
Tests run: 2, Failur
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/477
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/477
METRON-766: Release 0.3.1
* Remove the 0.3.0 bits in svn from release
* Add the 0.3.1 bits from RC5 to release
* Update the webpage with the 0.3.1 release
You can merge
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/477
@merrimanr Any ideas on the cause of that one or a reasonable fix for it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/477
I created https://issues.apache.org/jira/browse/METRON-773 to track this
most recent issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/483#discussion_r106898305
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -132,6 +133,9 @@ public void
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/482
+1. This is just what we needed. Thanks, @justinleet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/481
Ok, fair points all-the-way-around. I'm convinced that we're thinking it
through. I'll give this a bit more of a careful look today. Thanks for the
contribution!
---
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/486
METRON-793: Migrate to storm-kafka-client kafka spout from storm-kafka
## Contributor Comments
In order to eventually support kerberos, the suggested path is to migrate
to the new
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107428522
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107429992
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/Deserializers.java
---
@@ -16,31 +16,35
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107430400
--- Diff: pom.xml ---
@@ -67,20 +67,44 @@
1.0.1
1.0.1
-0.10.0.1
+0.10.0
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
I owe:
* Better Javadocs around some of the pcap infrastructure
* An acceptance test plan
---
If your project is set up for it, you can reply to this email and have your
reply
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107428876
--- Diff:
metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107434843
--- Diff:
metron-analytics/metron-profiler/src/main/config/profiler.properties ---
@@ -39,4 +39,4 @@ profiler.hbase.flush.interval.seconds=30
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
# Testing Plan
## Preliminaries
* Please perform the following tests on the `full-dev` vagrant environment.
* Set an environment variable to indicate `METRON_HOME
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/487
+1 by inspection, this one bit me last night.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
I was planning to make a JIRA around the multiple topic bit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
@justinleet good points overall. Yeah, I'm not thrilled with the HDP
dependency either, as I stated already.
Regarding "If flux gets support for the Builder items, wou
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107681982
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
---
@@ -106,19 +105,26
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107682535
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
---
@@ -96,16 +109,18 @@ public
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107683042
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/Deserializers.java
---
@@ -0,0 +1,59
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107683860
--- Diff:
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
---
@@ -0,0 +1,234
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107683873
--- Diff:
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
---
@@ -0,0 +1,234
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
Follow-on JIRA to move back to apache storm for storm-kafka-client is at
https://issues.apache.org/jira/browse/METRON-794
---
If your project is set up for it, you can reply to this
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107687487
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
---
@@ -96,16 +109,18 @@ public
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/481#discussion_r107766918
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/IndexingConfigurations.java
---
@@ -73,7 +74,30
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/481#discussion_r107765415
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -92,22 +177,51 @@ public
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/490
DO NOT MERGE METRON-797: Pass security.protocol and enable auto-renew for
the storm topologies
## Contributor Comments
METRON-793 migrated the storm topologies to the storm
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/490
If you want to review just this work independent of 793 prior to 793
getting merged, you can find the commits:
*
[here](https://github.com/apache/incubator-metron/pull/490/commits
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107915471
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
---
@@ -106,19 +105,22
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r107915490
--- Diff:
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
---
@@ -0,0 +1,234
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/490
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/490
DO NOT MERGE METRON-797: Pass security.protocol and enable auto-renew for
the storm topologies
## Contributor Comments
METRON-793 migrated the storm topologies to the storm
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/490
# Testing Plan
## Preliminaries
* Please perform the following tests on the `full-dev` vagrant environment.
* Set an environment variable to indicate `METRON_HOME
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
@justinleet It is unclear and that test has had some intermittent issues,
especially under load, I've noticed. I added a more descriptive message to
help diagnose.
---
If your pr
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
I found a couple of spots where we might not be cleaning things up and
exceptions that are thrown that kill the spout that shouldn't. I've run the
test 10 times locally and wil
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/490
Good catch. I made the dependency explicit and created the JIRAs that I
know about for kerberos
[here](https://issues.apache.org/jira/browse/METRON-802?jql=labels%20%3D%20kerberos%20AND
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
Ok, update on the integration tests. It appears that the issue is that
storm 1.0.3 does not consistently kill the slots when shutting down. It times
out after a minute and dies. The
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/486
Ok, I ran it 20 times independently over the weekend and again through
travis whenever I had a moment and the tests appear stable after I kill the
slots manually as a last resort in the
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/488
Piling on, +1 Thanks for the quick turnaround, @justinleet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
201 - 300 of 1218 matches
Mail list logo
