I poked around the RC briefly and spun up full dev with and without sensors, no issues so far.
Jon On Fri, Dec 8, 2017 at 4:34 AM Matt Foley <ma...@apache.org> wrote: > Colleagues, > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/ > > Given the complexity of this RC, I’d appreciate if a couple people would > be willing to kick the tires before we put it up for a vote. > > I will myself be going thru the Verify Build process this weekend, as I > won’t be able to do it Friday. > > Thanks, > --Matt > > > On 12/4/17, 2:05 PM, "zeo...@gmail.com" <zeo...@gmail.com> wrote: > > Can we resolve the conversation regarding the second repo? I was > waiting > to get more input/preferences from people There's also a documentation > update that fixes a few broken Stellar docs that already has aa +1, I > just > need to merge it. > > Jon > > On Mon, Dec 4, 2017, 17:01 Casey Stella <ceste...@gmail.com> wrote: > > > I would be in favor of a release at this point. > > > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote: > > > > > Hey all, > > > I see METRON-1252 was resolved over the weekend. Shall I go ahead > and > > > start the process with 0.4.2 release? > > > Does anyone have any commits they feel strongly should go in > before 0.4.2 > > > is done, or are we ready to call it good? > > > > > > I believe there is consensus the 0.4.2 release should include a > release > > of > > > the current state of the metron-bro-plugin-kafka. I will continue > the > > > discussion in that thread as to the process for accomplishing > that, but > > > plan on it happening. > > > > > > Regards, > > > --Matt > > > > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote: > > > > > > Hope everyone (at least in the U.S.) had a great Thanksgiving > > holiday. > > > Regarding status of the release effort, still pending > METRON-1252, so > > > not making the release branch yet. > > > > > > Regards, > > > --Matt > > > > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote: > > > > > > (With release manager hat on) > > > > > > The community has proposed a release of Metron in the near > > future, > > > focusing on Meta-alerts running in Elasticsearch. > > > Congrats on getting so many of the below already done. At > this > > > point, only METRON-1252, and the discussion of how to handle joint > > release > > > of the Metron bro plugin, remain as gating items for the release. > I > > > project these will be resolved next week, so let’s propose the > following: > > > > > > Sometime next week, after the last bits are done, I’ll > start the > > > release process and create the release branch. > > > > > > The proposed new version will be 0.4.2, unless there are > backward > > > incompatible changes that support making it 0.5.0. > > > Currently there are NO included Jiras labeled > > > ‘backward-incompatible’, nor having Docs Text indicating so. > > > ***If anyone knows that some of the commits included since > 0.4.1 > > > introduce backward incompatibility, please say so now on this > thread, and > > > mark the Jira as such.*** > > > > > > The 90 or so jiras/commits already in master branch since > 0.4.1 > > > are listed below. > > > Thanks, > > > --Matt > > > > > > METRON-1301 Alerts UI - Sorting on Triage Score > Unexpectedly > > > Filters Some Records (nickwallen) closes apache/metron#832 > > > METRON-1294 IP addresses are not formatted correctly > in facet > > > and group results (merrimanr) closes apache/metron#827 > > > METRON-1291 Kafka produce REST endpoint does not work > in a > > > Kerberized cluster (merrimanr) closes apache/metron#826 > > > METRON-1290 Only first 10 alerts are update when a > MetaAlert > > > status is changed to inactive (justinleet) closes apache/metron#842 > > > METRON-1311 Service Check Should Check Elasticsearch > Index > > > Templates (nickwallen) closes apache/metron#839 > > > METRON-1289 Alert fields are lost when a MetaAlert is > created > > > (merrimanr) closes apache/metron#824 > > > METRON-1309 Change metron-deployment to pull the > plugin from > > > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837 > > > METRON-1310 Template Delete Action Deletes Search > Indices > > > (nickwallen) closes apache/metron#838 > > > METRON-1275: Fix Metron Documentation closes > > > apache/incubator-metron#833 > > > METRON-1295 Unable to Configure Logging for REST API > > > (nickwallen) closes apache/metron#828 > > > METRON-1307 Force install of java8 since java9 does not > > appear > > > to work with the scripts (brianhurley via ottobackwards) closes > > > apache/metron#835 > > > METRON-1296 Full Dev Fails to Deploy Index Templates > > > (nickwallen via cestella) closes apache/incubator-metron#829 > > > METRON-1281 Remove hard-coded indices from the Alerts > UI > > > (merrimanr) closes apache/metron#821 > > > METRON-1287 Full Dev Fails When Installing EPEL > Repository > > > (nickwallen) closes apache/metron#820 > > > METRON-1267 Alerts UI returns a 404 when refreshing the > > > alerts-list page (iraghumitra via merrimanr) closes > apache/metron#819 > > > METRON-1283 Install Elasticsearch template as a part > of the > > > mpack startup scripts (anandsubbu via nickwallen) closes > > apache/metron#817 > > > METRON-1254: Conditionals as map keys do not function > in > > > Stellar closes apache/incubator-metron#801 > > > METRON-1261 Apply bro security patch (JonZeolla via > > > ottobackwards) closes apache/metron#805 > > > METRON-1284 Remove extraneous dead query in > ElasticsearchDao > > > (justinleet) closes apache/metron#818 > > > METRON-1270: fix for warnings missing @return tag > argument in > > > metron-analytics/metron-profiler-common and metron-profiler-client > closes > > > apache/incubator-metron#810 > > > METRON-1272 Hide child alerts from searches and > grouping if > > > they belong to meta alerts (justinleet) closes apache/metron#811 > > > METRON-1224 Add time range selection to search control > > > (iraghumitra via james-sirota) closes apache/metron#796 > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects > > > (cestella via justinleet) closes apache/metron#816 > > > METRON-1243: Add a REST endpoint which allows us to > get a > > list > > > of all indice closes apache/incubator-metron#797 > > > METRON-1196 Increment master version number to 0.4.2 > for > > > on-going development (mattf-horton) closes apache/metron#767 > > > METRON-1278 Strip "Build Status" widget from > root > > > README.md in site-book build (mattf-horton) closes > apache/metron#815 > > > METRON-1274 Master has failure in > > > StormControllerIntegrationTest (merrimanr) closes apache/metron#813 > > > METRON-1266 Profiler - SASL Authentication Failed > > (nickwallen) > > > closes apache/metron#809 > > > METRON-1260 Include Alerts UI in Ambari Service Check > > > (nickwallen) closes apache/metron#804 > > > METRON-1251: Typo and formatting fixes for metron-rest > README > > > closes apache/incubator-metron#800 > > > METRON-1241: Enable the REST API to use a cache for the > > > zookeeper config similar to the Bolts closes > apache/incubator-metron#795 > > > METRON-1267 Alerts UI returns a 404 when refreshing the > > > alerts-list page (merrimanr) closes apache/metron#808 > > > METRON-1262 Unable to add comment for a alert in a > meta-alert > > > (merrimanr) closes apache/metron#806 > > > METRON-1263 Start Alerts UI service after Metron REST > > > (anandsubbu via nickwallen) closes apache/metron#807 > > > METRON-1255 MetaAlert search is not filtering on status > > > (merrimanr) closes apache/metron#802 > > > METRON-1249 Improve Metron MPack Service Checks > (nickwallen) > > > closes apache/metron#799 > > > METRON-1237 address javadoc warnings in > metron-maas-common > > > (dbist via james-sirota) closes apache/metron#792 > > > METRON-1240 address javadoc warnings in > metron-platform and > > > metron-analytics (dbist via james-sirota) closes apache/metron#794 > > > METRON-1226 Searching Can Errantly Query the Wrong > Indices > > > (nickwallen) closes apache/metron#793 > > > METRON-1081 Fix Alerts and Ops UI Notices file > (james-sirota) > > > closes apache/metron#682 > > > METRON-1123 Add group by option using faceted search > > > capabilities of metron-rest-api (iraghumitra via james-sirota) > closes > > > apache/metron#768 > > > METRON-1223 Add support to add comments for alerts > > > (iraghumitra via james-sirota) closes apache/metron#788 > > > METRON-1083 Add filters using faceted search > capabilities of > > > metron-rest-api (iraghumitra via james-sirota) closes > apache/metron#710 > > > METRON-1232 Alert status changes are not reflected in > list > > > view (iraghumitra via merrimanr) closes apache/metron#787 > > > METRON-1247 REST search and findOne endpoints return > > > unexpected or incorrect results for guids (justinleet) closes > > > apache/metron#798 > > > METRON-1235: Document the properties pulled from the > global > > > configuration closes apache/incubator-metron#791 > > > METRON-1234: fix for WARNING 'dependencies.dependency.( > > > groupId:artifactId:type:classifier)' must be unique: > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes > > > apache/metron#790 > > > METRON-1222: fix warning for The expression > ${parent.version} > > > is deprecated. Please use ${project.parent.version} instead. > (dbist via > > > mmiklavc) closes apache/metron#782 > > > METRON-1220 Create documentation around alert nested > field > > > (justinleet) closes apache/metron#780 > > > METRON-1229 Management UI type is part of the > declarations of > > > 2 modules (merrimanr) closes apache/metron#784 > > > METRON-1228: Configuration Management PUSH immediately > does > > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783 > > > METRON-1218 Metron REST should return better error > messages > > > (merrimanr) closes apache/metron#779 > > > METRON-1161 Add ability to edit parser command line > options > > in > > > the management UI (merrimanr) closes apache/metron#737 > > > METRON-1209: Make stellar repl take logging > properties, like > > > other CLI apps in metron closes apache/incubator-metron#772 > > > METRON-1059 address checkstyle warning AvoidStarImport > in > > > metron-stellar (dbist via ottobackwards) closes apache/metron#664 > > > METRON-1204 UI does not time out after being idle, but > stops > > > functioning (merrimanr) closes apache/metron#771 > > > METRON-1052: Add forensic similarity hash functions to > > Stellar > > > closes apache/incubator-metron#781 > > > METRON-632: Added validation of "shew.enrichmentType" > and > > > "shew.keyColumns" closes apache/incubator-metron#732 > > > METRON-1194 Add Profiler Debug Functions to Profiler > README > > > (nickwallen via ottobackwards) closes apache/metron#765 > > > METRON-1055 Metron 0.4.0 manual installation guide for > CentOS > > > 6 updates (lvets via ottobackwards) closes apache/metron#661 > > > METRON-1079 STELLAR NaN should be a keyword > (ottobackwards) > > > closes apache/metron#681 > > > METRON-1085 Add REST endpoint to save a user profile > for the > > > Alerts UI (merrimanr) closes apache/metron#694 > > > METRON-1208 MPack for Alerts UI (merrimanr) closes > > > apache/metron#778 > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes > > > apache/metron#777 > > > METRON-1215 Fix link to RPMs chapter (DimDroll via > > justinleet) > > > closes apache/metron#776 > > > METRON-1206 Make alerts UI conform to ops UI for > install > > > (merrimanr) closes apache/metron#773 > > > METRON-1195 Meta alerts improperly handle updates to > > non-alert > > > fields (justinleet) closes apache/metron#766 > > > METRON-1189 Add alert escalation to the Alerts UI > (merrimanr) > > > closes apache/metron#762 > > > METRON-1156 Simulate Triage Rules in the Stellar REPL > > > (nickwallen) closes apache/metron#733 > > > METRON-1198 Pycapa - No such configuration property > > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769 > > > METRON-1202 ElasticsearchDao Has extraneous sleep call > > > (justinleet) closes apache/metron#770 > > > METRON-938 "service metron-rest start <password>" does > not > > > work on CentOS 7. (justinleet) closes apache/metron#757 > > > METRON-1182 Refactor Code in alert list to accommodate > new > > > view types (iraghumitra via merrimanr) closes apache/metron#756 > > > METRON-1188: Ambari global configuration management > > (mmiklavc) > > > closes apache/metron#760 > > > METRON-1191 update public web site to point at 0.4.1 > new > > > release (mattf-horton) closes apache/metron#764 > > > METRON-1063 address javadoc warnings in metron-stellar > (dbist > > > via ottobackwards) closes apache/metron#668 > > > METRON-1190 Fix Meta Alert Type handling in > calculation of > > > scores (justinleet) closes apache/metron#763 > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not > Setup > > > Correctly (nickwallen) closes apache/metron#759 > > > METRON-1185: Stellar REPL does not work on a kerberized > > > cluster when calling functions interacting with HBase closes > > > apache/incubator-metron#755 > > > METRON-1186: Profiler Functions use classutils from > shaded > > > storm closes apache/incubator-metron#758 > > > METRON-1173: Fix pointers to old stellar docs closes > > > apache/incubator-metron#746 > > > METRON-1179: Make STATS_ADD to take a list closes > > > apache/incubator-metron#750 > > > METRON-1180: Make Stellar Shell accept zookeeper > quorum as a > > > CSV list and not require a port closes apache/incubator-metron#751 > > > METRON-1183 Improve KDC Setup Instructions (nickwallen) > > closes > > > apache/metron#753 > > > METRON-1177 Stale running topologies seen > post-kerberization > > > and cause exceptions (nickwallen) closes apache/metron#748 > > > METRON-1158 Build backend for grouping alerts into meta > > alerts > > > (justinleet) closes apache/metron#734 > > > METRON-1146: Add ability to parse JSON string into > JSONObject > > > for stellar closes apache/incubator-metron#727 > > > METRON-1176 REST: HDFS Service should support setting > > > permissions on files when writing (ottobackwards) closes > > apache/metron#749 > > > METRON-1114 Add group by capabilities to search REST > endpoint > > > (merrimanr) closes apache/metron#702 > > > METRON-1167 Define Session Specific Global > Configuration > > > Values in the REPL (nickwallen) closes apache/metron#740 > > > METRON-1171: Better validation for the SUBSTRING > stellar > > > function closes apache/incubator-metron#745 > > > > > > > > > > > > On 11/17/17, 11:59 AM, "Nick Allen" <n...@nickallen.org> > wrote: > > > > > > I just wanted to send an update on where we are at. > We've > > > gotten a lot > > > done here recently as you can see below. > > > > > > ✓ DONE (1) First, METRON-1289 needs to go in. This > one was > > > a fairly big > > > effort and I am hearing that we are pretty close. > > > > > > ✓ DONE (2) METRON-1294 fixes an issue in how field > types > > are > > > looked-up. > > > > > > ✓ DONE (3) METRON-1290 is next. While this may have > been > > > fixed in > > > M-1289, there may be some test cases we want from this > PR. > > > > > > ✓ DONE (4) METRON-1301 addresses a problem with the > sorting > > > logic. > > > > > > ✓ DONE (5) METRON-1291 fixes an issue with > escalation of > > > metaalerts. > > > > > > (6) That leads us to Raghu's UI work in > METRON-1252. This > > > introduces the > > > UI bits that depend on all the previous backend work. > > > > > > (7) At this point, we should have our best effort at > > running > > > Metaalerts > > > on Elasticsearch 2.x. I propose that we cut a release > here. > > > > > > (8) After we cut the release, we can introduce the > work for > > > ES 5.x in > > > METRON-939. I know we will need lots of help testing > and > > > reviewing this > > > one. > > > > > > > > > > > > We also have an outstanding question that needs > resolved > > > BEFORE we > > > release. We need to come to a consensus on how to > release > > > having moved our > > > Bro Plugin to a separate repo. I don't think we've > heard > > from > > > everyone on > > > this. I'd urge everyone to chime in so we can choose > a path > > > forward. > > > > > > If anyone is totally confused in regards to that > discussion, > > I > > > can try and > > > send an options summary again as a separate discuss > thread. > > > The original > > > chain was somewhere around here [1]. > > > > > > [1] > > > https://lists.apache.org/thread.html/ > > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@% > > > 3Cdev.metron.apache.org%3E > > > > > > > > > > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen < > > > n...@nickallen.org> wrote: > > > > > > > Hi Guys - > > > > > > > > I want to follow-up on this discussion. It sounds > like > > most > > > people are in > > > > agreement with the general approach. > > > > > > > > A lot of people have been working hard on Metaalerts > and > > > Elasticsearch. I > > > > have checked-in with those doing the heavy lifting > and have > > > compiled a more > > > > detailed plan based on where we are at now. To the > best of > > > my knowledge > > > > here is the plan of attack for finishing out this > effort. > > > > > > > > (1) First, METRON-1289 needs to go in. This one > was a > > > fairly big effort > > > > and I am hearing that we are pretty close. > > > > > > > > (2) METRON-1294 fixes an issue in how field types > are > > > looked-up. > > > > > > > > (3) METRON-1290 is next. While this may have been > fixed > > > in M-1289, > > > > there may be some test cases we want from this PR. > > > > > > > > (4) METRON-1301 addresses a problem with the > sorting > > logic. > > > > > > > > (5) METRON-1291 fixes an issue with escalation of > > > metaalerts. > > > > > > > > (6) That leads us to Raghu's UI work in > METRON-1252. > > This > > > introduces > > > > the UI bits that depend on all the previous backend > work. > > > > > > > > (7) At this point, we should have our best effort > at > > > running Metaalerts > > > > on Elasticsearch 2.x. I propose that we cut a > release here. > > > > > > > > (8) After we cut the release, we can introduce the > work > > > for ES 5.x in > > > > METRON-939. I know we will need lots of help > testing and > > > reviewing this > > > > one. > > > > > > > > Please correct me if I am wrong. I will try and > send out > > > updates as we > > > > make progress. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com < > > > zeo...@gmail.com> wrote: > > > > > > > >> I agree, I think it's very reasonable to move in > line with > > > Nick's > > > >> proposal. I would also suggest that we outline > what the > > > target versions > > > >> would be to add in the METRON-777 components, since > it has > > > been functional > > > >> for a very long time but not reviewed and has some > really > > > rockstar > > > >> improvements. > > > >> > > > >> Jon > > > >> > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler < > > > ottobackwa...@gmail.com> > > > >> wrote: > > > >> > > > >> > I think the ES cutover should be the start of the > 0.5.x > > > series, and we > > > >> > continue on with 0.4.x for the > > > >> > metadata improvements etc. We could chose to > focus > > > 0.5.x’s first > > > >> releases > > > >> > on not only ES but > > > >> > getting a handle on kibana and the mpack > situation as > > > well. > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > On November 6, 2017 at 12:48:45, Michael > Miklavcic ( > > > >> > michael.miklav...@gmail.com) wrote: > > > >> > > > > >> > I agree with your proposal, Nick. I think having a > > > stabilizing release > > > >> > prior to upgrading ES/Kibana makes sense. > > > >> > > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen < > > > n...@nickallen.org> wrote: > > > >> > > > > >> > > I would like to start a discussion around > upcoming > > > releases. We have a > > > >> > > couple separate significant tracks of work that > we > > need > > > to reconcile > > > >> in > > > >> > our > > > >> > > release schedule. > > > >> > > > > > >> > > (1) We have had (and have in review) a good > number of > > > bug fixes > > > >> required > > > >> > to > > > >> > > support Metaalerts on the existing > Elasticsearch 2.x > > > infrastructure. > > > >> > > > > > >> > > > > > >> > > (2) We also have ongoing work to upgrade our > > > infrastructure to > > > >> > > Elasticsearch 5.x, which will not be backwards > > > compatible. > > > >> > > > > > >> > > > > > >> > > I would like to see a release that has our best > work > > on > > > ES 2.x before > > > >> we > > > >> > > migrate to 5.x. I would propose the following. > > > >> > > > > > >> > > Release N+1: Introduce Metaalerts running on ES > 2.x > > > >> > > > > > >> > > Release N+2: Cut-over to ES 5.x > > > >> > > > > > >> > > > > > >> > > (Q) Is it worth cutting a separate release for > ES 2.x? > > > Is there a > > > >> better > > > >> > > way to handle the cut-over to 5.x? > > > >> > > > > > >> > > > > >> -- > > > >> > > > >> Jon > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Jon > > > > -- Jon
