I think its important to keep moving toward a Metron where I can introduce new telemetry, add new enrichments, tweak the triage process, iteratively, on-the-fly, and safely in a single, production environment.
If a user wants to have a traditional test-to-production change management process, they can, but it is not necessary. I think the base that we have allows us to do this. In that regard, I do like the idea, Otto. On Sat, Feb 18, 2017 at 8:51 AM, Otto Fowler <[email protected]> wrote: > I agree with this. We could really do something cool here if we have > multiple training topologies, like steps > that fed the ui. > > Step one would be kafka to parser with output to some tmp area. The ui > would be used to train the parser/stellar configs > Step two would add on to enrichments to temp. The ui would be used to > train the enrichment configurations > etc > etc. > > Until it was all set and you were happy then the source was ‘activated’ > with the real topologies. > > > > On February 18, 2017 at 07:38:17, [email protected] ([email protected]) > wrote: > > I don't expect people to push data through Metron until they have an > expectation that it will be properly parsed and handled, otherwise you'll > flood the cluster with failures and noise. > > Here's what's in my head. A user grabs an upstream log sample (from their > environment), uses the UI to figure out how to parse it, adjusts Metron > accordingly (make Kafka topic, apply stellar functions, tweak enrich layer, > triage, optionally index templates) then heads back out of Metron and gives > it some juice by either just turning on the Kafka producer or doing a spot > check first (examine in UI things are working properly) then turning on the > producer (nifi, etc.). > > Jon > > On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball < > [email protected]> wrote: > > > At the moment the UI relies on Kafka topics already existing, and having > > data flowing into them to pull samples for the parser config panel. > > However, you can paste alternative samples if kafka isn’t already setup. > As > > such, the assumption is that the topics would already have data flowing > > from something like nifi, or some other producer before you got round to > > configuring a sensor in this UI. Would that work, or would you expect to > be > > creating the parser config before the data was flowing into kafka? In the > > later case, we could probably rely on copy-pasted samples to design the > > parser, and then auto-create topics if not already existing on parser > save. > > > > Simon > > > > > > > On 17 Feb 2017, at 23:42, [email protected] <[email protected]> wrote: > > > > > > Well, I think that a UI that does what you show is a huge step forward > > and > > > I definitely wouldn't want to stop it from getting into the code > because > > of > > > any of the additional features that we are talking about here. > > > > > > That said, I really would like to see the Kafka topic creation as a > part > > of > > > the UI because without it, data can't move. Templates would be a second > > > priority that I see as important, but depending on how Otto's thread > goes > > > how that could would may change and I wouldn't push to have that in a > MVP > > > UI. > > > > > > Aside from that, I still agree with my previous comments regarding nice > > to > > > have features, but definitely not needed for a first run. > > > > > > Jon > > > > > > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian <[email protected] > > > > > wrote: > > > > > >> Hi Dima and Jon, > > >> > > >> Do you think the UI: > > >> > > >> http://imgur.com/a/QAQyu? > > >> > > >> > > >> ...Is a reasonable place to start as it is?… or do we need to have the > > >> Elastic Search Templates right away? > > >> Are the templates a MUST HAVE in order to add this UI to Metron? > > >> > > >> As Simon said, I think we can file a JIRA on the templates and start > > >> working on what that looks like. > > >> > > >> What are your thoughts? > > >> > > >> > > >> > > >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" < > > [email protected]> > > >> wrote: > > >> > > >>> Seems like the UI discussion has fallen off the inboxes. > > >>> > > >>> We should definitely open up a separate Jira and discussion on the > best > > >> way to process elastic search template management. This feels like a > > non-UI > > >> question to me, since it would apply equally to any other parser > config > > >> method. I’ll start a separate discuss for that. > > >>> > > >>> In the meantime it would be great to hear more from the community on > > the > > >> UI at http://imgur.com/a/QAQyu? > > >>> > > >>> Is there anything you see missing around sensor config here? > > >>> Does it look useful? > > >>> Does the flow match the way you would currently think about setting > up > > >> sensors? > > >>> > > >>> It would be great to get some thoughts. > > >>> > > >>> Simon > > >>> > > >>> > > >>>> On 1 Feb 2017, at 12:07, Simon Elliston Ball < > > >> [email protected]> wrote: > > >>>> > > >>>> The MPack creates elastic templates for the standard parsers > included > > >> in the pack. > > >>>> > > >>>> However, when creating new sensors we should be able to infer > Elastic > > >> templates from the schema. There are certain fields that we need to > mark > > >> types for to generate those templates (e.g. date / time fields). This > > goes > > >> back to the debate we had a while ago about, for example, typing in > the > > >> grok parser, but applies for others too. I suspect that since we > should > > be > > >> able to infer all the types needed for templates, this is more an > > excellent > > >> feature enhancement for the REST api layer, unless anyone can think of > a > > >> reason we would need explicit type definition in the > > >>>> > > >>>> > > >>>> > > >>>>> On 1 Feb 2017, at 19:54, Nick Allen <[email protected]> wrote: > > >>>>> > > >>>>> The Index templates are currently handled in the Ambari MPack > (unless > > >> you > > >>>>> are suggesting some additional functionality beyond what the Ambari > > >> MPack > > >>>>> gives us today.) > > >>>>> > > >>>>> I think there is always going to be a fuzzy, gray line between what > > the > > >>>>> Ambari MPack should do and what the Management UI should do. I > think > > >> it > > >>>>> works fine in Ambari today, but I could be convinced otherwise. > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov < > > [email protected] > > >>> > > >>>>> wrote: > > >>>>> > > >>>>>> Thank you Houshang, > > >>>>>> > > >>>>>> That looks absolutely great, loved it! > > >>>>>> Will it make sense to also add ElasticSearch index templates > > >>>>>> querying/creation here? > > >>>>>> > > >>>>>> - Dima > > >>>>>> > > >>>>>> On 02/01/2017 02:06 AM, Houshang Livian wrote: > > >>>>>>> Hello Metron Community, > > >>>>>>> > > >>>>>>> We have constructed a Management Module UI, built on top of > > >> METRON-503 > > >>>>>> (REST API) (currently under review). This Module gives users the > > >> ability to > > >>>>>> setup and administer much of the product through the UI. > > >>>>>>> > > >>>>>>> Here are some screens to show you what we are thinking. Please > > take a > > >>>>>> look: > > >>>>>>> http://imgur.com/a/QAQyu? > > >>>>>>> > > >>>>>>> Does this look like a reasonable place to start? > > >>>>>>> Is there anything that is an absolute MUST have or MUST NOT have? > > >>>>>>> > > >>>>>>> Houshang Livian > > >>>>>>> Senior User Experience Designer > > >>>>>>> Hortonworks > > >>>>>>> > > >>>>>>> www.hortonworks.com<http://www.hortonworks.com/> > > >>>>>>> ________________________________ > > >>>>>>> > > >>>>>>> Mobile: (831) 521-4176<tel:(831)%20521-4176> > > >>>>>>> [email protected]<mailto:[email protected]> > > >>>>>>> ________________________________ > > >>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Nick Allen <[email protected]> > > >>>> > > >>> > > >> > > > -- > > > > > > Jon > > > > > > Sent from my mobile device > > > > -- > > Jon > > Sent from my mobile device >
