Re: [DISCUSS] Management of Elastic and other index schemas

There's another benefit of this abstraction which could comprehensively get rid of issues we've had in the past with Lucene specific limitations. For instance, we could force that any string has the right attributes to handle long untokenized strings across the board, taking that out of the

Re: [DISCUSS] Management of Elastic and other index schemas

Based on my experience, I think we will come to point of needing to work on a configuration, test approve and deploy it for this system to be usable On February 20, 2017 at 14:55:39, Nick Allen (n...@nickallen.org) wrote: > I'm just blowing smoke at 10,000 feet here. :) I think we could engineer

Re: [DISCUSS] Management of Elastic and other index schemas

We can also pretty cleanly hook in a test phase with this. Maybe even have testing topologies On February 20, 2017 at 14:54:09, Otto Fowler (ottobackwa...@gmail.com) wrote: > So such a system would -> > Pre-Commit -> stops the topology in such a way that it drains but does not > take in new

Re: [DISCUSS] Management of Elastic and other index schemas

I'm just blowing smoke at 10,000 feet here. :) I think we could engineer it to be performant in some manner. Per your thoughts, It would make sense to have this sort of thing hooked to configuration changes, not a check for every message that comes in. On Mon, Feb 20, 2017 at 2:51 PM, Otto

Re: [DISCUSS] Management of Elastic and other index schemas

So such a system would -> Pre-Commit -> stops the topology in such a way that it drains but does not take in new things Commit -> updates the configurations, which triggers index deployment Start -> start up with the new configuration On February 20, 2017 at 14:51:43, Otto Fowler

Re: [DISCUSS] Management of Elastic and other index schemas

I think that would be interesting to do, validateFields() updateIndexIf() insert() But do you want to take that hit every message? I’m not sure. What if we instead hooked to configuration such that when you ‘commit’ a configuration change it recalculates and fixes up the index instead? So

Re: [DISCUSS] Management of Elastic and other index schemas

I agree with you Otto. Metron is an amalgamation of a lot of different technologies. We need to try and make using all of these pieces together as seamless as possible for the user. We do need some way to deal with ES templates for new parsers, sooner rather than later, even if its half-baked.

Re: [DISCUSS] Management of Elastic and other index schemas

Since enrichments, and even parsers, can be added on-the-fly, should the ES indexer be intelligent enough to manage the index templates on-the-fly also? Ideally, I should never have to manually install something like an ES template. The indexer should just take care of all that. In the case of

Re: [DISCUSS] Management of Elastic and other index schemas

Taking this a step further, I think this challenge goes beyond just parsers. We would also need to solve this problem for enrichments. When I add an enrichment, I want the enriched data to be indexed accurately. How can we make that happen? - As part of defining an enrichment, should I

Re: [DISCUSS] Management of Elastic and other index schemas

But in the mean time, parser developers do have to do the ES or Solr templates, and do have to figure out how to deploy them without being in the main install. I think this should be the end game, but in the meantime we should have *something* On February 18, 2017 at 12:27:09, James Sirota

Re: [DISCUSS] Management of Elastic and other index schemas

I am not sure I agree with packaging source-specific templates with the parser. I think that would make it harder to add additional storage sources. For example, what happens if I have 50 parsers with Solr and ES schemas defined, but now I want to add druid? Now I have to add 50 schemas to

Re: [DISCUSS] Management of Elastic and other index schemas

I think this is a good direction to move things toward - moving indexing templates to be packaged with parsers (using multiple tiered options) that are then merged with the possible enrich fields before getting added to the indexing technology in use. Now, to read the proposal thread... Jon On

Re: [DISCUSS] Management of Elastic and other index schemas

I’d broadly agree with that tiered approach. The version where the parser emits a generic schema, and enrichments contribute generic schema chunks to that which get combined into an indexer specific template generated at the end of the flow, so yes, pretty much inline with your proposal. (I

Re: [DISCUSS] Management of Elastic and other index schemas

We already make them do this now, or they get the defaults. So this is no different. Having parsers emit names and types etc, that would be another step - or it could be the ‘generic schema’ as implemented actually. A tiered approach - from * you give nothing with the parser - you get whatever

Re: [DISCUSS] Management of Elastic and other index schemas

I like that, to an extent… Forcing the provision of explicit schema might be a bit of a load for parser development. I’m assuming that custom parsers would be pushed towards the same packaging approach. Would it make sense to require the parser to emit field names and types expected, and then

Re: [DISCUSS] Management of Elastic and other index schemas

I think we can get there from my proposal. A source may package: * explicit schemas ( ES, SOLR, FOO ) * a generic to be invented schema for a to be invented pluggable indexing component :) and we’ll be able to handle it. On February 17, 2017 at 14:39:07, Kyle Richardson

Re: [DISCUSS] Management of Elastic and other index schemas

I just sent the discuss/propose email out Simon. I was thinking to start just packaging the equiv. of what we have now - es template and logrotate scripts would be the way to go. If we have solr, then those would go in there too. I don’t want to split the thread - but I have some ui workflow

Re: [DISCUSS] Management of Elastic and other index schemas

I personally like the idea of a typed schema per parser that we could translate to multiple targets. This would allow us a lot more modularity and extensibility in indexing down the road. -Kyle On Fri, Feb 17, 2017 at 1:59 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > That

Re: [DISCUSS] Management of Elastic and other index schemas

That sounds like a great idea Otto. Do you have any early design on that we can look at. Also, rather than just elastic templates do you think we should have some sort of typed schema we could translate to multiple targets (solr, elastic, ur... other...) or are you thinking of packaging

Re: [DISCUSS] Management of Elastic and other index schemas

Not to jump the gun, but I’m crafting a proposal about parsers and one of the things I am going to propose relates to having the ES Template for a given parser installed or packaged with the parser. We could load the template from there, edit, save and deploy etc. We can extend that concept more

Re: [DISCUSS] Management of Elastic and other index schemas

If we had a UI to switch types for existing fields in a template, would that work? What else is necessary? On 2/17/17, 10:22 AM, "Simon Elliston Ball" wrote: >A little while ago the issue of managing Elastic templates for new sensor >configs came up, and we