On 17.03.2015 22:28, Achim Nierbeck wrote:
Regarding Topics and Indices, I don't think its a good idea to create
multiple indices, but the topic can be used differently.
Instead of sending karaf_events
I agree to not have separate indices. If elastic search can not correlate
between indices it is better to have them in one index.
Some people might still want it for some reason but I think the major case
is to put all into the same index. So I think we can delay providing
configs for several indexes until there is a first good use case.
Indices are usually time-based not content based (like the event topics)
So it just makes sense to maybe have either an index for each hour (if the
data is to much already)
Or an aggregated Index for a week.
I am not worried about the data size. More about the number of columns.
It is a rather academic concern though. Till now elastic search works
fine with
the many columns. One thing that might happen though is that data with
the same key but different meaning is stored. The types we discussed
about might help with that.
So you would use the Topic as the type. One thing to consider here is that
I currently translate the logger name into a topic name. This has the
advantage that it is quite easy in EventAdmin to filter for a logger name
including all sub names. If you create the type from this that would make
lots of different types. I am not sure what would happen in Elastic Search
with that.
In the standard ELK scenario for analysing log data, you have different
logfiles that produces those entries. Therefore those logfiles usually have
their own type. Like for example Access and Syslog.
I agree. The simplest solution would be to start with two types log
and jmx and set them in the appender. I can create it like that and we
can discuss if we should make the types more fine grained.
For logs I am not sure. Most log entries will be the same but as MDC comes
into play users can build their own kind of types. Not sure how to handle
that. Currently I simply put all into the same index in the end so the MDC
values are all there but there is not futher structure. Maybe this is
completely ok for how elastic search works.
yes, afaik this is sufficient already.
If you only use the JMX Collector you'll see that EL is already capable to
map different mappings to that type already.
Yes I checked the documentation of elastic search. If you add new
properties into an index then it will automatically work with the new keys.
You can not delete keys though.
As we agree on the general idea of using EventAdmin I will merge the
branch back into master. We can then refine how it works exactly in the
normal process.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
