On 08/02/2018 09:44 AM, eSX wrote:
Hi Rich,thanks for your reply.
> Is it acceptable to require a different action for retries, if retries
> are being used? Or should the goal be that there should be retry
> support, with _no_ changes required to any existing templates, and
_one_
> action for both the initial request and retries?
What I thought of at the beginning, is to keep the fields which can be
configured in _bulk request.
For example, a _bulk request will include:
{"index":{"_index":
"log_notice_20180802","_type":"events","pipeline":"pl_data_parser"}}\n
{_MESSAGE_}
There are 4 fields can be configured in omelasticsearch:
writeoperation("index"), index("_index"), searchType("_type") and
pipelineName("pipeline").
There are 5 dynamic fields which can be in the metadata header: _index,
pipeline, _parent, _type, and _id. Is it sufficient for your purpose if
we make all of these available in $.omes for the retry?
You would still need to have two actions - one for the initial attempt
and one for retries, and two different templates e.g. for the pipeline:
template (
name = "es_idx_tpl"
type = "string"
string = "%syslogseverity-text%"
)
template (
name = "es_idx_tpl_retry"
type = "string"
string = "%$.omes!pipeline%"
)
if strlen($.omes!pipeline) > 0 then {
action(type="omelasticsearch" ... dynpipelinename="es_idx_tpl_retry"
...)
} else {
action(type="omelasticsearch" ... dynpipelinename="es_idx_tpl" ...)
}
Is this ok?
In omes when retrying:
Only the "writeoperation" was parsed from response item and added to omes.
omelasticsearch skiped the request header(the 1st line) and save the
request body as smsg_t.
_index and _type copied from response of Elasticsearch.
So the request parameters we had configured("pipeline"), were lost in
request header.
I think, to keep the paramenters which generated by initial request is
enough for now.
A more easy way to configure omelasticsearch is all in one action, but
it's hard to code and inflexible?
In any case, I think it's necessary to keep the variables of initial
request.
Rich Megginson via rsyslog <rsyslog@lists.adiscon.com
<mailto:rsyslog@lists.adiscon.com>> 于2018年8月2日周四 下午10:30写道：
On 08/02/2018 01:18 AM, eSX via rsyslog wrote:
> Hello rsyslog
> omes is the retry information when omelasticsearch is in failure
and retry.
> The contents of omes are response message from Elasticsearch
mainly.
> But the response didn't give a full information for retring.
What information is missing from the response?
https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html
$.omes should have everything in the bulk response.
If you mean "missing local variables" then see below.
> And when retry is happened, the message(smsg_t) is reset.
> It means any local variables is invalided, so that people can't
get extra
> information from it.
Correct. I'm not sure how to handle this. Please file an issue at
https://github.com/rsyslog/rsyslog/issues
omelasticsearch doesn't keep the smsg_t - it just constructs the bulk
index request string JSON, then sends the request once the length has
reached the threshold.
When I wrote the retry logic, I was expecting everything that was
needed
to be in the bulk retry would be in $!, or at least in whatever was
returned by "template", since that is the only thing kept and
available
when omelasticsearch processes the bulk index response for retry.
That
allow using the same action for both the initial request and retries,
but that would involve adding extra fields to the outgoing
message, and
would cause users to rewrite their templates.
If retry is active (if retryfailures="on"), omelasticsearch could
somehow "cache" the evaluation of any templates, and make them
available
in the retry record, but that would definitely mean having different
actions for the "initial" request and the retries.
I suppose the only way to ensure that you could use the exact same
action for retry as for the initial request would be to cache all
of the
smsg_t, or at least all of the variables - built in and local.
Is it acceptable to require a different action for retries, if
retries
are being used? Or should the goal be that there should be retry
support, with _no_ changes required to any existing templates, and
_one_
action for both the initial request and retries?
>
> For example, If I set pipelineName and set dynPipelineName "on" in
> omelasticsearch.
> Which the pipelineName was generated by template, like:
>
> template (
> name = "es_idx_tpl"
> type = "string"
> string = "%syslogseverity-text%"
> )
>
> When a "status 429" failure is happened, and retry.
> Because message variables is reset, there isn't $syslogseverity-text
> anymore.
> I can't regenerate a valid pipelineName, so retry would be failed or
> mistaken.
>
> The searchIndex option with dynSearchIndex="on" in
omelasticsearch has same
> problem.
> Fortunately, I can retrieve it again from $.omes!_index.
>
> A workaround for above is:
> I have to make an retryFailure ruleset for each pipelineName.
> Like:
>
> ruleset (
> name = "es_retry_for_pipeline1"
> ) {
> if strlen($.omes!status) > 0 then {
> ...
> }
> action (
> name = "es_retry_with_pipeline1"
> type = "omelasticsearch"
> ....
> pipelineName = "pipeline1"
> )
> }
>
> action (
> name = "es_for_pipeline"
> type = "omelasticsearch"
> ...
> pipelineName = "pipeline1"
> retryFailures="on"
> retryRuleset="es_retry_for_pipeline1"
> )
>
> ruleset (
> name = "es_retry_for_pipeline2"
> ) {
> if strlen($.omes!status) > 0 then {
> ...
> }
> action (
> name = "es_retry_with_pipeline2"
> type = "omelasticsearch"
> ....
> pipelineName = "pipeline2"
> )
> }
>
> action (
> name = "es_for_pipeline2"
> type = "omelasticsearch"
> ...
> pipelineName = "pipeline2"
> retryFailures="on"
> retryRuleset="es_retry_for_pipeline2"
> )
> ......
> ruleset (
> name = "es_retry_for_pipelineN"
> ) {
> if strlen($.omes!status) > 0 then {
> ...
> }
> action (
> name = "es_retry_with_pipelineN"
> type = "omelasticsearch"
> ....
> pipelineName = "pipelineN"
> )
> }
>
> action (
> name = "es_for_pipelineN"
> type = "omelasticsearch"
> ...
> pipelineName = "pipelineN"
> retryFailures="on"
> retryRuleset="es_retry_for_pipelineN"
> )
>
>
> It's so...ugly, isn't it?
>
> So, like pipelineName or any others, it should in omes also?
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
--
eSX
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
