Can indexing slowlog level be modified in logging.yml?

[Tim Hopper]

I am trying to tune how indexing slowlogs are collected. I know that I can set 
a threshold 

 
for how long an indexing op on a shard must take before it is logged. I 
would also like to be choose what level I'm logging at. It would appear 
that modifying this line 

 in 
logging.yml would allow me to do this. However, unless I am 
misunderstanding the purpose of that setting, it is being ignored by 
ElasticSearch.

Here is how I came to that conclusion. I have ElasticSearch 1.3.2 running 
locally on my Mac. I these config files 
 in ~/es-config. 
This elasticsearch.yml file is bare-bones, as you can see. The logging.yml 
file only modifies this line 

 of 
the default file. My assumption is that this should mean *only* WARN level 
indexing ops are logged. 

However, if I start an ES instance using 

elasticsearch --config="/Users/tdhopper/es-config/elasticsearch.yml"

and then run this Python script

import elasticsearch, time
es = elasticsearch.Elasticsearch(hosts="localhost")

while True:
print ".",
es.index(index="index1", doc_type="test_doc", body = {"hot_body": 1})
time.sleep(.5)

my ~/es-logs/elasticsearch_index_indexing_slowlog.log file is immediately 
filled up with lines like 

[2014-08-22 10:24:52,162][INFO ][index.indexing.slowlog.index] [War 
Machine] [index1][2] took[1.4ms], took_millis[1], type[test_doc], 
id[WH83A0yvRHaQtQ34_6wncg], routing[], source[{"hot_body":1}]
[2014-08-22 10:24:52,666][INFO ][index.indexing.slowlog.index] [War 
Machine] [index1][1] took[1.5ms], took_millis[1], type[test_doc], 
id[sErCAr3BR_qWVfp0pnayGw], routing[], source[{"hot_body":1}]

That is, INFO level log statements.

Am I misunderstanding the purpose of this setting or is ES ignoring my 
request?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/855ae52d-8912-4b90-adda-faf5e7158128%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slowlog not populating after deletion

[Tim Hopper]

When the logs weren't populating initially, I created new log files with 
`touch`. I deleted all those files and restarted my cluster, and ES created 
new log files and they seem to be populating again. 

On Friday, August 15, 2014 10:03:53 AM UTC-4, Tim Hopper wrote:
>
> We added another node to our cluster yesterday, and its slowlog 
> immediately started populating. Any idea how to get these populating again 
> after they've been deleted?
>
> On Wednesday, August 13, 2014 3:45:45 PM UTC-4, Tim Hopper wrote:
>>
>> This morning, I enabled slowlogs on a bunch of indices in my cluster by 
>> issuing something like this 
>> <https://gist.github.com/tdhopper/a44cc4200b9c09aea389>. Because I set 
>> the threshold at 0 and we're doing lots of reads and writes, I got large 
>> logs rather quickly. I decided to raise the thresholds and delete the logs 
>> manually from all four machines on my cluster; I did this without 
>> restarting the cluster. After doing this, no new *slowlog.log files were 
>> created. I tried 'touch'ing the appropriate paths, but those files were not 
>> populated. 
>>
>> I have since set the slowlog thresholds manually in my ElasticSearch.yml 
>> file
>>
>> index.search.slowlog.threshold.query.warn: 10s
>> index.search.slowlog.threshold.query.info: 5s
>> index.search.slowlog.threshold.query.debug: 2s
>> index.search.slowlog.threshold.query.trace: 0s
>> index.search.slowlog.threshold.fetch.warn: 10s
>> index.search.slowlog.threshold.fetch.info: 5s
>> index.search.slowlog.threshold.fetch.debug: 2s
>> index.search.slowlog.threshold.fetch.trace: 0s
>> index.indexing.slowlog.threshold.index.warn: 10s
>> index.indexing.slowlog.threshold.index.info: 5s
>> index.indexing.slowlog.threshold.index.debug: 2s
>> index.indexing.slowlog.threshold.index.trace: 0s
>>
>>
>> and restarted the cluster. The log files still were not populated. 
>>
>> I again issued a curl command to set the thresholds to 0ms, and I also 
>> set 'additivity.index.search.slowlog' and 
>> 'additivity.index.indexing.slowlog' to 'true' for the cluster (though I'm 
>> not entirely clear on what those do). Still no logs. I have also tried 
>> fiddling with the log level with a command like this 
>> <https://gist.github.com/tdhopper/2dc0d2aa039f3dd598ab>.
>>
>> I've uncovered two 
>> <http://stackoverflow.com/questions/23195280/elasticsearch-slow-log-wont-write-to-log-file>
>>  
>> SO questions 
>> <http://stackoverflow.com/questions/23899327/elasticsearch-wont-log-slow-queries-anymore>
>>  
>> that seem to have similar problems that have not been resolved.
>>
>> Have I done something wrong that is preventing these logs from appearing? 
>>
>> I am using 1.3.0.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b1f39c58-0ba4-46da-906a-648e7a4b57f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slowlog not populating after deletion

[Tim Hopper]

We added another node to our cluster yesterday, and its slowlog immediately 
started populating. Any idea how to get these populating again after 
they've been deleted?

On Wednesday, August 13, 2014 3:45:45 PM UTC-4, Tim Hopper wrote:
>
> This morning, I enabled slowlogs on a bunch of indices in my cluster by 
> issuing something like this 
> <https://gist.github.com/tdhopper/a44cc4200b9c09aea389>. Because I set 
> the threshold at 0 and we're doing lots of reads and writes, I got large 
> logs rather quickly. I decided to raise the thresholds and delete the logs 
> manually from all four machines on my cluster; I did this without 
> restarting the cluster. After doing this, no new *slowlog.log files were 
> created. I tried 'touch'ing the appropriate paths, but those files were not 
> populated. 
>
> I have since set the slowlog thresholds manually in my ElasticSearch.yml 
> file
>
> index.search.slowlog.threshold.query.warn: 10s
> index.search.slowlog.threshold.query.info: 5s
> index.search.slowlog.threshold.query.debug: 2s
> index.search.slowlog.threshold.query.trace: 0s
> index.search.slowlog.threshold.fetch.warn: 10s
> index.search.slowlog.threshold.fetch.info: 5s
> index.search.slowlog.threshold.fetch.debug: 2s
> index.search.slowlog.threshold.fetch.trace: 0s
> index.indexing.slowlog.threshold.index.warn: 10s
> index.indexing.slowlog.threshold.index.info: 5s
> index.indexing.slowlog.threshold.index.debug: 2s
> index.indexing.slowlog.threshold.index.trace: 0s
>
>
> and restarted the cluster. The log files still were not populated. 
>
> I again issued a curl command to set the thresholds to 0ms, and I also set 
> 'additivity.index.search.slowlog' and 'additivity.index.indexing.slowlog' 
> to 'true' for the cluster (though I'm not entirely clear on what those do). 
> Still no logs. I have also tried fiddling with the log level with a command 
> like this <https://gist.github.com/tdhopper/2dc0d2aa039f3dd598ab>.
>
> I've uncovered two 
> <http://stackoverflow.com/questions/23195280/elasticsearch-slow-log-wont-write-to-log-file>
>  
> SO questions 
> <http://stackoverflow.com/questions/23899327/elasticsearch-wont-log-slow-queries-anymore>
>  
> that seem to have similar problems that have not been resolved.
>
> Have I done something wrong that is preventing these logs from appearing? 
>
> I am using 1.3.0.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1b7bdfa4-9e59-4f51-9b36-0b95ebd0a091%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

slowlog not populating after deletion

[Tim Hopper]

This morning, I enabled slowlogs on a bunch of indices in my cluster by 
issuing something like this 
. Because I set the 
threshold at 0 and we're doing lots of reads and writes, I got large logs 
rather quickly. I decided to raise the thresholds and delete the logs 
manually from all four machines on my cluster; I did this without 
restarting the cluster. After doing this, no new *slowlog.log files were 
created. I tried 'touch'ing the appropriate paths, but those files were not 
populated. 

I have since set the slowlog thresholds manually in my ElasticSearch.yml 
file

index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 0s
index.search.slowlog.threshold.fetch.warn: 10s
index.search.slowlog.threshold.fetch.info: 5s
index.search.slowlog.threshold.fetch.debug: 2s
index.search.slowlog.threshold.fetch.trace: 0s
index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 0s


and restarted the cluster. The log files still were not populated. 

I again issued a curl command to set the thresholds to 0ms, and I also set 
'additivity.index.search.slowlog' and 'additivity.index.indexing.slowlog' 
to 'true' for the cluster (though I'm not entirely clear on what those do). 
Still no logs. I have also tried fiddling with the log level with a command 
like this .

I've uncovered two 

 
SO questions 

 
that seem to have similar problems that have not been resolved.

Have I done something wrong that is preventing these logs from appearing? 

I am using 1.3.0.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/54d753c1-e9a5-4116-ac5c-0c8edd3a392b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.