Hi again,
I might have been a little optimistic, because my prod crashed again.
This time I used python code which has been adapted from the website from
Livestatus conceptor, so I guess that this time the queries are correctly
done
This error is really strange, because it seems to be realted to the number
of request I do in a short period of time. I managed to reproduce the error
on my dev environment *relief*. Here, everything works fine until the
broker crashes.
I might be totally wrong here as I don't know much of Livestatus coding in
Shinken, but is it possible that the maximum number of socket has been
reached or something like that?
Attached, you can see the scenario I used to reproduce the error :
I first added comments on running hosts (in order to have a certain amount
of hosts with comments on them)
Then I launched my script
Livestatus answered correctly for some hosts, then crashed, like on prod env
If this is indeed a problem with max number of sockets in livestatus
module, I could recode my script in order to avoid making too many queries
(getting all hosts, then getting all comments, only 2 queries here), but
I'd like to be sure it's really that in order to avoid further issues on
prod environnement
Regards
Denis GERMAIN
2011/10/27 Olivier Hanesse <olivier.hane...@gmail.com>
> hi,
>
> I used to work with netcat too, and I got some issues in the past with
> some netcat version.
> On debian for example, you have 2 versions :
>
> netcat-openbsd - TCP/IP swiss army knife
> netcat-traditional - TCP/IP swiss army knife
>
> One was working, but not the other one.
> Maybe you could give a try and install the other version.
>
> Regards
>
> Olivier
>
> 2011/10/27 Denis GERMAIN <dt.germ...@gmail.com>
>
>> Hi,
>>
>> Thanks for the answer, I had forgotten that we used netcat instead of
>> unixcat. I have still to confirm this, but i'm think we have a problem
>> because we're not correctly closing the connection with netcat.
>>
>> I'm going to code it in python in order to do this in a proper way.
>>
>> I'll keep you posted if this is not the issue but I'm pretty sure that
>> this is our problem.
>>
>> Thanks!
>>
>> Denis
>>
>> 2011/10/26 Gerhard Lausser <gerhard.laus...@consol.de>
>>
>>> Hi,****
>>>
>>> ** **
>>>
>>> which version of Shinken do you use?****
>>>
>>> I have here a 0.8 and can’t reproduce your error. There’s one host
>>> (omd-disco) which I made DOWN by submitting a passive command and which has
>>> two comments attached.****
>>>
>>> ** **
>>>
>>> OMD[disco]:~$ printf "GET hosts\nFilter: state = 1\nColumns: name
>>> comments\n"|unixcat /omd/sites/disco/tmp/run/live****
>>>
>>> omd-disco;3,4****
>>>
>>> ** **
>>>
>>> OMD[disco]:~$ printf 'GET comments\nColumns: comment_id comment\n' |
>>> unixcat /omd/sites/disco/tmp/run/live ****
>>>
>>> 3;i bi hi****
>>>
>>> 4;i kos ned oft gnou sogn, i bi hi****
>>>
>>> ** **
>>>
>>> OMD[disco]:~$ printf 'GET comments\nColumns: comment_id comment\nFilter:
>>> comment_id = 3\n' | unixcat /omd/sites/disco/tmp/run/live ****
>>>
>>> 3;i bi hi****
>>>
>>> ** **
>>>
>>> Gerhard****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> *Von:* Denis GERMAIN [mailto:dt.germ...@gmail.com]
>>> *Gesendet:* Mittwoch, 26. Oktober 2011 16:40
>>> *An:* shinken-devel@lists.sourceforge.net
>>> *Betreff:* [Shinken-devel] Slight problem with livestatus module with
>>> direct queries****
>>>
>>> ** **
>>>
>>> Hello list,
>>>
>>> For historic reporting purpose, we have a developper in our team that
>>> directly execute queries to the livestatus module via commands like
>>> printf 'GET hosts\nFilter: state = 1' | nc 10.197.64.97 50000 >
>>> /data/ftptrace/nagios/thruk.txt
>>> printf 'GET comments\nFilter comment_id = '$commid | nc 10.197.64.97
>>> 50000 | tail -1 | awk -F";" '{print $2}'*
>>>
>>> *This gives us informations about hosts down which are then reformatted
>>> in a dusty old report sent to our client.
>>> At the beginning we used only the first request for our report, as
>>> comments were directly available from the host information given by the
>>> query. After an update, we didn't realised that the comments were not
>>> directly in the query result, but only the comment ID (which I grant you is
>>> much more "clean")
>>>
>>> 1) Problem is that now, we need to query for the comments (selecting
>>> good comment ID) and that it constantly break livestatus module (which
>>> doesn't always manage to relaunch itself)
>>>
>>> 2) We also noted that when we use the Column statement (to select only
>>> the columns we need), the livestatus module instantly crash
>>>
>>> I realise that our problem is a little exotic, as the vast majority of
>>> livestatus users only use it through Thruk, but if something could be done
>>> (or as already been done), we would be grateful (our developper mostly :-p)
>>>
>>> Denis GERMAIN
>>>
>>> PS: An example traceback. I don't have much more yet, we are running
>>> Shinken on production (I already see admins frowning eyes, but it's stable
>>> enough if you don't try strange things like us) and my manager don't really
>>> like us crashing it on purpose. We'll try to reproduce on preproduction
>>> ASAP, but if this can help pinpointing the issue...
>>>
>>> [1319536549] [broker-1] Traceback (most recent call last):
>>> File
>>> "/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_broker.py",
>>> line 828, in main
>>> self.do_main()
>>> File
>>> "/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_broker.py",
>>> line 1052, in do_main
>>> response, keepalive =
>>> self.livestatus.handle_request(open_connections[socketid]['buffer'])
>>> File
>>> "/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus.py",
>>> line 129, in handle_request
>>> request.parse_input(data)
>>> File
>>> "/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_request.py",
>>> line 96, in parse_input
>>> query.parse_input('\n'.join(query_cmds))
>>> File
>>> "/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_query.py",
>>> line 259, in parse_input
>>> self.filter_stack.put(self.make_filter(operator, attribute,
>>> reference))
>>> File
>>> "/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_query.py",
>>> line 786, in make_filter
>>> converter = self.find_converter(attribute)
>>> File
>>> "/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_query.py",
>>> line 102, in find_converter
>>> out_map = LSout_map[self.out_map_name]
>>> KeyError: 'hosts'****
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> The demand for IT networking professionals continues to grow, and the
>>> demand for specialized networking skills is growing even more rapidly.
>>> Take a complimentary Learning@Cisco Self-Assessment and learn
>>> about Cisco certifications, training, and career opportunities.
>>> http://p.sf.net/sfu/cisco-dev2dev
>>> _______________________________________________
>>> Shinken-devel mailing list
>>> Shinken-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/shinken-devel
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> The demand for IT networking professionals continues to grow, and the
>> demand for specialized networking skills is growing even more rapidly.
>> Take a complimentary Learning@Cisco Self-Assessment and learn
>> about Cisco certifications, training, and career opportunities.
>> http://p.sf.net/sfu/cisco-dev2dev
>> _______________________________________________
>> Shinken-devel mailing list
>> Shinken-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/shinken-devel
>>
>>
>
>
> ------------------------------------------------------------------------------
> The demand for IT networking professionals continues to grow, and the
> demand for specialized networking skills is growing even more rapidly.
> Take a complimentary Learning@Cisco Self-Assessment and learn
> about Cisco certifications, training, and career opportunities.
> http://p.sf.net/sfu/cisco-dev2dev
> _______________________________________________
> Shinken-devel mailing list
> Shinken-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/shinken-devel
>
>
#!/usr/bin/python
import socket
socket_ip = "127.0.0.1"
socket_port = 50000
def send_query(query):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((socket_ip,socket_port))
s.send(query)
s.shutdown(socket.SHUT_WR)
answer = s.recv(100000000)
return [ line.split(';') for line in answer.split('\n')[:-1] ]
def main_function():
host_table = send_query("GET hosts\nFilter: state = 0\nColumns: name
comments\n")
for host in host_table:
host_name=host[0]
host_comments_id=host[1].split(',') #extract a table of the
comments
host_last_comment_id=host_comments_id[-1] #use only last
comment
comment = send_query("GET comments\nColumns: comment_id
comment\nFilter: comment_id = "+host_last_comment_id)
print comment
#print host_name+';'+comment[1]
main_function()
[['1', 'Test plantage']]
[['2', 'Test plantage']]
[['3', 'Test plantage']]
[['4', 'Test plantage']]
[['5', 'Test plantage']]
[['6', 'Test plantage']]
[['7', 'Test plantage']]
[['8', 'Test plantage']]
[['9', 'Test plantage']]
[['10', 'Test plantage']]
[['11', 'Test plantage']]
[['12', 'Test plantage']]
[['13', 'Test plantage']]
[['14', 'Test plantage']]
[['17', 'Test plantage']]
[['15', 'Test plantage']]
[['18', 'Test plantage']]
[['16', 'Test plantage']]
[]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in main_function
File "<stdin>", line 3, in send_query
File "<string>", line 1, in connect
socket.error: [Errno 111] Connection refused
vi brokerd.log
File
"/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_broker.py",
line 828, in main
self.do_main()
File
"/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_broker.py",
line 1052, in do_main
response, keepalive =
self.livestatus.handle_request(open_connections[socketid]['buffer'])
File
"/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus.py",
line 129, in handle_request
request.parse_input(data)
File
"/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_request.py",
line 96, in parse_input
query.parse_input('\n'.join(query_cmds))
File
"/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_query.py",
line 259, in parse_input
self.filter_stack.put(self.make_filter(operator, attribute, reference))
File
"/usr/lib/python2.6/site-packages/shinken/modules/livestatus_broker/livestatus_query.py",
line 788, in make_filter
reference = converter(reference)
ValueError: invalid literal for int() with base 10: ''
------------------------------------------------------------------------------
RSA® Conference 2012
Save $700 by Nov 18
Register now!
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Shinken-devel mailing list
Shinken-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shinken-devel
