OK, so I'm back to the original issue.
As soon as there are multiple nodes in the configuration, I start to see
the "Did not find meta info of this node. Re-Registering." in the logs. I
see this for all 3 nodes. I've checked the replica configuration in Mongo
(looks good), dropped down to a single Mongo node (no change).. the mongo
table has the hosts in there:
graylog:PRIMARY> db.nodes.find()
{ "_id" : ObjectId("573b620b89479f0ccf046c34"), "is_master" : false,
"hostname" : "gray02.somewhere.com", "last_seen" : 1463512536,
"transport_address" : "http://10.201.137.210:12900/";, "type" : "SERVER",
"node_id" : "8536ee95-b9c7-4553-9022-d997da315755" }
{ "_id" : ObjectId("573b6dbf05ee161654fa2122"), "is_master" : true,
"hostname" : "gray01.somewhere.com", "last_seen" : 1463512536,
"transport_address" : "http://10.201.137.209:12900/";, "type" : "SERVER",
"node_id" : "00ac0ad1-b96f-46c0-a2bc-bc9e7a90777f" }
{ "_id" : ObjectId("573b6dbfb2d64909218a56b8"), "is_master" : false,
"hostname" : "gray00.somewhere.com", "last_seen" : 1463512537,
"transport_address" : "http://10.201.137.208:12900/";, "type" : "SERVER",
"node_id" : "3116ac6b-604f-4436-955c-1458cb489415" }
last_seen is getting updated.. times are in-sync on all 3 nodes (as well as
the F5 and any web clients).. REST calls are configured to be sticky, as
are the webUI calls.. what am I missing?
On Tuesday, May 17, 2016 at 11:04:22 AM UTC-7, Jeff McCombs wrote:
>
> You can't tell, but I'm blushing right now.
>
> Thanks Jochen. :)
>
>
> On Tuesday, May 17, 2016 at 10:13:36 AM UTC-7, Jochen Schalanda wrote:
>>
>> Hi Jeff,
>>
>> you're probably looking for the web_endpoint_uri configuration setting
>> (see
>> http://docs.graylog.org/en/2.0/pages/configuring_webif.html#configuration-options).
>>
>> The rest_listen_uri (or rest_transport_uri) should always be an address
>> which the Graylog nodes in a given cluster can access.
>>
>> Cheers,
>> Jochen
>>
>> On Tuesday, 17 May 2016 17:49:54 UTC+2, Jeff McCombs wrote:
>>>
>>> Hi Jochen,
>>>
>>> Yes, that's actually intentional. Though it could just be that I'm
>>> misunderstanding the option..
>>>
>>> Consider the scenario below:
>>>
>>> +-----------------+
>>> | |
>>> | User |
>>> | (192.168.1.200)|
>>> +-----------------+
>>> |
>>> |
>>> +------------------------------------------------------------------+
>>> |
>>> |
>>> | graylog.somewhere.com
>>> | (192.168.1.100)
>>> +--------------+
>>> | |
>>> | F5 |
>>> | |
>>> | |
>>> +--------------+
>>> |
>>> |
>>> |
>>>
>>>
>>> +------------------------------------------------------------------------------------+
>>> | | |
>>> +-------------+ +-------------+ +-------------+
>>> | | | | | |
>>> | | | | | |
>>> | gray00 | | gray01 | | gray02 |
>>> | 10.201.5.1 | | 10.201.5.2 | | 10.201.5.3 |
>>> +-------------+ +-------------+ +-------------+
>>>
>>> (hope that shows up OK, if not, convert it to fixed width font)
>>>
>>> When the javscript running in the browser for a WebUI call needs to
>>> reach the individual nodes via the REST interface, the only way for that
>>> call to happen is to go through the F5 and be load balanced.. right? But if
>>> the nodes need to communicate with one another via the REST interface as
>>> well..then yeah I could see why the nodes would be complaining. They try
>>> and reach the 192 address, the request gets balanced, and winds up on the
>>> wrong node...
>>>
>>> So am I just misunderstanding the REST transport URI option? WITHOUT
>>> setting that configuration to the same address, the WebUI doesn't function
>>> properly because there's no direct communication between the individual
>>> graylog nodes, and the end user.
>>>
>>> Is there a WebUI/REST URI option somewhere I just don't know about?
>>>
>>>
>>> On Tue, May 17, 2016 at 5:07 AM, Jochen Schalanda wrote:
>>>
>>>> Hi Jeff,
>>>>
>>>> you're using the same transport address for the Graylog REST API on all
>>>> 3 Graylog nodes. Is this intentional? I'm asking because that won't work
>>>> in
>>>> the long run as Graylog nodes need to be able to communicate with each
>>>> other via the Graylog REST API and the announced transport address.
>>>>
>>>> Cheers,
>>>> Jochen
>>>>
>>>>
>>>> On Friday, 13 May 2016 22:53:07 UTC+2, Jeff McCombs wrote:
>>>>>
>>>>> So here's a question.. looking at the node output from tokred vs mine..
>>>>>
>>>>> When you have a cluster of Graylog servers behind a load balancer.. do
>>>>> you configure the API transport address to the cluster IP, or the
>>>>> individual nodes? Could this be the cause of the following errors I'm
>>>>> also
>>>>> seeing?
>>>>>
>>>>> 2016-05-13T13:43:27.749-07:00 WARN [ProxiedResource] Node
>>>>> <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call
>>>>> org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it.
>>>>> 2016-05-13T13:46:58.766-07:00 WARN [ProxiedResource] Node
>>>>> <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call
>>>>> org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it.
>>>>> 2016-05-13T13:49:14.735-07:00 WARN [ProxiedResource] Node
>>>>> <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call
>>>>> org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it.
>>>>>
>>>>> On Friday, May 13, 2016 at 1:50:17 PM UTC-7, Jeff McCombs wrote:
>>>>>>
>>>>>> Hi Jochen,
>>>>>>
>>>>>> I see the records for the nodes:
>>>>>>
>>>>>> graylog:PRIMARY> db.nodes.find()
>>>>>> { "_id" : ObjectId("57363bab05ee16689e192953"), "is_master" : false,
>>>>>> "hostname" : "gray01somewhere.com", "last_seen" : 1463172221,
>>>>>> "transport_address" : "http://graylog.somewhere.com:12900/";, "type"
>>>>>> : "SERVER", "node_id" : "00ac0ad1-b96f-46c0-a2bc-bc9e7a90777f" }
>>>>>> { "_id" : ObjectId("57363c0889479f6906e17de9"), "is_master" : false,
>>>>>> "hostname" : "gray02.somewhere.com", "last_seen" : 1463172221,
>>>>>> "transport_address" : "http://graylog.somewhere.com:12900/";, "type"
>>>>>> : "SERVER", "node_id" : "8536ee95-b9c7-4553-9022-d997da315755" }
>>>>>> { "_id" : ObjectId("57363c79b2d6491223d87222"), "is_master" : true,
>>>>>> "hostname" : "gray00.somewhere.com", "last_seen" : 1463172220,
>>>>>> "transport_address" : "http://graylog.somewhere.com:12900/";, "type"
>>>>>> : "SERVER", "node_id" : "3116ac6b-604f-4436-955c-1458cb489415" }
>>>>>>
>>>>>> Interestingly, when I shut down all but the master, it continues to
>>>>>> spit errors:
>>>>>>
>>>>>> /var/log/graylog-server/server.log
>>>>>> 2016-05-13T13:47:06.662-07:00 WARN [NodePingThread] Did not find
>>>>>> meta info of this node. Re-registering.
>>>>>> 2016-05-13T13:47:32.639-07:00 WARN [NodePingThread] Did not find
>>>>>> meta info of this node. Re-registering.
>>>>>>
>>>>>> mongo nodes query:
>>>>>> graylog:PRIMARY> db.nodes.find()
>>>>>> { "_id" : ObjectId("57363d64b2d6491223d87339"), "is_master" : true,
>>>>>> "hostname" : "gray00.somewhere.com", "last_seen" : 1463172464,
>>>>>> "transport_address" : "http://graylog.somewhere.com:12900/";, "type"
>>>>>> : "SERVER", "node_id" : "3116ac6b-604f-4436-955c-1458cb489415" }
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> On Friday, May 13, 2016 at 2:16:17 AM UTC-7, Jochen Schalanda wrote:
>>>>>>>
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> please check the "nodes" collection in MongoDB and that it contains
>>>>>>> valid node descriptors while Graylog is running.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Jochen
>>>>>>>
>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/c0285405-cb5a-41cd-98ec-3cec1ca46b14%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
