Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin - evosip.cloud: > I am able to see > 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding > DMQ node A host dmq-router-service=sip:42.100.109.113:5062 > > but in tcpdump no DNS query > > Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c > luster.local" and starting kamailio at the same time the result is always > the correct ip, no 42.100.109.113 coming up so this is very strange!
Hello, without looking too much into that, I have two remarks which may worth investigating: - for DNS names in the configuration file Kamailio does a lookup during startup and "compiles" this to structure which is evaluated during run time - there is also a DNS caching functionality, which could cause something like you observed. You can deactivate this with a core variable, I think. Best regards, Henning > 2018-08-02 15:29 GMT+02:00 Charles Chance <charles.cha...@sipcentric.com>: > > Hi Paolo, > > > > kamailio "resolves" A host but does not query the DNS > > > > > > What do you mean by this? > > > > I am not overly familiar with Kamailio's DNS internals, but what do you > > see if you run the following commands? > > > > kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local > > > > kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local > > > > Cheers, > > > > Charles > > > > > > On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud < > > > > paolo.visintin@evosip.cloud> wrote: > >> Hello again! > >> Another interesting thing found running in debug=3 > >> > >> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding > >> DMQ node A host dmq-router-service=sip:42.100.109.113:5062 > >> > >> so , kamailio "resolves" A host but does not query the DNS > >> this is dump of dns request inside kamailio container: > >> > >> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d > >> SRV dmq-router-service.paolo.svc.cluster.local > >> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query > >> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 > >> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A > >> 172.28.0.240 > >> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 > >> A dmq-router-service.paolo.svc.cluster.local > >> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query > >> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A > >> 172.28.0.240 > >> > >> > >> > >> *Paolo Visintin* > >> *CTO* > >> evosip.cloud > >> [image: Risultati immagini per evosip] > >> > >> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud < > >> > >> paolo.visintin@evosip.cloud>: > >>> Hello Charles, > >>> sorry for late reply, we've had some issues rebuilding docker images > >>> with kamailo nightly build! > >>> > >>> We are running on our local, private on premise kubernetes cluster so > >>> connections from outside is simply impossible (in dev we are using a > >>> "closed" network system) > >>> > >>> I've just tried to : > >>> - open a tcpdump trace in the instance tracing only the DMQ port defined > >>> [5062] - (tcpdump -i eth0 port 5062) > >>> - packet coming from nowhere > >>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e) > >>> - seeing this in tcpdump : > >>> > >>> root@router-0:/# tcpdump -i eth0 port 5062 > >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol > >>> decode > >>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 > >>> bytes > >>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > > >>> 42.100.109.113.5062: UDP, length 497 > >>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > > >>> 42.100.109.113.5062: UDP, length 497 > >>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > > >>> 42.100.109.113.5062: UDP, length 497 > >>> > >>> so seems that kamailio is trying to contact the foreign ip > >>> > >>> if i check inside the configuration: > >>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) > >>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) > >>> modparam("dmq", "multi_notify", 1) > >>> modparam("dmq", "num_workers", 4) > >>> modparam("dmq", "ping_interval", 15) > >>> > >>> > >>> ... > >>> > >>> if(is_method("KDMQ") && dst_port == DMQ_PORT){ > >>> > >>> dmq_handle_message(); > >>> exit; > >>> > >>> } > >>> > >>> and in DMQ_NOTIFICATION_ADDRESS I have : > >>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062" > >>> > >>> dmq-router-service DNS resolves: > >>> > dmq-router-service.paolo.svc.cluster.local > >>> > >>> Server: 10.96.0.10 > >>> Address: 10.96.0.10#53 > >>> > >>> Non-authoritative answer: > >>> Name: dmq-router-service.paolo.svc.cluster.local > >>> Address: 172.28.0.240 > >>> > >>> > >>> so -nowhere- i am able to find 42.100.109.113 > >>> > >>> > >>> and in kamailio log: > >>> 13(821) ERROR: dmq [notification_peer.c:599]: > >>> notification_resp_callback_f(): deleting server sip:42.100.109.113:5062 > >>> because of failed request > >>> > >>> > >>> > >>> Cheers, > >>> > >>> Paolo > >>> > >>> 2018-07-26 18:14 GMT+02:00 Charles Chance <charles.cha...@sipcentric.com > >>> > >>>> Hi Aleksandar, > >>>> > >>>> The initial depopulation of the nodes (following a period of 'pending' > >>>> state) is due to no response being received from them. Are you able to > >>>> trace the messages to/from one of them to confirm what is happening > >>>> there? > >>>> > >>>> As for the unrecognised IP, I'm afraid I can't answer that one. It must > >>>> be coming from somewhere - it's just a case of finding out where. > >>>> Following > >>>> the initial population/depopulation of the node list, DMQ simply > >>>> repeats > >>>> the process over and over until it eventually connects to one or more > >>>> other > >>>> nodes. Are you running on a public cloud somewhere or is it your own, > >>>> locally running Kubernetes cluster? > >>>> > >>>> Cheers, > >>>> > >>>> Charles > >>>> > >>>> > >>>> On 26 July 2018 at 15:30, Aleksandar Sosic <alex.sosic@evosip.cloud> > >>>> > >>>> wrote: > >>>>> Hi Charles, > >>>>> > >>>>> We're so glad about the improvements you just committed! Thanks! > >>>>> > >>>>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11 > >>>>> 65+xenial > >>>>> Kamailio starts even if the DNS record does not exist at first, that's > >>>>> great. I'm having this nodes up and running: > >>>>> ``` > >>>>> proxy-66f79498cc-8ws6d 3/3 Running 0 > >>>>> 4m 172.28.1.4 > >>>>> proxy-66f79498cc-b68dd 3/3 Running 0 > >>>>> 4m 172.28.1.5 > >>>>> proxy-66f79498cc-lhbxg 3/3 Running 0 > >>>>> 4m 172.28.1.6 > >>>>> proxy-66f79498cc-png6p 3/3 Running 0 > >>>>> 4m 172.28.1.7 > >>>>> ``` > >>>>> > >>>>> In those 4-5 minutes I've experienced a population and depopulation of > >>>>> the dmq nodes. > >>>>> > >>>>> For example: > >>>>> ``` > >>>>> kamcmd> dmq.list_nodes > >>>>> { > >>>>> host: 172.28.1.4 > >>>>> port: 5061 > >>>>> resolved_ip: 172.28.1.4 > >>>>> status: pending > >>>>> last_notification: 0 > >>>>> local: 0 > >>>>> } > >>>>> { > >>>>> host: 172.28.1.6 > >>>>> port: 5061 > >>>>> resolved_ip: 172.28.1.6 > >>>>> status: pending > >>>>> last_notification: 0 > >>>>> local: 0 > >>>>> } > >>>>> { > >>>>> host: 40.100.109.113 > >>>>> port: 5061 > >>>>> resolved_ip: 40.100.109.113 > >>>>> status: pending > >>>>> last_notification: 0 > >>>>> local: 0 > >>>>> } > >>>>> { > >>>>> host: 172.28.1.5 > >>>>> port: 5061 > >>>>> resolved_ip: 172.28.1.5 > >>>>> status: active > >>>>> last_notification: 0 > >>>>> local: 1 > >>>>> } > >>>>> ``` > >>>>> > >>>>> And after a few seconds only the loca node itself: > >>>>> ``` > >>>>> kamcmd> dmq.list_nodes > >>>>> { > >>>>> host: 172.28.1.5 > >>>>> port: 5061 > >>>>> resolved_ip: 172.28.1.5 > >>>>> status: active > >>>>> last_notification: 0 > >>>>> local: 1 > >>>>> } > >>>>> ``` > >>>>> And then again the 3 nodes and the 40.100.109.113 which is a Microsoft > >>>>> IP and I don't know what's doing that IP into my dmq server list. > >>>>> > >>>>> The DNS record is this: > >>>>> ``` > >>>>> root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service > >>>>> Server: 10.96.0.10 > >>>>> Address: 10.96.0.10#53 > >>>>> > >>>>> Non-authoritative answer: > >>>>> Name: dmq-proxy-service.alex.svc.cluster.local > >>>>> Address: 172.28.1.7 > >>>>> Name: dmq-proxy-service.alex.svc.cluster.local > >>>>> Address: 172.28.1.6 > >>>>> Name: dmq-proxy-service.alex.svc.cluster.local > >>>>> Address: 172.28.1.5 > >>>>> Name: dmq-proxy-service.alex.svc.cluster.local > >>>>> Address: 172.28.1.4 > >>>>> ``` > >>>>> Any clues why that ip is in the dmq nodes list? And why .7 is missing? > >>>>> Kind regards, > >>>>> -- > >>>>> Aleksandar Sosic > >>>>> mail: alex.sosic@evosip.cloud > >>>>> > >>>>> On Wed, Jul 25, 2018 at 3:06 PM Charles Chance > >>>>> > >>>>> <charles.cha...@sipcentric.com> wrote: > >>>>> > Hi, > >>>>> > > >>>>> > I have just pushed some changes to master - one of these allows > >>>>> > >>>>> startup to continue even if initial node resolution fails. > >>>>> > >>>>> > There are some other improvements, too, which I have been planning > >>>>> > >>>>> to push for some time and which should also help in your situation. > >>>>> > >>>>> > Can you try again with these changes applied and let me know the > >>>>> > >>>>> outcome? > >>>>> > >>>>> > Cheers, > >>>>> > > >>>>> > Charles > >>>>> > >>>>> _______________________________________________ > >>>>> Kamailio (SER) - Users Mailing List > >>>>> sr-users@lists.kamailio.org > >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > >>>> > >>>> -- > >>>> *Charles Chance* > >>>> Managing Director > >>>> > >>>> t. 0330 120 1200 m. 07932 063 891 > >>>> > >>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. > >>>> Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt > >>>> Street, > >>>> Birmingham Science Park, Birmingham B7 4BB. > >>>> > >>>> _______________________________________________ > >>>> Kamailio (SER) - Users Mailing List > >>>> sr-users@lists.kamailio.org > >>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > >> > >> _______________________________________________ > >> Kamailio (SER) - Users Mailing List > >> sr-users@lists.kamailio.org > >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > > > -- > > *Charles Chance* > > Managing Director > > > > t. 0330 120 1200 m. 07932 063 891 > > > > Sipcentric Ltd. Company registered in England & Wales no. 7365592. > > Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt > > Street, > > Birmingham Science Park, Birmingham B7 4BB. -- Henning Westerholt https://skalatan.de/blog/ _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users