Josh, Russell is correct, xcat is using Dynamic DNS, so the dns records are not directly added into db.xx.xx files, instead, they are added into db.xx.xx.jnl journal files, one of benefits of Dynamic DNS is named/bind daemon does not have to restart once the dns records are updated.
So in your case, it's correct that you can only see the headnode in
db.xx.xx file, because the others are in db.xx.xx.jnl files :) and you can
follow Russell's suggestion to dump dns records from journal files into
db.xx.xx files, or just use "nslookup" or "host" utilities to verify a
host name resolution.
For your concern about "makedns wiping out DNS records", I think it should
be related to the usage, please "man makedns" to see the details, but here
I want to just give a simple summary.
-n|--new
Use this flag to create new named configuration and db files.
if you specify the -n flag, then makedns will remove /etc/named.conf and
/<bind db dir>/db.*, then regenerate/create the new files. so if you want
to test delete and re-add a node to dns, such as xcat-serv1, here is the
samples:
# to delete it from dns:
makedns -d xcat-serv1
# to add it into dns
makedns xcat-serv1 # here, please notice, there is no "-n" flag
needed, because we are not wanting to refresh all named configurations(
/etc/named.conf and /<bind db dir>/db.*), we just want to add a new dns
record
Lissa has posted all Examples we listed in makedns manpage, FYI.
Any questions, please let me know.
Thx.
Sun Jing
Josh Nielsen <[email protected]>
2014/01/17 04:45
Please respond to
xCAT Users Mailing list <[email protected]>
To
xCAT Users Mailing list <[email protected]>,
cc
Subject
Re: [xcat-user] Makedns wiping out DNS records
Thanks, that is very helpful. Yeah, to cut my teeth on xCAT for the first
time in early 2012 I set up a local VirtualBox VM environment to test it
in. But thank heavens we have VMWare now so it may be easier to set up a
test environment there. And I have tried 'dig' before but never took the
time to research it in depth. I have borrowed my manager's "DNS and BIND"
O'Reilly book and keep telling myself I'll read it (I told him last
November that I might take it with me to a family outing over Thanksgiving
for "pleasure reading" - heh, yeah right...) but haven't gotten around to
it yet. One of these days I'll be a DNS/BIND expert I'm sure though. :-)
Thanks,
Josh
On Thu, Jan 16, 2014 at 2:33 PM, Russell Jones <[email protected]>
wrote:
Correct. You may be able to also cause it to flush by doing an rndc freeze
/ rndc thaw on the zone. Google "bind journal files flush".
I'm sorry that you are having to learn all of this on a production system.
It might be worth setting up some VMs in virtualbox to work out the kinks
with DNS / service node deployment before diving head first into
production if possible :-)
On 1/16/2014 2:27 PM, Josh Nielsen wrote:
That would make sense of a few things. I was afraid to restart the daemon
when I didn't see anything in the db.X.X files because I thought it was
currently running on some kind of cached or memory resident results, and
that if I restarted it would break everything after rereading an empty
zone file. So you are saying if I restarted the daemon again a second time
after makedns (which I think forces a restart during it doesn't it?) that
the zone files would be populated?
The only real snag in me testing this is that this is in a heavily
utilized production environment that runs cluster jobs 24x7 except for
scheduled maintenance outages, and I have seen the DNS break on one
occasion and the result was not pretty on the cluster. I don't actually
have a choice for now, but this is just to explain why I was afraid to
restart named without knowing what it would do when I saw nothing in the
zone files. Once I get some Service Nodes up hopefully they can provide me
a redundant DNS to the cluster so I can more easily take risks like
restarting the named daemon multiple times. :-)
-Josh
On Thu, Jan 16, 2014 at 12:50 PM, Russell Jones <[email protected]
> wrote:
Just a tip and something that bit me before - bind updates its data in the
journal and won't dump new data to the zone files immediately. You can
force it do dump the data so that you can see it by restarting bind.
You can also test to make sure DNS is being added by digging for the
record. IE, dig @localhost dayhoff.morgan.haib.org. If you get an A record
back, your DNS is working fine.
On 1/16/2014 12:39 PM, Josh Nielsen wrote:
Is there any way to debug what exactly the makedns command does to
named/bind, as in what command it issues? Or does it directly edit the
named files? I could run the named daemon with "-d 9" for debugging level
9 but I've done that before and the output is daunting and nearly
incomprehensible for the non-veteran in DNS. As I said, running "makedns"
by it self works fine, but using -d or -n causes problems. I have no way
of knowing what xCAT does in the background with those options.
I checked and /etc/hosts is correct, and /etc/resolv.conf is as well. The
site table has all five possible dns attributes (domain, dnshandler,
forwarders, master, and nameservers) set correctly, and the networks are
defined for every subnet we use. There is nothing that I can see that
would cause it to parse incorrectly, especially since it echoed to
/var/log/messages that it was going through /etc/hosts and creating
entries (but really it didn't) for each host. I did have to add one entry
manually into /etc/hosts though for a SAN storage system. Would seeing an
unrecognized node cause problems? And if so why is there nothing in the
logs about it? I saw it say that it was adding an entry for that node
(even though it didn't - like every other entry):
"Jan 16 10:32:53 x3650-head01 named[8249]: client 10.20.0.1#56079:
updating zone 'morgan.haib.org/IN': adding an RR at '
dayhoff.morgan.haib.org' A"
If I can find out what xCAT is doing to make the changes perhaps I can
reproduce it manually in a test environment and see what is causing it to
wipe out the entries.
-Josh
On Thu, Jan 16, 2014 at 12:01 PM, Lissa Valletta <[email protected]>
wrote:
You will see in the document that makedns is very dependent on what you do
to setup you Management node. Your /etc/hosts /etc/resolv.conf, etc
Setting that must be added to the database for your cluster such as in
the site table, defining your network in the networks table. You can use
that document to check your database configuration.
Lissa K. Valletta
8-3/B10
Poughkeepsie, NY 12601
(tie 293) 433-3102
Josh Nielsen ---01/16/2014 12:44:27 PM---Okay, I will look through that
document. This is the first and only time I've used XCATBYPASS with t
From: Josh Nielsen <[email protected]>
To: xCAT Users Mailing list <[email protected]>,
Date: 01/16/2014 12:44 PM
Subject: Re: [xcat-user] Makedns wiping out DNS records
Okay, I will look through that document. This is the first and only time
I've used XCATBYPASS with that command (to get more clarity into what was
going on in the background). Makedns has been presenting us with the
problem(s) I mentioned for well over a year now, with previous versions of
xCAT as well. I'm guessing that if this doesn't happen for other people
that it must have been some setting when xCAT was originally installed
which was misconfigured, or maybe something funny in one of the xCAT db
tables like 'site'. I just wanted to see if anyone has had problems with
using the "-n" or "-d" options deleting more than they are supposed to.
-Josh
On Thu, Jan 16, 2014 at 11:23 AM, Lissa Valletta <[email protected]>
wrote:
We have an entire doc on name resolution also I would not make it a
common practice to use XCATBYPASS mode. Running with the xcatd is much
more secure and some things do not work well in XCATBYPASS mode.
https://sourceforge.net/apps/mediawiki/xcat/index.php?title=Cluster_Name_Resolution
Lissa K. Valletta
8-3/B10
Poughkeepsie, NY 12601
(tie 293) 433-3102
Josh Nielsen ---01/16/2014 12:03:28 PM---Regardless of the version of xCAT
we have used in the past, my coworkers and I have noticed that run
From: Josh Nielsen <[email protected]>
To: xCAT Users Mailing list <[email protected]>,
Date: 01/16/2014 12:03 PM
Subject: [xcat-user] Makedns wiping out DNS records
Regardless of the version of xCAT we have used in the past, my coworkers
and I have noticed that running "makedns" with any following options can
be dangerous for us, because something happens where it will wipe out all
the records in DNS. If we run "makedns" with no options generally any new
nodes that have been added to /etc/hosts will be parsed and added
appropriately. Maybe I am misunderstanding what happens when the command
is run.
Here is a recent example. My entry for a new node "xcat-serv1" for some
reason didn't have the reverse lookup from the IP in the DNS file for that
subnet (db.10.20), and so I decided to try to remove just the DNS entry
for that node and then re-add it. Here is what I ran:
----
# makedns -d xcat-serv1
Handling xcat-serv1 in /etc/hosts.
Getting reverse zones, this may take several minutes for a large cluster.
Completed getting reverse zones.
Updating zones.
Completed updating zones.
Updating DNS records, this may take several minutes for a large cluster.
Completed updating DNS records.
----
Then I ran it with the "-n" parameter to re-add that node (with
XCATBYPASS=1):
----
# export XCATBYPASS=1
[root@x3650-head01 centos]# makedns -n xcat-serv1
Unrecognized escape \h passed through at
/opt/xcat/lib/perl/xCAT/zvmUtils.pm line 2408.
Subroutine handled_commands redefined at /opt/xcat/lib/perl/xCAT_plugin/
hosts.pm line 28.
Subroutine delnode redefined at /opt/xcat/lib/perl/xCAT_plugin/hosts.pm
line 33.
Subroutine addnode redefined at /opt/xcat/lib/perl/xCAT_plugin/hosts.pm
line 58.
Subroutine build_line redefined at /opt/xcat/lib/perl/xCAT_plugin/hosts.pm
line 117.
Subroutine addotherinterfaces redefined at /opt/xcat/lib/perl/xCAT_plugin/
hosts.pm line 182.
Subroutine delotherinterfaces redefined at /opt/xcat/lib/perl/xCAT_plugin/
hosts.pm line 209.
Subroutine add_hosts_content redefined at /opt/xcat/lib/perl/xCAT_plugin/
hosts.pm line 234.
Subroutine process_request redefined at /opt/xcat/lib/perl/xCAT_plugin/
hosts.pm line 305.
Subroutine writeout redefined at /opt/xcat/lib/perl/xCAT_plugin/hosts.pm
line 511.
Subroutine donics redefined at /opt/xcat/lib/perl/xCAT_plugin/hosts.pm
line 545.
Subroutine getIPdomain redefined at /opt/xcat/lib/perl/xCAT_plugin/
hosts.pm line 801.
Handling xcat-serv1 in /etc/hosts.
Getting reverse zones, this may take several minutes for a large cluster.
Completed getting reverse zones.
Stopping named: . [ OK ]
Updating zones.
Completed updating zones.
Restarting named
Restarting named complete
Updating DNS records, this may take several minutes for a large cluster.
Completed updating DNS records.
DNS setup is completed
....
....
----
When I did an 'ls' in /var/named many records were missing and the ones
still left we virtually empty except one entry for the headnode:
BEFORE:
# ls /var/named
data db.10.29 db.10.40 db.127.0.0 db.172.26.41
db.172.29 db.192.168.100 db.haig db.morgan.haib.org.jnl
db.10.29.jnl db.10.60 db.172.20 db.172.26.42 db.172.30
db.cache db.morgan slaves
db.10.20 db.10.30 db.10.60.jnl db.172.26.40 db.172.26.43
db.172.40 db.haib db.morgan.haib.org
AFTER:
# ls /var/named
data db.10.20.jnl db.10.29.jnl db.10.30.jnl db.10.40.jnl
db.10.60.jnl db.172.26.41 db.172.26.43 db.morgan.haib.org.jnl
db.10.20 db.10.29 db.10.30 db.10.40 db.10.60
db.172.26.40 db.172.26.42 db.morgan.haib.org slaves
As you can see several files are missing, Luckily I made a full backup
yesterday and was able to restore the old files.
>From /var/log/messages you can see the initial deletion message for
"xcat-serv1":
----
Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: updating
zone '40.10.IN-ADDR.ARPA/IN': adding an RR at '11.102.40.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: updating
zone '40.10.IN-ADDR.ARPA/IN': adding an RR at '7.101.40.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: updating
zone '40.10.IN-ADDR.ARPA/IN': adding an RR at '10.102.40.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: updating
zone '40.10.IN-ADDR.ARPA/IN': adding an RR at '27.102.40.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: updating
zone '40.10.IN-ADDR.ARPA/IN': adding an RR at '33.101.40.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: updating
zone '40.10.IN-ADDR.ARPA/IN': adding an RR at '1.102.40.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: updating
zone
Then for the makedens -a xcat-serv you actually see it update EVERY ZONE:
Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: updating
zone '30.10.IN-ADDR.ARPA/IN': adding an RR at '1.50.30.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: updating
zone '30.10.IN-ADDR.ARPA/IN': adding an RR at '4.50.30.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: updating
zone '30.10.IN-ADDR.ARPA/IN': adding an RR at '2.2.30.10.IN-ADDR.ARPA' PTR
Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: updating
zone '30.10.IN-ADDR.ARPA/IN': adding an RR at '3.50.30.10.IN-ADDR.ARPA'
PTR
Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: updating
zone '30.10.IN-ADDR.ARPA/IN': adding an RR at '1.2.30.10.IN-ADDR.ARPA' PTR
Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: updating
zone '30.10.IN-ADDR.ARPA/IN': adding an RR at '2.50.30.10.IN-ADDR.ARPA'
PTR
....
[long list]
----
After this is when I listed and saw files missing and most of the records
empty. My guess is the "-d" wiped out all the records somehow and not just
"xcat-serv1". Either that or the makedns -a deleted all the records in
order to recreate them. But why would it affect ALL the records when I
specifically typed "xcat-serv1"? I tried a plain "makedns" with no options
after this and it also says "adding an RR at" (is that even the expected
message?) for all the IPs and zones but nothing actually gets added. What
is up with that? If it just something with our installation or does
makedns do this foe other people as well?
Regards,
Josh
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
<<image/gif>>
<<image/gif>>
------------------------------------------------------------------------------ CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments & Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________ xCAT-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/xcat-user
