Re: [cobbler] How to debug...
So, my adventure has yielded some interesting results.
I've been putting debug instrumenting/tracing statements into the code
as I go along and I've found out that indeed the code in pxegen is the
jumping off point for this issue. (N.B. I use my initials MCT to mark my
debugs and the first thing I do is put a marker into each
procedure/subroutine/method/function I go through.)
I tracked it down to the point where I figured out that the output I'm
expecting should be coming from:
pxegen.write_all_system_files(system, menu_items)
So I tracked this down and found that it is in the tftpd module named
manage_in_tftpd.py but not in the alternative one manage_tftpd_py.py
So, I tried using both.
First test I ran as I've been configured previously, using the tftpd.py
server:
task started: 2014-08-08_094804_sync
task started (id=Sync, time=Fri Aug 8 09:48:04 2014)
running pre-sync triggers
cleaning trees
removing: /var/www/cobbler/images/scilinall-x86_64
removing: /var/lib/tftpboot/pxelinux.cfg/default
removing: /var/lib/tftpboot/pxelinux.cfg/01-f4-6d-04-37-26-03
removing: /var/lib/tftpboot/pxelinux.cfg/01-f4-6d-04-37-26-91
removing: /var/lib/tftpboot/grub/01-F4-6D-04-37-26-91
removing: /var/lib/tftpboot/grub/grub-x86.efi
removing: /var/lib/tftpboot/grub/grub-x86_64.efi
removing: /var/lib/tftpboot/grub/efidefault
removing: /var/lib/tftpboot/grub/01-F4-6D-04-37-26-03
removing: /var/lib/tftpboot/grub/images
removing: /var/lib/tftpboot/images/scilinall-x86_64
removing: /var/lib/tftpboot/images/memtest
copying bootloaders
copying: /var/lib/cobbler/loaders/pxelinux.0 - /var/lib/tftpboot/pxelinux.0
copying: /var/lib/cobbler/loaders/menu.c32 - /var/lib/tftpboot/menu.c32
copying: /boot/memtest - /var/lib/tftpboot/images/memtest
copying: /var/lib/cobbler/loaders/yaboot - /var/lib/tftpboot/yaboot
copying: /var/lib/cobbler/loaders/grub-x86_64.efi -
/var/lib/tftpboot/grub/grub-x86_64.efi
copying: /var/lib/cobbler/loaders/grub-x86.efi -
/var/lib/tftpboot/grub/grub-x86.efi
copying files for distro: scilinall-x86_64
MCT Not remote kernel.
trying hardlink
/var/www/cobbler/ks_mirror/scilinall-x86_64/images/pxeboot/vmlinuz -
/var/www/cobbler/images/scilinall-x86_64/vmlinuz
MCT Not remote initrd.
trying hardlink
/var/www/cobbler/ks_mirror/scilinall-x86_64/images/pxeboot/initrd.img -
/var/www/cobbler/images/scilinall-x86_64/initrd.img
Writing template files for scilinall-x86_64
MCT make_actual_pxe_menu.
MCT get_menu_items.
MCT write_pxe_file.
MCT write_pxe_file. filename=None; system=None; profile.name=T2_Server;
distro.name=scilinall-x86_64
MCT write_pxe_file.
MCT write_pxe_file. filename=None; system=None; profile.name=T2_Server;
distro.name=scilinall-x86_64
MCT write_pxe_file.
MCT write_pxe_file. filename=None; system=None; profile.name=T2_Worker;
distro.name=scilinall-x86_64
MCT write_pxe_file.
MCT write_pxe_file. filename=None; system=None; profile.name=T2_Worker;
distro.name=scilinall-x86_64
MCT write_pxe_file.
MCT write_pxe_file. filename=None; system=None; profile.name=scilinall-x86_64;
distro.name=scilinall-x86_64
MCT write_pxe_file.
MCT write_pxe_file. filename=None; system=None; profile.name=scilinall-x86_64;
distro.name=scilinall-x86_64
rendering DHCP files
generating /etc/dhcp/dhcpd.conf
rendering TFTPD files
generating /etc/xinetd.d/tftp
cleaning link caches
running: find /var/lib/tftpboot/images/.link_cache -maxdepth 1 -type f -links 1
-exec rm -f '{}' ';'
received on stdout:
received on stderr:
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running: dhcpd -t -q
received on stdout:
received on stderr:
running: service dhcpd restart
received on stdout: Shutting down dhcpd: [ OK ]
Starting dhcpd: [ OK ]
received on stderr:
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running shell triggers from /var/lib/cobbler/triggers/change/*
*** TASK COMPLETE ***
And when I looked into the tftp directory I got:
# find /var/lib/tftpboot/ -type f
/var/lib/tftpboot/pxelinux.cfg/default
/var/lib/tftpboot/pxelinux.0
/var/lib/tftpboot/grub/grub-x86.efi
/var/lib/tftpboot/grub/grub-x86_64.efi
/var/lib/tftpboot/grub/efidefault
/var/lib/tftpboot/boot/grub/menu.lst
/var/lib/tftpboot/yaboot
/var/lib/tftpboot/memdisk
/var/lib/tftpboot/menu.c32
/var/lib/tftpboot/images/memtest
As we know, this is what I've been complaining about, no system specific
PXE files. So, now I switch tftp modules and use in_tftpd in the
modules.conf file and do the exact same test:
task started: 2014-08-08_100040_sync
task started (id=Sync, time=Fri Aug 8 10:00:40 2014)
running pre-sync triggers
cleaning trees
removing: /var/www/cobbler/images/scilinall-x86_64
removing: /var/lib/tftpboot/pxelinux.cfg/default
removing: /var/lib/tftpboot/grub/grub-x86.efi
removing:
Re: [cobbler] How to debug...
On 8/2/14 5:57 AM, André Gemünd wrote:
thats a bit strange, as you should at least get some error message.
Was that a full cobbler sync output you posted? There should e.g. at
least be a 'generating PXE configuration files' output. Which
version of Cobbler are you using? cobbler distro list, cobbler
profile list cobbler system list all output the correct items and
give you no error message? Nothing in the logs? The code you are
looking for is in pxegen.py btw.
Yeah, a bit strange should be my middle name these days.
I've attached a number of outputs to this email so you can see all the
strangeness for yourself. It has gotten to the point that I'm using a
really limited configuration right now to get to the bottom of this
problem. For lots of reasons (internal) I've had to reload the machine
multiple times so this is a very clean installation. I've been using
simple scripts to populate the key components such as profiles, and
systems so not a lot has been changed.
The thing is, the DHCP part works fine. The tftp service is responding
properly. Everything seems to be working fine *except* that when the
sync is called, it doesn't seem to recognize that it has a machine
enabled for netboot and therefore has to generate the PXE files needed
to make it happen.
Opinions are welcome and thank you to all for your help.
Here's that output:
::
sync-out.txt
::
task started: 2014-08-04_094050_sync
task started (id=Sync, time=Mon Aug 4 09:40:50 2014)
running pre-sync triggers
cleaning trees
removing: /var/www/cobbler/images/scilin_6.4-x86_64
removing: /var/lib/tftpboot/pxelinux.cfg/default
removing: /var/lib/tftpboot/grub/images
removing: /var/lib/tftpboot/grub/efidefault
removing: /var/lib/tftpboot/grub/grub-x86.efi
removing: /var/lib/tftpboot/grub/grub-x86_64.efi
removing: /var/lib/tftpboot/s390x/profile_list
copying bootloaders
copying: /var/lib/cobbler/loaders/pxelinux.0 - /var/lib/tftpboot/pxelinux.0
copying: /var/lib/cobbler/loaders/menu.c32 - /var/lib/tftpboot/menu.c32
copying: /var/lib/cobbler/loaders/yaboot - /var/lib/tftpboot/yaboot
copying: /var/lib/cobbler/loaders/grub-x86_64.efi -
/var/lib/tftpboot/grub/grub-x86_64.efi
copying: /var/lib/cobbler/loaders/grub-x86.efi -
/var/lib/tftpboot/grub/grub-x86.efi
copying files for distro: scilin_6.4-x86_64
trying hardlink
/var/www/cobbler/ks_mirror/scilin_6.4/images/pxeboot/vmlinuz -
/var/www/cobbler/images/scilin_6.4-x86_64/vmlinuz
trying hardlink
/var/www/cobbler/ks_mirror/scilin_6.4/images/pxeboot/initrd.img -
/var/www/cobbler/images/scilin_6.4-x86_64/initrd.img
Writing template files for scilin_6.4-x86_64
rendering DHCP files
generating /etc/dhcp/dhcpd.conf
rendering TFTPD files
generating /etc/xinetd.d/tftp
cleaning link caches
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running: dhcpd -t -q
received on stdout:
received on stderr:
running: service dhcpd restart
received on stdout: Shutting down dhcpd: [ OK ]
Starting dhcpd: [ OK ]
received on stderr:
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running shell triggers from /var/lib/cobbler/triggers/change/*
*** TASK COMPLETE ***
::
/etc/cobbler/settings
::
allow_duplicate_hostnames: 0
allow_duplicate_ips: 0
allow_duplicate_macs: 0
allow_dynamic_settings: 0
always_write_dhcp_entries: 1
anamon_enabled: 0
auth_token_expiration: 3600
authn_pam_service: login
bind_chroot_path: ''
bind_master: 127.0.0.1
build_reporting_email: [root@localhost]
build_reporting_enabled: 0
build_reporting_ignorelist: ['']
build_reporting_sender: ''
build_reporting_smtp_server: localhost
build_reporting_subject: ''
build_reporting_to_address: ''
buildisodir: /var/cache/cobbler/buildiso
cheetah_import_whitelist: [random, re, time]
client_use_https: 0
client_use_localhost: 0
cobbler_master: ''
consoles: /var/consoles
createrepo_flags: -c cache -s sha
default_deployment_method: ssh
default_kickstart: /var/lib/cobbler/kickstarts/default.ks
default_name_servers: []
default_name_servers_search: []
default_ownership: [admin]
default_password_crypted:
default_template_type: cheetah
default_virt_bridge: xenbr0
default_virt_disk_driver: raw
default_virt_file_size: 5
default_virt_ram: 512
default_virt_type: xenpv
enable_gpxe: 0
enable_menu: 1
func_auto_setup: 0
func_master: overlord.example.org
http_port: 80
isc_set_host_name: 0
iso_template_dir: /etc/cobbler/iso
kerberos_realm: EXAMPLE.COM
kernel_options: {ksdevice: bootif, lang: ' ', text: null}
kernel_options_s390x: {RUNKS: 1, ip: false, ramdisk_size: 4, ro:
null, root: /dev/ram0,
vnc: null}
ldap_anonymous_bind: 1
ldap_base_dn: DC=example,DC=com
ldap_management_default_type: authconfig
ldap_port: 389
ldap_search_bind_dn: ''
ldap_search_passwd: ''
Re: [cobbler] How to debug...
On 8/2/14 12:02 PM, Alastair Munro wrote: Are you sure static was not ticked for ip on the interface ? It is and that's legal. It still generates the correct DHCP information (which works properly.) (And yes, I've tried it without now too.) ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
Hi Michael, which version is this? 2.6? Your sync output is indeed incomplete. The interesting part is right before copying files for distro (which is already handled by a different code branch). You are missing a lot of output there, e.g. copying distros to tftpboot, copying images, generating PXE configuration files, etc. Thats why I'd assume that it fails inside pxegen.copy_bootloaders(). Unfortunately, it does not contain much logging. I'd start there and add logging myself if I was you (e.g. to the last line of copy_bootloaders, to see if it ever reaches there). Are you sure static was not ticked for ip on the interface ? It is and that's legal. It still generates the correct DHCP information (which works properly.) (And yes, I've tried it without now too.) Yes, the static is just for post-installation configuration, otherwise you could never PXE install these machines. Although if always_write_dhcp_entries is off, and netboot is disabled, the DHCP entry is left out, imho. Greetings Andre -- André Gemünd Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemu...@scai.fraunhofer.de Tel: +49 2241 14-2193 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
Hi Michael, thats a bit strange, as you should at least get some error message. Was that a full cobbler sync output you posted? There should e.g. at least be a 'generating PXE configuration files' output. Which version of Cobbler are you using? cobbler distro list, cobbler profile list cobbler system list all output the correct items and give you no error message? Nothing in the logs? The code you are looking for is in pxegen.py btw. Greetings Andre - Ursprüngliche Mail - On 7/31/14 7:41 PM, alast...@alastair-munro.com wrote: Is there a file with the mac address in this dir? This is the problem, no MAC based files are being created. I'm sure it's something that will be obvious once it's found. :( ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler -- André Gemünd Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemu...@scai.fraunhofer.de Tel: +49 2241 14-2193 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
Are you sure static was not ticked for ip on the interface ? -- Alastair Munro -Original Message- From: André Gemünd andre.gemu...@scai.fraunhofer.de To: mtier...@mit.edu, cobbler mailing list cobbler@lists.fedorahosted.org Sent: Sat, 02 Aug 2014 10:57 Subject: Re: [cobbler] How to debug... Hi Michael, thats a bit strange, as you should at least get some error message. Was that a full cobbler sync output you posted? There should e.g. at least be a 'generating PXE configuration files' output. Which version of Cobbler are you using? cobbler distro list, cobbler profile list cobbler system list all output the correct items and give you no error message? Nothing in the logs? The code you are looking for is in pxegen.py btw. Greetings Andre - Ursprüngliche Mail - On 7/31/14 7:41 PM, alast...@alastair-munro.com wrote: Is there a file with the mac address in this dir? This is the problem, no MAC based files are being created. I'm sure it's something that will be obvious once it's found. :( ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler -- André Gemünd Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemu...@scai.fraunhofer.de Tel: +49 2241 14-2193 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
On 7/31/14 7:41 PM, alast...@alastair-munro.com wrote: Is there a file with the mac address in this dir? This is the problem, no MAC based files are being created. I'm sure it's something that will be obvious once it's found. :( ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
Okay, here's what I'm running into now... I have a system that I just rebuilt as my host server. Everything built without a hitch. However, now, I've added a distro, two profiles, and a system. The system definition ends up in the dhcp config file but the sync process does NOT build the PXE boot files required for this system (MAC/UUID based), it just skips right over that part of the process. I need a suggestion for what to look for to troubleshoot this. Oh yea, the system has the netboot enabled so it *should* be getting built. Any thoughts anyone? I'm getting pretty irked by this. (I've rebuilt it twice for other purposes and keep running into this.) ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
if you are using the python tftp server these system definitions don't get generated iirc. check with: grep ^module.*tftp.* /etc/cobbler/modules.conf On Thu, Jul 31, 2014 at 8:45 PM, Michael Tiernan mtier...@mit.edu wrote: Okay, here's what I'm running into now... I have a system that I just rebuilt as my host server. Everything built without a hitch. However, now, I've added a distro, two profiles, and a system. The system definition ends up in the dhcp config file but the sync process does NOT build the PXE boot files required for this system (MAC/UUID based), it just skips right over that part of the process. I need a suggestion for what to look for to troubleshoot this. Oh yea, the system has the netboot enabled so it *should* be getting built. Any thoughts anyone? I'm getting pretty irked by this. (I've rebuilt it twice for other purposes and keep running into this.) ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler -- Grtz, Jörgen Maas ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
On 7/31/14 2:57 PM, Jörgen Maas wrote: if you are using the python tftp server these system definitions don't get generated iirc. I'm sorry, maybe I'm dense. I'm obviously saying the wrong thing here because the tftp server selection doesn't change this. (I've tested it too.) When the target system initiates the PXE boot process, after the address is assigned, the target system requests, via tftp protocol a file which, if the MAC address of the target matches, is sent back to the target to then boot from. It is *those* files and confgs that are not being created and prepared for these target systems. ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
Check next_server in /etc/cobbler settings, restart xinetd, add -v (or -vv or -vvv) to /etc/xinetd.d/tftp file, what is in /var/lib/tftp/pxelinux.cfg? Is there a file with the mac address in this dir? Check logging in /var/log/messages... All this should give you an idea of what is going on. If you are using a different subnet to the primary one in cobbler settings, set the server override to be the ip of the interface for the subnet (this propogates to the tftp next server and the server address in the pxelinux cfg file; change the next_server setting in the dhcp template to be $iface.next_server). On 2014-07-31 20:17, Michael Tiernan wrote: On 7/31/14 2:57 PM, Jörgen Maas wrote: if you are using the python tftp server these system definitions don't get generated iirc. I'm sorry, maybe I'm dense. I'm obviously saying the wrong thing here because the tftp server selection doesn't change this. (I've tested it too.) When the target system initiates the PXE boot process, after the address is assigned, the target system requests, via tftp protocol a file which, if the MAC address of the target matches, is sent back to the target to then boot from. It is *those* files and confgs that are not being created and prepared for these target systems. ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Re: [cobbler] How to debug...
On 7/25/14 7:36 PM, Drew Decker wrote: It looks like you are having it manage your DHCP for you. Yeah, sorry, that's not the problem I'm running into. I'm assuming that there's a point in the 'sync' process where cobbler goes through the list of systems looking for conditions to trigger it creating the PXE boot stuff appropriate for that MAC address. This is what I'm looking for. How to debug this aspect of the process. ___ cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
[cobbler] How to debug...
I'm sure I can solve this by hard work but I'm wondering if there's a
way to trigger a debugging so that the steps that cobbler's supposed to
be doing are obvious.
I've got a single system in my cobbler instance and it *should* prepare
the tftp stuff for the pxe boot but I'll be darned if I can see why it's
NOT doing this.
I'm hoping there's a flag one can turn on (or whatever) that will allow
me to see the steps it's attempting to take. I'm sure I'm missing
something obvious and I'd like to find it before I go crazy. :)
Thanks for everyone's time.
task started: 2014-07-22_163236_sync
task started (id=Sync, time=Tue Jul 22 16:32:36 2014)
running pre-sync triggers
cleaning trees
removing: /var/www/cobbler/images/scilinall-x86_64
removing: /var/lib/tftpboot/pxelinux.cfg/default
removing: /var/lib/tftpboot/grub/images
removing: /var/lib/tftpboot/grub/efidefault
removing: /var/lib/tftpboot/grub/grub-x86_64.efi
removing: /var/lib/tftpboot/grub/grub-x86.efi
removing: /var/lib/tftpboot/s390x/profile_list
copying bootloaders
copying: /var/lib/cobbler/loaders/pxelinux.0 - /var/lib/tftpboot/pxelinux.0
copying: /var/lib/cobbler/loaders/menu.c32 - /var/lib/tftpboot/menu.c32
copying: /var/lib/cobbler/loaders/yaboot - /var/lib/tftpboot/yaboot
copying: /var/lib/cobbler/loaders/grub-x86_64.efi -
/var/lib/tftpboot/grub/grub-x86_64.efi
copying: /var/lib/cobbler/loaders/grub-x86.efi -
/var/lib/tftpboot/grub/grub-x86.efi
copying files for distro: scilinall-x86_64
trying hardlink /var/www/cobbler/ks_mirror/scilinall/images/pxeboot/vmlinuz -
/var/www/cobbler/images/scilinall-x86_64/vmlinuz
trying hardlink /var/www/cobbler/ks_mirror/scilinall/images/pxeboot/initrd.img
- /var/www/cobbler/images/scilinall-x86_64/initrd.img
Writing template files for scilinall-x86_64
rendering DHCP files
generating /etc/dhcp/dhcpd.conf
rendering TFTPD files
generating /etc/xinetd.d/tftp
cleaning link caches
running: find /var/lib/tftpboot/images/.link_cache -maxdepth 1 -type f -links 1
-exec rm -f '{}' ';'
received on stdout:
received on stderr:
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running: dhcpd -t -q
received on stdout:
received on stderr:
running: service dhcpd restart
received on stdout: Shutting down dhcpd: [ OK ]
Starting dhcpd: [ OK ]
received on stderr:
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running shell triggers from /var/lib/cobbler/triggers/change/*
*** TASK COMPLETE ***
___
cobbler mailing list
cobbler@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/cobbler
