Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Geert Stappers]

On Wed, Dec 21, 2016 at 01:38:23PM +0100, Pali Rohár wrote:
> On Wednesday 21 December 2016 10:16:24 Philip Hands wrote:
> > If so, we could just check for '#!ipxe' and if found downgrade the
> > error to a warning, or perhaps to log what happened but otherwise
> > ignore it.
> 
> As DHCP bootfile can be served for any application, I think that Debian 
> Installer should check that file is for him, not that file is for PXE...
> 
> With your way, to just check for '#!ipxe' can be broken again in future 
> once there will be another application which will use bootfile option. 
> And it could be any fork of ipxe with new name... as it was before with 
> gpxe...

True

In #848726 ( https://bugs.debian.org/848726 848...@bugs.debian.org )
is a discussion about choosing a string that makes a preseed file
easy identifable by computer.


Groeten
Geert Stappers
-- 
Leven en laten leven

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Pali Rohár]

On Wednesday 21 December 2016 10:36:30 Philip Hands wrote:
> Philip Hands  writes:
> > Pali Rohár  writes:
> >> Package: debian-installer
> >> Severity: normal
> >> 
> >> Dear Maintainer,
> >> 
> >> when DHCP server is configured to send bootfile option with iPXE
> >> script
> >> 
> >> then debian-installer fails with following "red" error:
> >>   [!!] Download debconf preconfiguration file
> >>   
> >>   Failed to process the preconfiguration file
> >>   
> >>   The installer failed to process the preconfiguration file from
> >>   . The file may be corrupt.
> >> 
> >> (where  is value of that DHCP option)
> >> 
> >> It looks like debian-installer expects that DHCP bootfile option
> >> will contains correct preseed file. And if that DHCP option
> >> contains not Debian preseed file, then it show above "red" error
> >> message.
> > 
> > Is it the case that iPXE config files are the only things we need
> > to worry about here?  (seems probable given the lack of previous
> > reports)
> > 
> > If so, we could just check for '#!ipxe' and if found downgrade the
> > error to a warning, or perhaps to log what happened but otherwise
> > ignore it.
> 
> It occurs to me that we could fix this by adding a feature.
> 
> If we had something that looked out for some string, and would
> interpret it as an instruction to immediately chain into a new file,
> ignoring the rest of the current file, then one could specify the
> preseed to use in the iPXE config, by adding something like this
> near the start:
> 
>   # DEBCONF_CHAIN_LOAD(preseed.cfg)
> 
> If we were looking for /DEBCONF_CHAIN_LOAD([^(]*)/ then it would
> probably work in anything that has a way of specifying comments,
> since it could as easily find it after '// ', 

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Pali Rohár]

On Wednesday 21 December 2016 10:16:24 Philip Hands wrote:
> Pali Rohár  writes:
> > Package: debian-installer
> > Severity: normal
> > 
> > Dear Maintainer,
> > 
> > when DHCP server is configured to send bootfile option with iPXE
> > script
> > 
> > then debian-installer fails with following "red" error:
> >   [!!] Download debconf preconfiguration file
> >   
> >   Failed to process the preconfiguration file
> >   
> >   The installer failed to process the preconfiguration file from
> >   . The file may be corrupt.
> > 
> > (where  is value of that DHCP option)
> > 
> > It looks like debian-installer expects that DHCP bootfile option
> > will contains correct preseed file. And if that DHCP option
> > contains not Debian preseed file, then it show above "red" error
> > message.
> 
> Is it the case that iPXE config files are the only things we need to
> worry about here?  (seems probable given the lack of previous
> reports)

I do not know. Maybe there are other applications which uses DHCP 
bootfile option for own setup...

> If so, we could just check for '#!ipxe' and if found downgrade the
> error to a warning, or perhaps to log what happened but otherwise
> ignore it.

As DHCP bootfile can be served for any application, I think that Debian 
Installer should check that file is for him, not that file is for PXE...

With your way, to just check for '#!ipxe' can be broken again in future 
once there will be another application which will use bootfile option. 
And it could be any fork of ipxe with new name... as it was before with 
gpxe...

-- 
Pali Rohár
pali.ro...@gmail.com


signature.asc
Description: This is a digitally signed message part.

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Philip Hands]

Philip Hands  writes:

> Pali Rohár  writes:
>
>> Package: debian-installer
>> Severity: normal
>>
>> Dear Maintainer,
>>
>> when DHCP server is configured to send bootfile option with iPXE script
>> then debian-installer fails with following "red" error:
>>
>>   [!!] Download debconf preconfiguration file
>>
>>   Failed to process the preconfiguration file
>>
>>   The installer failed to process the preconfiguration file from .
>>   The file may be corrupt.
>>
>> (where  is value of that DHCP option)
>>
>> It looks like debian-installer expects that DHCP bootfile option will
>> contains correct preseed file. And if that DHCP option contains not
>> Debian preseed file, then it show above "red" error message.
>
> Is it the case that iPXE config files are the only things we need to
> worry about here?  (seems probable given the lack of previous reports)
>
> If so, we could just check for '#!ipxe' and if found downgrade the error
> to a warning, or perhaps to log what happened but otherwise ignore it.

It occurs to me that we could fix this by adding a feature.

If we had something that looked out for some string, and would interpret
it as an instruction to immediately chain into a new file, ignoring the
rest of the current file, then one could specify the preseed to use in
the iPXE config, by adding something like this near the start:

  # DEBCONF_CHAIN_LOAD(preseed.cfg)

If we were looking for /DEBCONF_CHAIN_LOAD([^(]*)/ then it would
probably work in anything that has a way of specifying comments, since
it could as easily find it after '// ', 

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Philip Hands]

Pali Rohár  writes:

> Package: debian-installer
> Severity: normal
>
> Dear Maintainer,
>
> when DHCP server is configured to send bootfile option with iPXE script
> then debian-installer fails with following "red" error:
>
>   [!!] Download debconf preconfiguration file
>
>   Failed to process the preconfiguration file
>
>   The installer failed to process the preconfiguration file from .
>   The file may be corrupt.
>
> (where  is value of that DHCP option)
>
> It looks like debian-installer expects that DHCP bootfile option will
> contains correct preseed file. And if that DHCP option contains not
> Debian preseed file, then it show above "red" error message.

Is it the case that iPXE config files are the only things we need to
worry about here?  (seems probable given the lack of previous reports)

If so, we could just check for '#!ipxe' and if found downgrade the error
to a warning, or perhaps to log what happened but otherwise ignore it.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Pali Rohár]

On Monday 19 December 2016 00:19:02 Geert Stappers wrote:
> On Sun, Dec 18, 2016 at 11:56:30PM +0100, Geert Stappers wrote:
> > On Sun, Dec 18, 2016 at 10:28:03PM +0100, Pali Rohár wrote:
> > > I know that they are two DHCP requests. Problem is that Debian
> > > installer try to interpret bootfile (returned by second request)
> > > as preseed file even it is not preseed file (but looks like
> > > URL).
> > 
> > The DHCP bootfilename in the second request, the d-i request,
> > should point to a preseedfile ...
> 
> And when it is not a preseedfile
> then debian-installer fails with following "red" error:
> 
>   [!!] Download debconf preconfiguration file
> 
>   Failed to process the preconfiguration file
> 
>   The installer failed to process the preconfiguration file from
> . The file may be corrupt.
> 
> 
> At least the text 'The file may be corrupt' could be changed into
> 'The file is not in expected format or simply corrupt'

Better: The file provided by DHCP bootfile option 67 is not in expected 
Debian preseed format or is simply corrupted.

> Another change might be offering a way to retrieve preseed file from
> another location. (not checked it that is allready existing)

That would be better. The whole problem is that one DHCP option is used 
for several different things and recipients (in this case debian 
installer) cannot distinguish if received option is for them or not.

-- 
Pali Rohár
pali.ro...@gmail.com


signature.asc
Description: This is a digitally signed message part.

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Geert Stappers]

Control: reassign -1 preseed
Control: severity -1 wishlist
Control: retitle -1 handling non-preseed files

On Sun, Dec 18, 2016 at 11:56:30PM +0100, Geert Stappers wrote:
> On Sun, Dec 18, 2016 at 10:28:03PM +0100, Pali Rohár wrote:
> >
> > I know that they are two DHCP requests. Problem is that Debian installer
> > try to interpret bootfile (returned by second request) as preseed file
> > even it is not preseed file (but looks like URL).
>
> The DHCP bootfilename in the second request, the d-i request,
> should point to a preseedfile ...

And when it is not a preseedfile
then debian-installer fails with following "red" error:

  [!!] Download debconf preconfiguration file

  Failed to process the preconfiguration file

  The installer failed to process the preconfiguration file from .
  The file may be corrupt.


At least the text 'The file may be corrupt' could be changed into
'The file is not in expected format or simply corrupt'


Another change might be offering a way to retrieve preseed file from another
location. (not checked it that is allready existing)


> > And I consider this as a bug.
> [1]
>
> > As more things can look like URL, e.g.
> > iPXE script which can also be in bootfile option.



> > I know what is happening there...
>
> Then show your networksniff to the world.
Please do.

> I here by volenteer to analyze .pcap files.
That still stands.


Groeten
Geert Stappers
[1] Space left blank, because it is way too easy to fuel a flame war.
--
Leven en laten leven

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Pali Rohár]

On Sunday 18 December 2016 23:56:30 Geert Stappers wrote:
> > > I think that bugreporter is missing that there are *2* DHCP
> > > requests.
> > 
> > I know that they are two DHCP requests. Problem is that Debian
> > installer try to interpret bootfile (returned by second request)
> > as preseed file even it is not preseed file (but looks like URL).
> 
> The DHCP bootfilename in the second request, the d-i request,
> should point to a preseedfile ...

For this is needed to configure DHCP in such way (that for different 
DHCP clients it returns different response). And for example qemu does 
not support it.

-- 
Pali Rohár
pali.ro...@gmail.com


signature.asc
Description: This is a digitally signed message part.

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Geert Stappers]

On Sun, Dec 18, 2016 at 10:28:03PM +0100, Pali Rohár wrote:
> On Sunday 18 December 2016 21:32:32 Geert Stappers wrote:
> > 
> > if substring (option vendor-class-identifier, 0, 3) = "d-i" {
> > filename "http://host/preseed.cfg;;
> > }
> > 
> > Note that the above example limits this filename to DHCP clients that
> > identify themselves as "d-i", so it will not affect regular DHCP
> > clients, but only the installer. You can also put the text in a
> > stanza for only one particular host to avoid preseeding all installs
> > on your network. 

> > I think that bugreporter is missing that there are *2* DHCP requests.
> 
> I know that they are two DHCP requests. Problem is that Debian installer 
> try to interpret bootfile (returned by second request) as preseed file 
> even it is not preseed file (but looks like URL).

The DHCP bootfilename in the second request, the d-i request,
should point to a preseedfile ...


> And I consider this as a bug.
[1]

> As more things can look like URL, e.g. 
> iPXE script which can also be in bootfile option.
> 
> > One from a bootROM as iPXE and the other DHCP request by
> > Debian-Installer. (Hence the "if statement" in the manual ( quoted
> > above ) that should go into DHCP server configuration)
> > 
> > On "CDROM boot" or "USBstick boot" there is the same Debian-Installer
> > code that does network configuration, which uses DHCP.
> > 
> > 
> > Advice: Networksniff (tcpdump, wireshark, tshark) the whole
> > installation. Pay special attention to the DHCP request by
> > Debian-Installer which can be recognisted on "d-i". Check that the
> } DHCP reply has the URL of the wanted preseed file.
> 
> I know what is happening there...

Then show your networksniff to the world.

I here by volenteer to analyze .pcap files.



Groeten
Geert Stappers

[1] Space left blank, because it is way too easy to fuel a flame war.
-- 
Leven en laten leven


signature.asc
Description: Digital signature

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Pali Rohár]

On Sunday 18 December 2016 21:32:32 Geert Stappers wrote:
> On Sun, Dec 18, 2016 at 08:33:47PM +0100, Pali Rohár wrote:
> > when DHCP server is configured to send bootfile option with iPXE
> > script
> 
> > then debian-installer fails with following "red" error:
>  
> 
> > Debian installer should not try to always process DHCP bootfile
> > option as it does not have to contain Debian preseed file...
> 
> That is why Debian-Installer manual has this
> 
>  from="https://www.debian.org/releases/jessie/armhf/apbs02.html.en#pr
> eseed-dhcp">
> 
> B.2.5. Using a DHCP server to specify preconfiguration files
> 
> It's also possible to use DHCP to specify a preconfiguration file to
> download from the network. DHCP allows specifying a filename.
> Normally this is a file to netboot, but if it appears to be an URL
> then installation media that support network preseeding will
> download the file from the URL and use it as a preconfiguration
> file. Here is an example of how to set it up in the dhcpd.conf for
> version 3 of the ISC DHCP server (the isc-dhcp-server Debian
> package).
> 
> if substring (option vendor-class-identifier, 0, 3) = "d-i" {
> filename "http://host/preseed.cfg;;
> }
> 
> Note that the above example limits this filename to DHCP clients that
> identify themselves as "d-i", so it will not affect regular DHCP
> clients, but only the installer. You can also put the text in a
> stanza for only one particular host to avoid preseeding all installs
> on your network. 

I have not able to find anything about it... Nor in documentation. 
Thanks for link & explanation.

> > Due to this bug booting Debian installer from iPXE setup is broken.
> > 
> :-)
> 
> Please note that there is difference
> between "bug" and "does not work I expect it to work"

Yes, understood.

> > On http://boot.oskarcz.net/ is running such iPXE setup. You can
> > easily reproduce this problem by starting iPXE in qemu and trying
> > to start installation of Debian.
> > 
> >   qemu-system-x86_64 -net nic -net
> >   user,bootfile=http://boot.oskarcz.net/
> 
> I think that bugreporter is missing that there are *2* DHCP requests.

I know that they are two DHCP requests. Problem is that Debian installer 
try to interpret bootfile (returned by second request) as preseed file 
even it is not preseed file (but looks like URL).

And I consider this as a bug. As more things can look like URL, e.g. 
iPXE script which can also be in bootfile option.

> One from a bootROM as iPXE and the other DHCP request by
> Debian-Installer. (Hence the "if statement" in the manual ( quoted
> above ) that should go into DHCP server configuration)
> 
> On "CDROM boot" or "USBstick boot" there is the same Debian-Installer
> code that does network configuration, which uses DHCP.
> 
> 
> Advice: Networksniff (tcpdump, wireshark, tshark) the whole
> installation. Pay special attention to the DHCP request by
> Debian-Installer which can be recognisted on "d-i". Check that the
> DHCP reply as the URL of the wanted preseed file. [1]

I know what is happening there...

> Please report the results to this bugreport.
> ( Keep  848...@bugs.debian.org in the loop )
> 
> 
> Groeten
> Geert Stappers
> 
> [1] If not, come back with "My setup work does not work"
>   (or other wording which saves "bug" for later.)

-- 
Pali Rohár
pali.ro...@gmail.com


signature.asc
Description: This is a digitally signed message part.

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Geert Stappers]

On Sun, Dec 18, 2016 at 08:33:47PM +0100, Pali Rohár wrote:
> 
> when DHCP server is configured to send bootfile option with iPXE script
> then debian-installer fails with following "red" error:
  
> Debian installer should not try to always process DHCP bootfile option
> as it does not have to contain Debian preseed file...

That is why Debian-Installer manual has this

https://www.debian.org/releases/jessie/armhf/apbs02.html.en#preseed-dhcp;>

B.2.5. Using a DHCP server to specify preconfiguration files

It's also possible to use DHCP to specify a preconfiguration file to
download from the network. DHCP allows specifying a filename. Normally
this is a file to netboot, but if it appears to be an URL then
installation media that support network preseeding will download the file
from the URL and use it as a preconfiguration file. Here is an example of
how to set it up in the dhcpd.conf for version 3 of the ISC DHCP server
(the isc-dhcp-server Debian package).

if substring (option vendor-class-identifier, 0, 3) = "d-i" {
filename "http://host/preseed.cfg;;
}

Note that the above example limits this filename to DHCP clients that
identify themselves as "d-i", so it will not affect regular DHCP clients,
but only the installer. You can also put the text in a stanza for only
one particular host to avoid preseeding all installs on your network.


> 
> Due to this bug booting Debian installer from iPXE setup is broken.

:-)

Please note that there is difference
between "bug" and "does not work I expect it to work"

> 
> On http://boot.oskarcz.net/ is running such iPXE setup. You can easily
> reproduce this problem by starting iPXE in qemu and trying to start
> installation of Debian.
> 
>   qemu-system-x86_64 -net nic -net user,bootfile=http://boot.oskarcz.net/
> 

I think that bugreporter is missing that there are *2* DHCP requests.

One from a bootROM as iPXE and the other DHCP request by Debian-Installer.
(Hence the "if statement" in the manual ( quoted above ) that should go
into DHCP server configuration)

On "CDROM boot" or "USBstick boot" there is the same Debian-Installer code
that does network configuration, which uses DHCP.


Advice: Networksniff (tcpdump, wireshark, tshark) the whole installation.
Pay special attention to the DHCP request by Debian-Installer which
can be recognisted on "d-i". Check that the DHCP reply as the URL
of the wanted preseed file. [1]

Please report the results to this bugreport.
( Keep  848...@bugs.debian.org in the loop )


Groeten
Geert Stappers

[1] If not, come back with "My setup work does not work"
  (or other wording which saves "bug" for later.)
-- 
Leven en laten leven


signature.asc
Description: Digital signature

Bug#848597: debian-installer: iPXE script in DHCP bootfile option is interpreted as preseed filename

[Pali Rohár]

Package: debian-installer
Severity: normal

Dear Maintainer,

when DHCP server is configured to send bootfile option with iPXE script
then debian-installer fails with following "red" error:

  [!!] Download debconf preconfiguration file

  Failed to process the preconfiguration file

  The installer failed to process the preconfiguration file from .
  The file may be corrupt.

(where  is value of that DHCP option)

It looks like debian-installer expects that DHCP bootfile option will
contains correct preseed file. And if that DHCP option contains not
Debian preseed file, then it show above "red" error message.

On the other hand, DHCP bootfile option is used for iPXE to specify boot
script too (can be detected by first shebang '#!ipxe' line).

Debian installer should not try to always process DHCP bootfile option
as it does not have to contain Debian preseed file...

Due to this bug booting Debian installer from iPXE setup is broken.

On http://boot.oskarcz.net/ is running such iPXE setup. You can easily
reproduce this problem by starting iPXE in qemu and trying to start
installation of Debian.

  qemu-system-x86_64 -net nic -net user,bootfile=http://boot.oskarcz.net/