Re: [petsc-dev] [Suggestion] Configure QOL Improvements

[Jed Brown via petsc-dev]

Matthew Knepley  writes:

> On Wed, Oct 23, 2019 at 4:59 PM Jed Brown  wrote:
>
>> Matthew Knepley  writes:
>>
>> > That is an unreliable check for Z. You would not eliminate the case
>> > where you give --with-Z, but the check fails, so Z is not available,
>> > but you do not find out until checking X or Y.
>>
>> You can verify that Z works in a fraction of a second, but building X
>> may take minutes.  Work bottom up, verify everything that is given
>> before building anything.
>>
>
> That is exactly what we do now. What are you talking about?

No, it does not.  See how libceed is built before determining that
libpng does not have its required dependency of zlib?

$ time ./configure PETSC_ARCH=ompi-png --download-libceed --download-libpng 
--with-fortran-interfaces=0
===
 Configuring PETSc to compile on your system   
===
=== 

   
  Warning: PETSC_ARCH from environment does not match command-line or name 
of script.  

  Warning: Using from command-line or name of script: ompi-png, ignoring 
environment: ompi-optg  
  
=== 

   
=== 

   
  Trying to download git://https://github.com/CEED/libceed.git for LIBCEED  

   
=== 

   
=== 

   
  Compiling libceed; this may take several minutes  

   
=== 

   
=== 

   
  Installing libceed; this may take several minutes 

   
=== 

   
TESTING: checkDependencies from 
config.packages.libpng(config/BuildSystem/config/package.py:834)
   
***
 UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for 
details):
---
Did not find package ZLIB needed by libpng.
Enable the package using --with-zlib or --download-zlib
***

38.987 real   29.069 user   8.760 sys   97.03 cpu


Note that this is most of the way through the configure before
recognizing that anything is missing.  In contrast, spack can tell me in
a fraction of a second that libpng needs zlib.  We should work out the
dependency graph first to make sure it has no unsatisfied nodes, then
check bottom-up that provided stuff works before building.

$ time spack spec libpng
Input spec

libpng

Concretized

libpng@1.6.34%gcc@9.2.0 arch=linux-arch-skylake
^zlib@1.2.11%gcc@9.2.0+optimize+pic+shared arch=linux-arch-skylake

0.407 real   0.358 user   0.034 sys   96.44 cpu

Re: [petsc-dev] [Suggestion] Configure QOL Improvements

[Smith, Barry F. via petsc-dev]

https://www.youtube.com/watch?v=NVopDink4uQ

https://gitlab.com/petsc/petsc/merge_requests/2207


> On Oct 23, 2019, at 8:34 PM, Matthew Knepley via petsc-dev 
>  wrote:
> 
> On Wed, Oct 23, 2019 at 5:08 PM Faibussowitsch, Jacob 
>  wrote:
> I think Jed is referring to the fact that configure appears to do all of 
> these checks after doing the compiler,
> 
> How would you verify that Z works without the compiler? Again, this whole 
> discussion is not listening.
> 
>   Matt
>  
> MPI, MathEnv, etc which take a long time. The change would be to check 
> missing external library dependencies before even doing the compiler checks.
> 
> Best,
> 
> Jacob
> 
>> On Oct 23, 2019, at 4:03 PM, Matthew Knepley  wrote:
>> 
>> On Wed, Oct 23, 2019 at 4:59 PM Jed Brown  wrote:
>> Matthew Knepley  writes:
>> 
>> > That is an unreliable check for Z. You would not eliminate the case
>> > where you give --with-Z, but the check fails, so Z is not available,
>> > but you do not find out until checking X or Y.
>> 
>> You can verify that Z works in a fraction of a second, but building X
>> may take minutes.  Work bottom up, verify everything that is given
>> before building anything.
>> 
>> That is exactly what we do now. What are you talking about?
>> 
>>   Matt
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments 
>> is infinitely more interesting than any results to which their experiments 
>> lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] [Suggestion] Configure QOL Improvements

[Faibussowitsch, Jacob via petsc-dev]

I think Jed is referring to the fact that configure appears to do all of these 
checks after doing the compiler, MPI, MathEnv, etc which take a long time. The 
change would be to check missing external library dependencies before even 
doing the compiler checks.

Best,

Jacob

On Oct 23, 2019, at 4:03 PM, Matthew Knepley 
mailto:knep...@gmail.com>> wrote:

On Wed, Oct 23, 2019 at 4:59 PM Jed Brown 
mailto:j...@jedbrown.org>> wrote:
Matthew Knepley mailto:knep...@gmail.com>> writes:

> That is an unreliable check for Z. You would not eliminate the case
> where you give --with-Z, but the check fails, so Z is not available,
> but you do not find out until checking X or Y.

You can verify that Z works in a fraction of a second, but building X
may take minutes.  Work bottom up, verify everything that is given
before building anything.

That is exactly what we do now. What are you talking about?

  Matt

--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] [Suggestion] Configure QOL Improvements

[Jed Brown via petsc-dev]

Matthew Knepley  writes:

> That is an unreliable check for Z. You would not eliminate the case
> where you give --with-Z, but the check fails, so Z is not available,
> but you do not find out until checking X or Y.

You can verify that Z works in a fraction of a second, but building X
may take minutes.  Work bottom up, verify everything that is given
before building anything.

Re: [petsc-dev] [Suggestion] Configure QOL Improvements

[Jed Brown via petsc-dev]

Matthew Knepley  writes:

> On Wed, Oct 23, 2019 at 2:27 PM Jed Brown  wrote:
>
>> Matthew Knepley via petsc-dev  writes:
>>
>> > On Wed, Oct 23, 2019 at 11:24 AM Faibussowitsch, Jacob via petsc-dev <
>> > petsc-dev@mcs.anl.gov> wrote:
>> >> As I am largely unfamiliar with the internals of the configure process,
>> >> this is potentially more of an involved change than I am imagining,
>> given
>> >> that many libraries likely have many small dependencies and hooks which
>> >> have to be set throughout the configuration process, and so its possible
>> >> not everything could be skipped.
>> >>
>> >
>> > We had this many years ago. It was removed because the benefits did not
>> > outweigh the costs.
>>
>> I don't know if it's still the case, but it should be possible to run
>> non-interactively (like apt-get -y).  My bigger complaint is that
>> missing dependencies aren't resolved in the first couple seconds.
>>
>
> How do you know that you actually have something until you actually
> run the tests? This is the classic misconception of pkg-config, "I'll
> just believe the user", which generated 99% of user mail over the
> first 20 years of PETSc.

All other build systems get this right.  You're asking to build X and Y,
where Y depends on Z, and there is no --download-Z or --with-Z.  You
shouldn't need to build X before noticing that Z is unavailable.

Re: [petsc-dev] [Suggestion] Configure QOL Improvements

[Paul T. Bauman via petsc-dev]

On Wed, Oct 23, 2019 at 1:27 PM Jed Brown via petsc-dev <
petsc-dev@mcs.anl.gov> wrote:

>  My bigger complaint is that
> missing dependencies aren't resolved in the first couple seconds.
>

>From a user point-of-view, I would echo this. I don't mind having to go
back and add an option, but I would like to be notified ASAP rather than
after 10-15 mins of building packages.

Re: [petsc-dev] [Suggestion] Configure QOL Improvements

[Jed Brown via petsc-dev]

Matthew Knepley via petsc-dev  writes:

> On Wed, Oct 23, 2019 at 11:24 AM Faibussowitsch, Jacob via petsc-dev <
> petsc-dev@mcs.anl.gov> wrote:
>> As I am largely unfamiliar with the internals of the configure process,
>> this is potentially more of an involved change than I am imagining, given
>> that many libraries likely have many small dependencies and hooks which
>> have to be set throughout the configuration process, and so its possible
>> not everything could be skipped.
>>
>
> We had this many years ago. It was removed because the benefits did not
> outweigh the costs.

I don't know if it's still the case, but it should be possible to run
non-interactively (like apt-get -y).  My bigger complaint is that
missing dependencies aren't resolved in the first couple seconds.

Re: [petsc-dev] [Suggestion] Configure QOL Improvements

[Matthew Knepley via petsc-dev]

On Wed, Oct 23, 2019 at 11:24 AM Faibussowitsch, Jacob via petsc-dev <
petsc-dev@mcs.anl.gov> wrote:

> Hello All,
>
> How feasible would it be to accept user input during the configure process
> when installing a new package using —download-package when the dependency
> is missing? For example say you are configuring with the following options:
>
> ./configure —download-pragmatic
>
> Which depends on Metis, so halfway through configure you will receive the
> following error:
>
>
> ***
>  UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for
> details):
>
> ---
> Did not find package METIS needed by pragmatic.
> Enable the package using --with-metis or --download-metis
>
> ***
>
> Why can’t there instead be a prompt that states “[Package] depends on
> [Dependency], would you like to also download [Dependency] (y , n)?”
> whenever PETSc detects that you are missing a dependency. Configure would
> then restart from the appropriate section (wherever that may be) in the
> configuration process instead of all the way from the beginning. Use case
> for this would be for configuring on systems such as Theta where the
> configure process takes roughly 13.772 billion years to complete. Having to
> restart from the beginning every time is tedious.
>
> As I am largely unfamiliar with the internals of the configure process,
> this is potentially more of an involved change than I am imagining, given
> that many libraries likely have many small dependencies and hooks which
> have to be set throughout the configuration process, and so its possible
> not everything could be skipped.
>

We had this many years ago. It was removed because the benefits did not
outweigh the costs.

  Thanks,

Matt


> Best,
>
> Jacob
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/