Re: monolithic-concatenate-source-op misses a few dependencies

[Faré]

1- Regarding eval-when, see my livejournal post:
https://fare.livejournal.com/146698.html

2- features in build files are a very bad idea, inherited from the dark
times of using REQUIRE to load code (not so portably). Please don't use
them. Avoid using putting anything but build information in build files.
Use :read-file-form or :read-file-line for :version, use
:defsystem-depends-on for code.

3- Send patches to chipz, drakma, usocket. Users of bazel will thank you,
too.

—♯ƒ • François-René ÐVB Rideau •Reflection•
http://fare.tunes.org
Experience is the worst teacher.  It always gives the test first and
the instruction afterward.






On Oct 14, 2017 16:44, "Ben Vulpes"  wrote:

> Fare et al.,
>
> The Postmodern/usocket problem I encountered was in fact due to
> cl-postgres not setting a feature flag in an :execute context:
> https://github.com/marijnh/Postmodern/blob/57dc8cb4acc4599ae
> e757e8f81999a0fa83c7111/cl-postgres/features.lisp#L8
> . To digress from the list topic for half a second, this confuses me,
> as that code refused to execute until I patched cl-postgres at that
> point to also eval-when in :execute, even when loading the
> concatenated sources with sbcl ... --load full-concatenation.lisp
>
> More relevant to the list, other observations include: chipz mutates
> *features* in :chipz-system, a package defined in chipz.asd, which
> causes problems because the .asd files are not concatenated with the
> sources (not suggesting this is something asdf should even concern
> itself with); and both drakma and usocket call (asdf:component-version
> (asdf:find-system ...)) at read time, which fails to find the package
> in question when loading the concatenated source on a machine without
> those packages in *central-registry*, because (I believe) as noted
> above, monolithic-concatenate-source-op doesn't include system .asd
> files. Again, I'm not suggesting that this behavior change.
>
> Yours,
> Ben
>
> On Sat, Oct 14, 2017 at 11:52 AM, Faré  wrote:
> > On Sat, Oct 14, 2017 at 1:17 PM, Robert Goldman 
> wrote:
> >> Will you please clarify for my benefit, since I don't actually use any
> of
> >> the image operations.
> >>
> >> Is the problem that somewhere in the process of loading Postmodern, or
> one
> >> of its dependencies, some bit of code invokes REQUIRE? Or is this an
> issue
> >> with ASDF's REQUIRE-SYSTEM.
> >>
> > monolithic-concatenate-source-op is supposed to create a source file
> > that has "all the (transitive) source code required to load a system".
> > Problem is, it is naive in that the current implementation believes
> > there are only source files to concatenate. And so it doesn't handle
> > (:require "foo") dependencies which would require to generate a
> > temporary "source file" that contains (require "FOO") in it — or
> > better a refactoring such that instead of source files, you use
> > arbitrary arguments to vomit-output, and for require dependencies you
> > use a function that outputs that instead of a pathname  (and wrap a
> > map () 'vomit-output in a with-output-file).
> >
> >> If it's the former, then I believe this is simply a bad implementation
> in
> >> the relevant system.
> >>
> >> If it's the latter, we should do something about it. However, I believe
> that
> >> REQUIRE-SYSTEM, despite its name, doesn't actually use REQUIRE, instead
> it
> >> "acts like require."
> >>
> > Well, there are unhappily two things called "require-system": the
> > class that backs the (:require "foo") dependencies, and the function
> > that is now deprecated because it didn't have a good composable
> > meaning in presence of multiple build phases. The one that matters
> > here is the former.
> >
> > —♯ƒ • François-René ÐVB Rideau •Reflection•
> http://fare.tunes.org
> > All problems in computer science can be solved by another level of
> indirection
> > — David Wheeler
> > Almost all programming can be viewed as an exercise in caching
> > — Terje Mathisen, well-known programming optimization guru
>
>
>
> --
> Ben Vulpes
> 503-313-5838
> MAVN
>

Re: monolithic-concatenate-source-op misses a few dependencies

[Ben Vulpes]

Fare et al.,

The Postmodern/usocket problem I encountered was in fact due to
cl-postgres not setting a feature flag in an :execute context:
https://github.com/marijnh/Postmodern/blob/57dc8cb4acc4599aee757e8f81999a0fa83c7111/cl-postgres/features.lisp#L8
. To digress from the list topic for half a second, this confuses me,
as that code refused to execute until I patched cl-postgres at that
point to also eval-when in :execute, even when loading the
concatenated sources with sbcl ... --load full-concatenation.lisp

More relevant to the list, other observations include: chipz mutates
*features* in :chipz-system, a package defined in chipz.asd, which
causes problems because the .asd files are not concatenated with the
sources (not suggesting this is something asdf should even concern
itself with); and both drakma and usocket call (asdf:component-version
(asdf:find-system ...)) at read time, which fails to find the package
in question when loading the concatenated source on a machine without
those packages in *central-registry*, because (I believe) as noted
above, monolithic-concatenate-source-op doesn't include system .asd
files. Again, I'm not suggesting that this behavior change.

Yours,
Ben

On Sat, Oct 14, 2017 at 11:52 AM, Faré  wrote:
> On Sat, Oct 14, 2017 at 1:17 PM, Robert Goldman  wrote:
>> Will you please clarify for my benefit, since I don't actually use any of
>> the image operations.
>>
>> Is the problem that somewhere in the process of loading Postmodern, or one
>> of its dependencies, some bit of code invokes REQUIRE? Or is this an issue
>> with ASDF's REQUIRE-SYSTEM.
>>
> monolithic-concatenate-source-op is supposed to create a source file
> that has "all the (transitive) source code required to load a system".
> Problem is, it is naive in that the current implementation believes
> there are only source files to concatenate. And so it doesn't handle
> (:require "foo") dependencies which would require to generate a
> temporary "source file" that contains (require "FOO") in it — or
> better a refactoring such that instead of source files, you use
> arbitrary arguments to vomit-output, and for require dependencies you
> use a function that outputs that instead of a pathname  (and wrap a
> map () 'vomit-output in a with-output-file).
>
>> If it's the former, then I believe this is simply a bad implementation in
>> the relevant system.
>>
>> If it's the latter, we should do something about it. However, I believe that
>> REQUIRE-SYSTEM, despite its name, doesn't actually use REQUIRE, instead it
>> "acts like require."
>>
> Well, there are unhappily two things called "require-system": the
> class that backs the (:require "foo") dependencies, and the function
> that is now deprecated because it didn't have a good composable
> meaning in presence of multiple build phases. The one that matters
> here is the former.
>
> —♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
> All problems in computer science can be solved by another level of indirection
> — David Wheeler
> Almost all programming can be viewed as an exercise in caching
> — Terje Mathisen, well-known programming optimization guru



-- 
Ben Vulpes
503-313-5838
MAVN

Re: monolithic-concatenate-source-op misses a few dependencies

[Faré]

On Sat, Oct 14, 2017 at 1:17 PM, Robert Goldman  wrote:
> Will you please clarify for my benefit, since I don't actually use any of
> the image operations.
>
> Is the problem that somewhere in the process of loading Postmodern, or one
> of its dependencies, some bit of code invokes REQUIRE? Or is this an issue
> with ASDF's REQUIRE-SYSTEM.
>
monolithic-concatenate-source-op is supposed to create a source file
that has "all the (transitive) source code required to load a system".
Problem is, it is naive in that the current implementation believes
there are only source files to concatenate. And so it doesn't handle
(:require "foo") dependencies which would require to generate a
temporary "source file" that contains (require "FOO") in it — or
better a refactoring such that instead of source files, you use
arbitrary arguments to vomit-output, and for require dependencies you
use a function that outputs that instead of a pathname  (and wrap a
map () 'vomit-output in a with-output-file).

> If it's the former, then I believe this is simply a bad implementation in
> the relevant system.
>
> If it's the latter, we should do something about it. However, I believe that
> REQUIRE-SYSTEM, despite its name, doesn't actually use REQUIRE, instead it
> "acts like require."
>
Well, there are unhappily two things called "require-system": the
class that backs the (:require "foo") dependencies, and the function
that is now deprecated because it didn't have a good composable
meaning in presence of multiple build phases. The one that matters
here is the former.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
All problems in computer science can be solved by another level of indirection
— David Wheeler
Almost all programming can be viewed as an exercise in caching
— Terje Mathisen, well-known programming optimization guru

Re: monolithic-concatenate-source-op misses a few dependencies

[Robert Goldman]

Will you please clarify for my benefit, since I don't actually use any 
of the image operations.


Is the problem that somewhere in the process of loading Postmodern, or 
one of its dependencies, some bit of code invokes `REQUIRE`?  Or is this 
an issue with ASDF's `REQUIRE-SYSTEM`.


If it's the former, then I believe this is simply a bad implementation 
in the relevant system.


If it's the latter, we should do something about it.  However, I believe 
that `REQUIRE-SYSTEM`, despite its name, doesn't actually use `REQUIRE`, 
instead it "acts like `require`."


Thanks,
R

On 14 Oct 2017, at 11:45, Faré wrote:


Dear Ben,

sorry I won't be developing this feature, but I'll happily merge your
patch if you do. "Just" add support for dumping source code for a
(:require ...) dependency.

—♯ƒ • François-René ÐVB Rideau •Reflection• 
http://fare.tunes.org
Guns & bullets don't kill people — blood loss and organ damage kills 
people.



On Thu, Oct 12, 2017 at 4:07 PM, Ben Vulpes  wrote:

On SBCL 1.3.11, when producing a monolithic source concatenation with
the library "Postmodern", asdf produces a file that needs a Lisp 
image
to need manual calls to (require :usocket) and (require :md5) in 
order

to load completely.

Given:

concatenatrix.asd as:

(asdf:defsystem :concatenatrix
  :build-operation monolithic-concatenate-source-op
  :build-pathname "build/full-concatenation"
  :depends-on (:postmodern)
  :components
  ((:file "concatenatrix")))

concatenatrix.lisp as:

(defpackage :concatenatrix
  (:use :cl :postmodern))
(in-package :concatenatrix)

(defun wat (it)
  (format t "~A~%" it))

Concatenated sources produced with:

(asdf:make :concatenatrix)

Loading tested with:

sbcl --noinform --disable-debugger --load 
build/full-concatenation.lisp


Produces:

Unhandled SB-C::INPUT-ERROR-IN-LOAD in thread #"main

thread" RUNNING
 {10027FE873}>:
  READ error during LOAD:

Package SB-ROTATE-BYTE does not exist.

  Line: 221, Column: 29, File-Position: 8706

  Stream: #


Backtrace for: #
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK # #)
1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK*
#)
2: (INVOKE-DEBUGGER #)
3: (ERROR #)
4: (SB-C:COMPILER-ERROR SB-C::INPUT-ERROR-IN-LOAD :CONDITION
# :STREAM #)
5: (SB-C::%DO-FORMS-FROM-INFO # #
SB-C::INPUT-ERROR-IN-LOAD)
6: (SB-INT:LOAD-AS-SOURCE # :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
7: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) 
#for "file 
/home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"

{1003CFCA03}> NIL)
8: (LOAD #P"build/full-concatenation.lisp" :VERBOSE NIL :PRINT NIL
:IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :DEFAULT)
9: (SB-IMPL::PROCESS-EVAL/LOAD-OPTIONS ((:LOAD .
"build/full-concatenation.lisp")))
10: (SB-IMPL::TOPLEVEL-INIT)
11: ((FLET #:WITHOUT-INTERRUPTS-BODY-90 :IN 
SB-EXT:SAVE-LISP-AND-DIE))

12: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))

There is another complaint about usocket, which is confusing, as
cl-postgres explicitly doesn't require usocket on sbcl by my read (
http://marijnhaverbeke.nl/git/?p=postmodern;a=blob;f=cl-postgres.asd;h=683edf0f131a4ebe172b44425f597d7f67656e70;hb=HEAD#l16
).

Loading is resolved by requiring both libraries in question, as:

sbcl --eval "(require :md5)" --eval "(require :usocket)" --load
build/full-concatenation.lisp

Yours,
Benjamin

Re: monolithic-concatenate-source-op misses a few dependencies

[Faré]

Dear Ben,

sorry I won't be developing this feature, but I'll happily merge your
patch if you do. "Just" add support for dumping source code for a
(:require ...) dependency.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Guns & bullets don't kill people — blood loss and organ damage kills people.


On Thu, Oct 12, 2017 at 4:07 PM, Ben Vulpes  wrote:
> On SBCL 1.3.11, when producing a monolithic source concatenation with
> the library "Postmodern", asdf produces a file that needs a Lisp image
> to need manual calls to (require :usocket) and (require :md5) in order
> to load completely.
>
> Given:
>
> concatenatrix.asd as:
>
> (asdf:defsystem :concatenatrix
>   :build-operation monolithic-concatenate-source-op
>   :build-pathname "build/full-concatenation"
>   :depends-on (:postmodern)
>   :components
>   ((:file "concatenatrix")))
>
> concatenatrix.lisp as:
>
> (defpackage :concatenatrix
>   (:use :cl :postmodern))
> (in-package :concatenatrix)
>
> (defun wat (it)
>   (format t "~A~%" it))
>
> Concatenated sources produced with:
>
> (asdf:make :concatenatrix)
>
> Loading tested with:
>
> sbcl --noinform --disable-debugger --load build/full-concatenation.lisp
>
> Produces:
>
> Unhandled SB-C::INPUT-ERROR-IN-LOAD in thread # thread" RUNNING
>  {10027FE873}>:
>   READ error during LOAD:
>
> Package SB-ROTATE-BYTE does not exist.
>
>   Line: 221, Column: 29, File-Position: 8706
>
>   Stream: # /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"
> {1003CFCA03}>
>
>
> Backtrace for: #
> 0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK # {1004014AC3}> #)
> 1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK*
> #)
> 2: (INVOKE-DEBUGGER #)
> 3: (ERROR #)
> 4: (SB-C:COMPILER-ERROR SB-C::INPUT-ERROR-IN-LOAD :CONDITION
> # {10040149E3}> :STREAM # /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"
> {1003CFCA03}>)
> 5: (SB-C::%DO-FORMS-FROM-INFO # :CURRENT-INDEX ) :IN SB-INT:LOAD-AS-SOURCE)
> {1003D004FB}> #
> SB-C::INPUT-ERROR-IN-LOAD)
> 6: (SB-INT:LOAD-AS-SOURCE # /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"
> {1003CFCA03}> :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
> 7: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) # for "file 
> /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"
> {1003CFCA03}> NIL)
> 8: (LOAD #P"build/full-concatenation.lisp" :VERBOSE NIL :PRINT NIL
> :IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :DEFAULT)
> 9: (SB-IMPL::PROCESS-EVAL/LOAD-OPTIONS ((:LOAD .
> "build/full-concatenation.lisp")))
> 10: (SB-IMPL::TOPLEVEL-INIT)
> 11: ((FLET #:WITHOUT-INTERRUPTS-BODY-90 :IN SB-EXT:SAVE-LISP-AND-DIE))
> 12: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))
>
> There is another complaint about usocket, which is confusing, as
> cl-postgres explicitly doesn't require usocket on sbcl by my read (
> http://marijnhaverbeke.nl/git/?p=postmodern;a=blob;f=cl-postgres.asd;h=683edf0f131a4ebe172b44425f597d7f67656e70;hb=HEAD#l16
> ).
>
> Loading is resolved by requiring both libraries in question, as:
>
> sbcl --eval "(require :md5)" --eval "(require :usocket)" --load
> build/full-concatenation.lisp
>
> Yours,
> Benjamin
>

monolithic-concatenate-source-op misses a few dependencies

[Ben Vulpes]

On SBCL 1.3.11, when producing a monolithic source concatenation with
the library "Postmodern", asdf produces a file that needs a Lisp image
to need manual calls to (require :usocket) and (require :md5) in order
to load completely.

Given:

concatenatrix.asd as:

(asdf:defsystem :concatenatrix
  :build-operation monolithic-concatenate-source-op
  :build-pathname "build/full-concatenation"
  :depends-on (:postmodern)
  :components
  ((:file "concatenatrix")))

concatenatrix.lisp as:

(defpackage :concatenatrix
  (:use :cl :postmodern))
(in-package :concatenatrix)

(defun wat (it)
  (format t "~A~%" it))

Concatenated sources produced with:

(asdf:make :concatenatrix)

Loading tested with:

sbcl --noinform --disable-debugger --load build/full-concatenation.lisp

Produces:

Unhandled SB-C::INPUT-ERROR-IN-LOAD in thread #:
  READ error during LOAD:

Package SB-ROTATE-BYTE does not exist.

  Line: 221, Column: 29, File-Position: 8706

  Stream: #


Backtrace for: #
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK # #)
1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK*
#)
2: (INVOKE-DEBUGGER #)
3: (ERROR #)
4: (SB-C:COMPILER-ERROR SB-C::INPUT-ERROR-IN-LOAD :CONDITION
# :STREAM #)
5: (SB-C::%DO-FORMS-FROM-INFO # #
SB-C::INPUT-ERROR-IN-LOAD)
6: (SB-INT:LOAD-AS-SOURCE # :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
7: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) # NIL)
8: (LOAD #P"build/full-concatenation.lisp" :VERBOSE NIL :PRINT NIL
:IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :DEFAULT)
9: (SB-IMPL::PROCESS-EVAL/LOAD-OPTIONS ((:LOAD .
"build/full-concatenation.lisp")))
10: (SB-IMPL::TOPLEVEL-INIT)
11: ((FLET #:WITHOUT-INTERRUPTS-BODY-90 :IN SB-EXT:SAVE-LISP-AND-DIE))
12: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))

There is another complaint about usocket, which is confusing, as
cl-postgres explicitly doesn't require usocket on sbcl by my read (
http://marijnhaverbeke.nl/git/?p=postmodern;a=blob;f=cl-postgres.asd;h=683edf0f131a4ebe172b44425f597d7f67656e70;hb=HEAD#l16
).

Loading is resolved by requiring both libraries in question, as:

sbcl --eval "(require :md5)" --eval "(require :usocket)" --load
build/full-concatenation.lisp

Yours,
Benjamin