Re: maintaining non-Lisp systems with ASDF

[Robert Goldman]

On 9/23/15 Sep 23 -3:55 PM, Robert Dodier wrote:
> Hey everybody,
> 
> I've made some more progress with a Maxima extension for ASDF
> (attached). At this point it works pretty much as expected, for the
> simple examples I've tried. I think you should be able to use in
> Maxima like this: load("maxima_asdf.lisp"); asdf_load("foo"); to load
> foo.asd, which may contain :maxima-file components. asdf_load
> translates :maxima-files and compiles the resulting Lisp, while
> asdf_load_source just interprets the :maxima-files.
> 
> At this point the only thing I wish I could change (and this is a
> minor item) is that output file paths are something like
> $HOME/.cache/common-lisp/.
> I'm constructing the /... part of it. I wonder if it's
> possible to tell ASDF it doesn't need to put in the
>  part.
> 
> Thanks for your help, and if you have any comments, I would be
> interested to hear about it.

Robert, please consider writing up your experience, even as a short
bulleted list of items, with information about what was *missing* from
the documentation.  That will help us make this easier for the next person.

thanks!
r

Re: maintaining non-Lisp systems with ASDF

[Robert P. Goldman]

That's what :defsystem-depends-on is for.

Please let me know if you don't find it adequately documented in the manual...

Sent from my iPad

> On Sep 24, 2015, at 19:41, Robert Dodier  wrote:
> 
>> On Wed, Sep 23, 2015 at 2:39 PM, Faré  wrote:
>> 
>> What you really need to do is:
> 
> Thanks, I will take a look at that. It is working well enough for the
> moment, but I will circle back later to polish it up.
> 
> I have a question now about how to ensure that the :maxima-file
> component type is defined before it is used. Let's say I have foo.asd
> which has :maxima-file components. Ideally I would like to simply say
> (asdf:load-system :foo) without having to load the maxima-file
> definition before. It occurred to me to put (defsystem foo :depends-on
> ("maxima-file") ...) and move the :maxima-file stuff into another
> system. But that doesn't work, of course, because :maxima-file is not
> yet defined before the :foo system is parsed, and I get an error about
> :maxima-file being unrecognized.
> 
> I can put (load-system :maxima-file) before (defsystem foo ...) and
> then it works OK. I'm happy that it works, but it seems a little messy
> to me -- there are now two kinds of dependencies, one kind that has to
> be loaded before the defsystem, and the other after ...
> 
> It's not a big deal, but can anyone think of a way to move the
> maxima-file dependency back into the :depends-on list?
> 
> Thanks for your help,
> 
> Robert Dodier
> 

Re: maintaining non-Lisp systems with ASDF

[Robert Dodier]

On Wed, Sep 23, 2015 at 2:39 PM, Faré  wrote:

> What you really need to do is:

Thanks, I will take a look at that. It is working well enough for the
moment, but I will circle back later to polish it up.

I have a question now about how to ensure that the :maxima-file
component type is defined before it is used. Let's say I have foo.asd
which has :maxima-file components. Ideally I would like to simply say
(asdf:load-system :foo) without having to load the maxima-file
definition before. It occurred to me to put (defsystem foo :depends-on
("maxima-file") ...) and move the :maxima-file stuff into another
system. But that doesn't work, of course, because :maxima-file is not
yet defined before the :foo system is parsed, and I get an error about
:maxima-file being unrecognized.

I can put (load-system :maxima-file) before (defsystem foo ...) and
then it works OK. I'm happy that it works, but it seems a little messy
to me -- there are now two kinds of dependencies, one kind that has to
be loaded before the defsystem, and the other after ...

It's not a big deal, but can anyone think of a way to move the
maxima-file dependency back into the :depends-on list?

Thanks for your help,

Robert Dodier

Re: maintaining non-Lisp systems with ASDF

[Faré]

Dear Robert,

> I've made some more progress with a Maxima extension for ASDF
> (attached).

This looks much more like it. Congrats!

> At this point the only thing I wish I could change (and this is a
> minor item) is that output file paths are something like
> $HOME/.cache/common-lisp/.
> I'm constructing the /... part of it. I wonder if it's
> possible to tell ASDF it doesn't need to put in the
>  part.
>
You can do your own arbitrary translations in your output-files, and return
(values (list ...) t)

I recommend you "just" prepend your maxima-version to the component's
relative directory, or append it to its directory always, and let the
usual translations take your from there. That would fit the ASDF cache
model better, and thus be more clash-proof. These files are hidden
from users, anyway. Also, on many implementations, lisp compilation
has more than one output.

What you really need to do is:
(1) inherit from cl-source-file, not source-file, so you can reuse its
compile-op
(2) have compile-op depend on a new operation translate-op that does
just the translation step.
(3) Have a generic method:
(defmethod output-files ((o translate-op) (c maxima-file))
  (list (merge-pathnames*
   (make-pathname :directory `(:relative ,(strcat "maxima-"
maxima::*autoconf-version*)) :type "lisp")
   (component-pathname c

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Anarchy is order, government is civil war. — Anselme Bellegarrigue

Re: maintaining non-Lisp systems with ASDF

[Robert Dodier]

Hey everybody,

I've made some more progress with a Maxima extension for ASDF
(attached). At this point it works pretty much as expected, for the
simple examples I've tried. I think you should be able to use in
Maxima like this: load("maxima_asdf.lisp"); asdf_load("foo"); to load
foo.asd, which may contain :maxima-file components. asdf_load
translates :maxima-files and compiles the resulting Lisp, while
asdf_load_source just interprets the :maxima-files.

At this point the only thing I wish I could change (and this is a
minor item) is that output file paths are something like
$HOME/.cache/common-lisp/.
I'm constructing the /... part of it. I wonder if it's
possible to tell ASDF it doesn't need to put in the
 part.

Thanks for your help, and if you have any comments, I would be
interested to hear about it.

best

Robert Dodier


maxima_asdf.lisp
Description: Binary data

Re: maintaining non-Lisp systems with ASDF

[Faré]

On Mon, Sep 21, 2015 at 7:15 PM, Robert Dodier  wrote:
> Thanks a lot to everybody for the advice. I think I've made some
> progress. I've attached a patch showing the initial changes I've
> made, and with this much I can load a trivial system into Maxima.
>
Do NOT patch ASDF, unless there's a bug.
Create an extension and use defsystem-depends-on,
as documented in the manual.

> I'm working with asdf.lisp 2.26 which is packaged with Quicklisp,
> since the larger goal is to use Quicklisp to download and install
> Maxima packages.
>
Do NOT use ASDF 2.26.
It is not supported, and internals are somewhat incompatible.
ASDF 3 is available on every single maintained implementation.

> Incidentally I'm working with Clisp 2.49.
>
CLISP isn't currently maintained.
Unless you need it due for portability reasons,
or love it enough to become its new maintainer,
I suggest to use another implementation.
If you *must* use CLISP, use our script tools/install-asdf.lisp
to overwrite its ASDF with the latest ASDF release (currently 3.1.5).

> I've run into some strangeness, maybe someone can explain
> what's going on.
>
> (1) (asdf:load-system :foo) seems to always cause files to
> be compiled. That's kind of puzzling.
>
This typically means that your input-files or output-files are wrong.
Your code is not even wrong: it's missing a method on output-files.
ASDF 2.26 has also deep unfixable bugs regarding incremental build,
and it is unsupported. Use ASDF 3.

> (2) The first time asdf.lisp is loaded, Quicklisp writes asdf.fas
> into its own cache. The second time, I get an error about ASDF:OOS.
> Any ideas? Here's the error:
>
Use ASDF 3.

> *** - EVAL: undefined function ASDF:OOS
> The following restarts are available:
> USE-VALUE  :R1  Input a value to be used instead of
> (FDEFINITION 'ASDF:OOS).
> RETRY  :R2  Retry
> STORE-VALUE:R3  Input a new value for (FDEFINITION 'ASDF:OOS).
> SKIP   :R4  skip (LET # #)
> RETRY  :R5  retry (LET # #)
> STOP   :R6  stop loading file /home/robert/quicklisp/setup.lisp
> SKIP   :R7  skip (LET # #)
> RETRY  :R8  retry (LET # #)
> STOP   :R9  stop loading file /home/robert/.clisprc.lisp
> Break 1 QL-SETUP[4]>
>
This is completely useless.
Give us a backtrace. From CCL or SBCL, not CLISP.
While compiling ASDF and/or your extension with extra debugging information.

Also, don't use (component-pathname c) in your perform method.
Use (first (input-files o c)), so users can suitable modify or extend
your methods.

Please come back after you've read the manual (notably section 7)
and actually tried to apply the recipes there.
And/or at least cargo culted from an existing extension.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
A politician divides mankind into two classes: tools and enemies. — Nietzsche

Re: maintaining non-Lisp systems with ASDF

[Robert Dodier]

Thanks a lot to everybody for the advice. I think I've made some
progress. I've attached a patch showing the initial changes I've
made, and with this much I can load a trivial system into Maxima.

I'm working with asdf.lisp 2.26 which is packaged with Quicklisp,
since the larger goal is to use Quicklisp to download and install
Maxima packages.

Incidentally I'm working with Clisp 2.49.

I've run into some strangeness, maybe someone can explain
what's going on.

(1) (asdf:load-system :foo) seems to always cause files to
be compiled. That's kind of puzzling.

(2) The first time asdf.lisp is loaded, Quicklisp writes asdf.fas
into its own cache. The second time, I get an error about ASDF:OOS.
Any ideas? Here's the error:

$ maxima
;; Loading file /home/robert/.clisprc.lisp ...
;;  Loading file /home/robert/quicklisp/setup.lisp ...
HELLO FROM QUICKLISP/ASDF.LISP !!

*** - EVAL: undefined function ASDF:OOS
The following restarts are available:
USE-VALUE  :R1  Input a value to be used instead of
(FDEFINITION 'ASDF:OOS).
RETRY  :R2  Retry
STORE-VALUE:R3  Input a new value for (FDEFINITION 'ASDF:OOS).
SKIP   :R4  skip (LET # #)
RETRY  :R5  retry (LET # #)
STOP   :R6  stop loading file /home/robert/quicklisp/setup.lisp
SKIP   :R7  skip (LET # #)
RETRY  :R8  retry (LET # #)
STOP   :R9  stop loading file /home/robert/.clisprc.lisp
Break 1 QL-SETUP[4]>

Thanks again,

Robert

PS. Here's the example foo.asd I've been working with.

(defsystem foo
  :name "foo"
  :version "2015.09.21"
  :maintainer "Robert Dodier"
  :author "Robert Dodier"
  :licence "GPL"
  :description "Foo, Bar, and Baz"
  :long-description "Apply Foo's algorithm to Bar, yielding Baz"

  :components
((:maxima-source-file "foo1")
 (:maxima-source-file "foo2")))


tmp-asdf-maxima-wip-2015-09-21-patch
Description: Binary data

Re: maintaining non-Lisp systems with ASDF

[Robert Goldman]

On 9/21/15 Sep 21 -6:15 PM, Robert Dodier wrote:
> Thanks a lot to everybody for the advice. I think I've made some
> progress. I've attached a patch showing the initial changes I've
> made, and with this much I can load a trivial system into Maxima.
> 
> I'm working with asdf.lisp 2.26 which is packaged with Quicklisp,
> since the larger goal is to use Quicklisp to download and install
> Maxima packages.
> 
> Incidentally I'm working with Clisp 2.49.
> 
> I've run into some strangeness, maybe someone can explain
> what's going on.
> 
> (1) (asdf:load-system :foo) seems to always cause files to
> be compiled. That's kind of puzzling.

I don't know if this is why, but I see that there are no dependencies in
your system.  Was that intentional? Did you expect DEFSYSTEM to default
to :SERIAL T.

If that causes this problem, though, that would be an ASDF bug.
> 
> (2) The first time asdf.lisp is loaded, Quicklisp writes asdf.fas
> into its own cache. The second time, I get an error about ASDF:OOS.
> Any ideas? Here's the error:

This isn't enough for me to help, since it looks like "maxima" is some
kind of script that does something.  I don't know what. Also, I don't
know what's in your .clisprc.lisp...

At the bare minimum, we'd need to see a backtrace from the debugger.

cheers,
r

> 
> $ maxima
> ;; Loading file /home/robert/.clisprc.lisp ...
> ;;  Loading file /home/robert/quicklisp/setup.lisp ...
> HELLO FROM QUICKLISP/ASDF.LISP !!
> 
> *** - EVAL: undefined function ASDF:OOS
> The following restarts are available:
> USE-VALUE  :R1  Input a value to be used instead of
> (FDEFINITION 'ASDF:OOS).
> RETRY  :R2  Retry
> STORE-VALUE:R3  Input a new value for (FDEFINITION 'ASDF:OOS).
> SKIP   :R4  skip (LET # #)
> RETRY  :R5  retry (LET # #)
> STOP   :R6  stop loading file /home/robert/quicklisp/setup.lisp
> SKIP   :R7  skip (LET # #)
> RETRY  :R8  retry (LET # #)
> STOP   :R9  stop loading file /home/robert/.clisprc.lisp
> Break 1 QL-SETUP[4]>
> 
> Thanks again,
> 
> Robert
> 
> PS. Here's the example foo.asd I've been working with.
> 
> (defsystem foo
>   :name "foo"
>   :version "2015.09.21"
>   :maintainer "Robert Dodier"
>   :author "Robert Dodier"
>   :licence "GPL"
>   :description "Foo, Bar, and Baz"
>   :long-description "Apply Foo's algorithm to Bar, yielding Baz"
> 
>   :components
> ((:maxima-source-file "foo1")
>  (:maxima-source-file "foo2")))
> 

Re: maintaining non-Lisp systems with ASDF

[Robert Brown]

Take a look at
https://github.com/brown/protobuf/blob/master/protobuf.asd
and the other ".asd" files in that project.

The file above modifies ASDF so that it can compile and load
Google's protocol buffer definition files.  It may be more
complicated than you need because ".proto" files are first
translated into ".lisp" files, then compiled or loaded.

Bob


On Thu, Sep 17, 2015 at 1:57 PM, Robert Dodier  wrote:
> Hi, I am exploring the possibility of using ASDF to define systems of
> non-Lisp code, specifically for Maxima (http://maxima.sourceforge.net).
> Maxima is written in Lisp but has its own language. I'd like to
> be able to load programs written in Maxima's language and/or
> Lisp.
>
> It seems like ASDF should be suitable if I can just replace calls to
> COMPILE-FILE with Maxima's compile_file, and replace LOAD with
> Maxima's load. After looking over the 3.1.5 source code, it seems
> like one could clobber COMPILE-FILE* and LOAD* with Maxima-specific
> versions. That seems suboptimal, since it would be helpful to have
> the original versions of those functions for loading Lisp files.
>
> Can someone give some advice about how to go about adapting
> ASDF to Maxima? Have there been similar attempts for other
> applications?
>
> The bigger picture is that I'd like to use Quicklisp to download and
> install Maxima packages, and Quicklisp uses ASDF to compile
> and load stuff.
>
> Thanks in advance for any light you can shed on this problem.
>
> Robert Dodier
>

Re: maintaining non-Lisp systems with ASDF

[Robert Goldman]

On 9/17/15 Sep 17 -12:57 PM, Robert Dodier wrote:
> Hi, I am exploring the possibility of using ASDF to define systems of
> non-Lisp code, specifically for Maxima (http://maxima.sourceforge.net).
> Maxima is written in Lisp but has its own language. I'd like to
> be able to load programs written in Maxima's language and/or
> Lisp.
> 
> It seems like ASDF should be suitable if I can just replace calls to
> COMPILE-FILE with Maxima's compile_file, and replace LOAD with
> Maxima's load. After looking over the 3.1.5 source code, it seems
> like one could clobber COMPILE-FILE* and LOAD* with Maxima-specific
> versions. That seems suboptimal, since it would be helpful to have
> the original versions of those functions for loading Lisp files.
> 
> Can someone give some advice about how to go about adapting
> ASDF to Maxima? Have there been similar attempts for other
> applications?
> 
> The bigger picture is that I'd like to use Quicklisp to download and
> install Maxima packages, and Quicklisp uses ASDF to compile
> and load stuff.
> 
This should be easy for you to do.

First, you need to subclass some kind of component for maxima source files.

You could either subclass cl-source-file, or perhaps something more
foundational.

You should, at least, provide methods for

PERFORM LOAD-OP 
and
PERFORM COMPILE-OP 

I believe information about doing this should be in the ASDF manual.

You may also need to tell ASDF what the file-extension of your types of
file is.

If there is a MAXIMA equivalent of a FASL file, you need to define
methods for OUTPUT-FILES and INPUT-FILES for your operations on your
component class.

That should be most of the solution.

Then you can make a new subclass of SYSTEM that will establish your
component type as the default file type for that system.  Then if you
specify that you are making one of your system subclass type (see the
manual for how to do this), then every :FILE component will be a
MAXIMA-SOURCE-FILE instead of a CL-SOURCE-FILE.

You might also want to add an implicit DEPENDS-ON to MAXIMA for this
system type.

I think that should give you what you need to find more info in the manual.

If there's something missing from the manual that you find you need to
do, please post a ticket to ASDF on launchpad.net so that we can make
necessary improvements to the manual.

Best,
r

Re: maintaining non-Lisp systems with ASDF

[Faré]

This section of the manual, and the surrounding chapter, may help:
https://common-lisp.net/project/asdf/asdf/Creating-new-operations.html#Creating-new-operations

You basically need to define a new component :maxima-file, :maxima, or
whatever the name.

If it produces a Lisp file that then gets compiled to FASL, make it a
subclass of cl-source-file indeed, and add a process-op or such to
create the lisp file from the maxima source (see how
cffi-grovel:grovel-file does it).

Otherwise, make it a subclass of source-file (or then again, possibly
cl-source-file, if it produces FASLs),
and define methods for compile-op and load-op on it.

Do NOT for whatever reason clobber any existing function (unless they
have bugs indeed, in which case, submit a patch) — there lies madness
(I should know, we were doing that at ITA before I bit the bullet and
became ASDF maintainer). Define new classes and use ad hoc
polymorphism as designed to specialize the behavior of the software.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Guns & bullets don't kill people — blood loss and organ damage kills people.