Re: Quicklisp importer bug in tarball->extract->parse
swedebugia writes: > Hi > > Thanks a lot for taking the time to look at this. > > On 2019-04-11 16:31, Pierre Neidhardt wrote: >> Hi! >> >> I've had time to look at it a little. >> >> If you want to debug, you can, for instance, put a breakpoint in >> guess-requirements-from-source, > > How exactly do you do this? See --8<---cut here---start->8--- ,help debug --8<---cut here---end--->8--- from the Guile REPL. In particular --8<---cut here---start->8--- ,break PROCEDURE --8<---cut here---end--->8--- should do what you want. > (memoize > ... `memoize' takes a procedure as argument, not an arbitrary expression. For instance in the following --8<---cut here---start->8--- (let ((mfoo (memoize foo))) (mfoo 17) (mfoo 17)) --8<---cut here---end--->8--- `foo' will be called only once. You don't have to use `memoize' though, you can also store the value you want in a local variable and check if it's set or not. We can always refine the code later. >> It seems that you are on the right track, great job! > > Thanks for the encouragement! And thanks for giving this a shot! :) -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Quicklisp importer bug in tarball->extract->parse
Hi
Thanks a lot for taking the time to look at this.
On 2019-04-11 16:31, Pierre Neidhardt wrote:
> Hi!
>
> I've had time to look at it a little.
>
> If you want to debug, you can, for instance, put a breakpoint in
> guess-requirements-from-source,
How exactly do you do this?
I looked in the guile manual and found the module (system vm trap-state)
and tried calling add-trap-at-procedure-call! like this:
(add-trap-at-procedure-call!
(system* "tar" "xf" tarball "-C" dir "--wildcards" (string-append "*"
file-name)
But it failed with:
;;; (file-name "1am.asd")
Backtrace:
13 (apply-smob/1 #)
In ice-9/boot-9.scm:
705:2 12 (call-with-prompt _ _ #)
In ice-9/eval.scm:
619:8 11 (_ #(#(#)))
In ice-9/boot-9.scm:
2312:4 10 (save-module-excursion _)
3831:12 9 (_)
In guix/import/quicklisp.scm:
404:3 8 (_)
In guix/utils.scm:
618:8 7 (call-with-temporary-output-file _)
632:8 6 (call-with-temporary-directory _)
In guix/import/quicklisp.scm:
374:32 5 (_ "/tmp/guix-directory.Z4chBm")
In srfi/srfi-1.scm:
592:17 4 (map1 ("1am.asd"))
In guix/import/quicklisp.scm:
380:41 3 (_ _)
In system/vm/trap-state.scm:
213:15 2 (add-trap-at-procedure-call! 0 _)
In system/vm/traps.scm:
137:45 1 (trap-at-procedure-call 0 # #:our-frame? _)
In unknown file:
0 (scm-error misc-error #f "~A ~S" ("Not a VM program" 0) #f)
ERROR: In procedure scm-error:
Not a VM program 0
> There is nothing wrong with find-files: the "dir" directory is empty.
> This is because the extraction failed:
> check the exit-code of system* before proceeding, you'll see what
> happens ;)
>
> --8<---cut here---start->8---
> (system* "tar" "xf" tarball "-C" dir file-name)
> --8<---cut here---end--->8---
>
> The above will error out if "file-name" is not found. For 1am:
>
> Queried file-name:
>
> "1am.asd"
>
> Tarball actual file:
>
> "1am-20141106-git/1am.asd"
>
> I don't think we can predict the root folder, and more generally
> projects have different trees.
> So I suggest we extract the whole tarball and proceed from there. So
> your code should work by just removing "file-name":
>
> --8<---cut here---start->8---
> (system* "tar" "xf" tarball "-C" dir)
> --8<---cut here---end--->8---
>
> Makes sense?
Yes. I implemented Ricardos wildcard-suggestion:
(system* "tar" "xf" tarball "-C" dir "--wildcards" (string-append "*"
file-name))
>
> Unrelated comment: If I'm not mistaken, you are fetching the meta-file
> and the index-file on every query. I suggest you keep those in memories
> as soon as possible (memoize them), this will allow you to iterate much
> faster when developing.
I already tried by adding:
(memoize
(let* ( ;;(name "circular-streams")
(name "1am")
(source-url (ql-extract 'url name))
(asd (ql-extract 'system-files name)))
(peek 'asd asd)
(call-with-temporary-output-file
(lambda (temp port)
(begin (url-fetch source-url temp)
(guess-requirements-from-source source-url asd temp)
)
and
(let* ( ;;(name "circular-streams")
(name "1am")
(source-url (memoize (ql-extract 'url name)))
(asd (memoize (ql-extract 'system-files name
(peek 'asd asd)
(call-with-temporary-output-file
(lambda (temp port)
(begin (url-fetch source-url temp)
(guess-requirements-from-source source-url asd temp)
But it still fetches the index-file 2 times regardless where I put it.
Any ideas?
> It seems that you are on the right track, great job!
Thanks for the encouragement!
--
Cheers Swedebugia
signature.asc
Description: OpenPGP digital signature
Re: Quicklisp importer bug in tarball->extract->parse
Pierre Neidhardt writes: > The above will error out if "file-name" is not found. For 1am: > > Queried file-name: > > "1am.asd" > > Tarball actual file: > > "1am-20141106-git/1am.asd" > > I don't think we can predict the root folder, and more generally > projects have different trees. We can add “--strip-components=1” and do the whole thing in “with-directory-excursion”. This way we’ll always know the directory name. You can also extract individual files in the archive by wildcard, so you don’t need to know the exact name. -- Ricardo
Re: Quicklisp importer bug in tarball->extract->parse
Hi! I've had time to look at it a little. If you want to debug, you can, for instance, put a breakpoint in guess-requirements-from-source, then enter the following in your guile prompt: ,backtrace #:full? #t or shorter ,bt #:full? #t That will print the backtrace will all local variables. There is nothing wrong with find-files: the "dir" directory is empty. This is because the extraction failed: check the exit-code of system* before proceeding, you'll see what happens ;) --8<---cut here---start->8--- (system* "tar" "xf" tarball "-C" dir file-name) --8<---cut here---end--->8--- The above will error out if "file-name" is not found. For 1am: Queried file-name: "1am.asd" Tarball actual file: "1am-20141106-git/1am.asd" I don't think we can predict the root folder, and more generally projects have different trees. So I suggest we extract the whole tarball and proceed from there. So your code should work by just removing "file-name": --8<---cut here---start->8--- (system* "tar" "xf" tarball "-C" dir) --8<---cut here---end--->8--- Makes sense? Unrelated comment: If I'm not mistaken, you are fetching the meta-file and the index-file on every query. I suggest you keep those in memories as soon as possible (memoize them), this will allow you to iterate much faster when developing. It seems that you are on the right track, great job! -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Quicklisp importer bug in tarball->extract->parse
Hi, thanks for your work! I'll look into it Very Soon™. Cheers! -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Quicklisp importer bug in tarball->extract->parse
Hi I continued working on it and have run into a bug with extracting and parsing files in the tarball (like the pypi importer does) This is really putting my lisp skills to the test I feel like i'm in over my head here. The idea is to - first extract information from quicklisp - done - then extract the tarball - done - then find the 'name'.asd-file in the extracted directory - fails - then parse with PEG - todo - then print the result - todo Does anyone have a clue what is wrong or how to debug (find-files)? Here is the code: https://gitlab.com/swedebugia/guix/tree/quicklisp Here is the output: Starting download of /tmp/guix-file.Q7fTrf From https://beta.quicklisp.org/dist/quicklisp.txt... quicklisp.txt 408B 202KiB/s 00:00 [##] 100.0% Starting download of /tmp/guix-file.N8gfUL From http://beta.quicklisp.org/dist/quicklisp/2019-03-07/releases.txt... releases.txt 423KiB 1.1MiB/s 00:00 [##] 100.0% Starting download of /tmp/guix-file.B6vRet From https://beta.quicklisp.org/dist/quicklisp.txt... quicklisp.txt 408B 223KiB/s 00:00 [##] 100.0% Starting download of /tmp/guix-file.QA07gb From http://beta.quicklisp.org/dist/quicklisp/2019-03-07/releases.txt... releases.txt 423KiB 768KiB/s 00:01 [##] 100.0% ;;; (asd "1am.asd") Starting download of /tmp/guix-file.4QyAk4 From http://beta.quicklisp.org/archive/1am/2014-11-06/1am-20141106-git.tgz... …06-git.tgz 3KiB4.2MiB/s 00:00 [##] 100.0% ;;; (dirname "1am-20141106-git") ;;; (system-files "1am.asd") ;;; (dir "/tmp/guix-directory.DnfwBX") ;;; (file-name "1am.asd") ;;; (find-files ()) warning: Failed to extract requirements files -- Cheers Swedebugia signature.asc Description: OpenPGP digital signature
