Re: WIP patch: build-system: cargo: Make lots and lots of rust packages work

[Danny Milosavljevic]

> > Can you elaborate on why you think it's a bad idea to separate the
> > source from the binaries into different outputs?  
> 
> Huh? I think it's a good idea. It just doesn't work yet and I didn't find out 
> why :)

Hmm I think I would have to update all my imported crates' package recipes to 
depend on the "src" output for inputs. I didn't do that and that's probably (?) 
why it doesn't work.

Re: WIP patch: build-system: cargo: Make lots and lots of rust packages work

[Danny Milosavljevic]

Hi David,

On Tue, 3 Jan 2017 12:08:56 +0100
David Craven  wrote:

> Can you elaborate on why you think it's a bad idea to separate the
> source from the binaries into different outputs?

Huh? I think it's a good idea. It just doesn't work yet and I didn't find out 
why :)

> I see that you are trying to mimic what cargo vendor does. Cool proof
> of concept and nice solution to the computing sha256 hashes on the
> build side.

Yeah. I thought they can't exactly replace all the bundled files in all the 
repos (written by anyone) if they ever changed the file format - so it's there 
to stay "forever". There's no downside I can see.

That said, we can also do it via cargo vendor if it's easy. But it's not really 
required - we can always just change it to that if it ever breaks.

> The question is if this is the best approach. I was thinking of trying
> [0] first and see how that works for the build system use case. The
> cargo package example using vendoring is intended to be a temporary
> solution, and I'm not sure we have finished exploring the "design
> space" yet.
> 
> [0] https://github.com/alexcrichton/cargo-local-registry

That's interesting too! We can try both.

My patch is just a WIP patch of something that works. We should explore the 
other alternatives, too.

But about the current approach in master, I talked to Alex Crichton [1], he 
said that the "replace" approach won't work without a registry because it tries 
to replace crates by the exact same version and what you specify for "replace" 
is just an API version. I think by then you can just go all the way to a custom 
registry and dispense of the "replace".

[1] https://github.com/rust-lang/cargo/issues/3476

Re: WIP patch: build-system: cargo: Make lots and lots of rust packages work

[David Craven]

> For it to work, you also have to back out the (cons "src" outputs) (in 
> ./guix/build-system/cargo.scm) and the (assoc-ref outputs "src") (in 
> guix/build/cargo-build-system.scm) starting from current master.

Can you elaborate on why you think it's a bad idea to separate the
source from the binaries into different outputs?

I see that you are trying to mimic what cargo vendor does. Cool proof
of concept and nice solution to the computing sha256 hashes on the
build side.

The question is if this is the best approach. I was thinking of trying
[0] first and see how that works for the build system use case. The
cargo package example using vendoring is intended to be a temporary
solution, and I'm not sure we have finished exploring the "design
space" yet.

[0] https://github.com/alexcrichton/cargo-local-registry

WIP patch: build-system: cargo: Make lots and lots of rust packages work

[Danny Milosavljevic]

For it to work, you also have to back out the (cons "src" outputs) (in 
./guix/build-system/cargo.scm) and the (assoc-ref outputs "src") (in 
guix/build/cargo-build-system.scm) starting from current master.

--- guix/build/cargo-build-system.scm   2017-01-03 04:31:01.691543854 +0100
+++ /home/dannym/src/guix/guix/build/cargo-build-system.scm 2017-01-03 
04:55:26.810917585 +0100
@@ -19,6 +19,8 @@
 (define-module (guix build cargo-build-system)
   #:use-module ((guix build gnu-build-system) #:prefix gnu:)
   #:use-module (guix build utils)
+  #:use-module (ice-9 popen)
+  #:use-module (ice-9 rdelim)
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 format)
   #:use-module (ice-9 match)
@@ -43,35 +45,86 @@
 
 (define* (configure #:key inputs #:allow-other-keys)
   "Replace Cargo.toml [dependencies] section with guix inputs."
-  ;; Make sure Cargo.toml is writeable when the crate uses git-fetch.
-  (chmod "Cargo.toml" #o644)
-  (let ((port (open-file "Cargo.toml" "a" #:encoding "utf-8")))
-(format port "~%[replace]~%")
-(for-each
- (match-lambda
-   ((name . path)
-(let ((crate (package-name->crate-name name)))
-  (when (and crate path)
-(match (string-split (basename path) #\-)
-  ((_ ... version)
-   (format port "\"~a:~a\" = { path = \"~a/share/rust-source\" }~%"
-   crate version path)))
- inputs)
-(close-port port))
+  (system* "chmod" "+w" "Cargo.toml")
+  (system* "chmod" "+w" ".")
+  (if (not (file-exists? "vendor"))
+(begin
+  (if (not (file-exists? "Cargo.lock"))
+(substitute* "Cargo.toml"
+  ((".*32-sys.*") "
+")
+  ((".*winapi.*") "
+")))
+  (mkdir "vendor")
+  (for-each
+(match-lambda
+  ((name . path)
+   (let ((crate (package-name->crate-name name)))
+ (when (and crate path)
+   (match (string-split (basename path) #\-)
+ ((_ ... version)
+  (symlink (string-append path "/share/rust-source") 
(string-append "vendor/" (basename path)
+inputs)
+  (mkdir-p ".cargo")
+  (let ((port (open-file ".cargo/config" "w" #:encoding "utf-8")))
+(display "
+[source.crates-io]
+registry = 'https://github.com/rust-lang/crates.io-index'
+replace-with = 'vendored-sources'
+
+[source.vendored-sources]
+directory = '" port)
+(display (getcwd) port)
+(display "/vendor" port)
+(display "'
+" port)
+(close-port port
+(setenv "CC" (string-append (assoc-ref inputs "gcc") "/bin/gcc"))
+
+;(setenv "CARGO_HOME" "/gnu/store")
+; (setenv "CMAKE_C_COMPILER" cc)
   #t)
 
-(define* (build #:key (cargo-build-flags '("--release" "--frozen"))
-#:allow-other-keys)
+(define* (build #:key (cargo-build-flags '("--release")) #:allow-other-keys)
   "Build a given Cargo package."
   (if (file-exists? "Cargo.lock")
-  (zero? (apply system* `("cargo" "build" ,@cargo-build-flags)))
-  #t))
+(zero? (apply system* `("cargo" "build" ,@cargo-build-flags)))
+#t))
 
 (define* (check #:key tests? #:allow-other-keys)
   "Run tests for a given Cargo package."
-  (if (and tests? (file-exists? "Cargo.lock"))
-  (zero? (system* "cargo" "test"))
-  #t))
+  (when tests?
+(zero? (system* "cargo" "test"
+
+(define (file-sha256 file-name)
+(let ((port (open-pipe* OPEN_READ
+"sha256sum"
+"--"
+file-name)))
+  (let ((result (read-delimited " " port)))
+(close-pipe port)
+result)))
+
+;; Example /gnu/store/hwlr49riz3la33m6in2n898ly045ylld-rust-rand-0.3.15: ; 
generally not a lot of store items.
+(define (generate-checksums dir-name src-name)
+  (let* ((file-names (find-files dir-name "."))
+ (dir-prefix-name (string-append dir-name "/"))
+ (dir-prefix-name-len (string-length dir-prefix-name))
+ (checksums-file-name (string-append dir-name 
"/.cargo-checksum.json")))
+(call-with-output-file checksums-file-name
+  (lambda (port)
+(display "{\"files\":{" port)
+(let ((sep ""))
+  (for-each (lambda (file-name)
+(let ((file-relative-name (string-drop file-name 
dir-prefix-name-len)))
+  (display sep port)
+  (set! sep ",")
+  (write file-relative-name port)
+  (display ":" port)
+  (write (file-sha256 file-name) port))) file-names))
+(display "},\"package\":" port)
+(write (file-sha256 src-name) port)
+(display "}" port)
 
 (define* (install #:key inputs outputs #:allow-other-keys)
   "Install a given Cargo package."
@@ -87,15 +140,17 @@
 ;; references in Cargo.toml with store paths.
 (copy-recursively "src" (string-append rsrc "/src"))
 (install-file "Cargo.toml" rsrc)
+(system* "touch" (string-append rsrc "/.cargo-ok"))
+