Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Andreas Enge writes: > >> Hello Ricardo, >> >> On Sat, Feb 17, 2018 at 02:04:31PM +0100, Ricardo Wurmus wrote: >>> I’ve built ghc-resourcet successfully with this change. I’d be happy if >>> those of you who reported build failures could please retry building >>> ghc-resourcet with current master or core-updates (the fix to ghc-mtl is >>> on both branches now). >> >> it works for me on current core-updates. Thanks! > > Thank you for the feedback! > > I’ll merge core-updates into master now. > > Thanks everyone for working on this! Excellent! Pjotr mentioned he wanted to fix ldc, but since it only has a handful of dependents, I think that can be done on master, right? --8<---cut here---start->8--- [0] marusich@garuda.local:~/guix-core-updates $ ./pre-inst-env guix refresh -l ldc@0.17.4 Building the following 2 packages would ensure 4 dependent packages are rebuilt: sambamba@0.6.5 dub@1.5.0 [0] marusich@garuda.local:~/guix-core-updates $ ./pre-inst-env guix refresh -l ldc@1.1.1 Building the following 2 packages would ensure 3 dependent packages are rebuilt: sambamba@0.6.5 dub@1.5.0 [0] marusich@garuda.local:~/guix-core-updates $ --8<---cut here---end--->8--- -- Chris signature.asc Description: PGP signature
Re: heads-up: Haskell updates
Hi Timothy, > We could follow the style of Hackage and force everything to reference > the base libraries. To make this work, we would have to make builds > fail if they don’t reference a base library that they need. Maybe > there’s a way to hide the base libraries in the Haskell build system, > and use packages to expose them. The major downside to this is adding a > “ghc-base” input to every Haskell package (and “ghc-binary” to a bunch, > etc.). The upside is that it is more intuitive: the inputs look more > like Hackage, and you could try new versions of the base libraries using > standard Guix tools like: > > $ guix package -i ghc-pandoc \ > --with-input=ghc-transformers=ghc-transformers-new > > (This would update all the dependencies, too, leaving the GHC-provided > library hidden and only exposing the new library, thus avoiding all the > conflicting version problems.) This wouldn’t be good, because we would have to make sure that the base packages are kept at the versions of the packages that are provided by GHC itself. Newer versions for these base packages are often different enough to cause problems. I’ve tried building many packages with newer versions of e.g. transformers and it rarely worked without problems. If there’s only one package where we have to use an older version of one of the base packages we would introduce problems when that package were to be used with other packages. We must avoid this. > The second approach would be to leave everything implicit, and add notes > everywhere not to break things (in the docs, the linter, and the > importer). I guess we would have to be careful when updating GHC in > case it adopts new base libraries. The appeal of this approach is that > it is basically what we just did, so it’s done modulo the changes to the > linter and importer. I much prefer this. FWIW we already do this for R packages. We just have to accept that GHC provides some modules that are also available as separate packages. Leaving these packages off when writing package definitions is the only solution that ensures that we won’t run into conflicts at some later point. When we update the default GHC we will have to read the migration notes anyway (these notes tell us what modules are now part of GHC or have been spun out as separate packages). -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Hi Danny, Danny Milosavljevic writes: > I don't see where the diamond depenency is... GHC includes “transformers”, which is what most packages use, but “ghc-mtl” includes a different version (it has the same version number, so it is hard to see from the warnings). > But Timothy has a patchset that removes the customized "backported" > ghc-transformer > package dependency since ghc-transformer is now part of ghc proper. > > Its subject is "gnu: Remove Haskell packages provided by GHC". > > Maybe that helps... Actually, Ricardo already pushed a fix that should clear up the “ghc-mtl” problem. My patch just applies the same idea to all the other packages that have the same problem (originally it included “ghc-mtl”, but Ricardo beat me to it). -- Tim
Re: heads-up: Haskell updates
Hi Mark,
> Unfortunately, the patch applied by Danny didn't fix the problem on
> Hydra. I didn't look very closely, but from a glance the failure looks
> the same:
Yeah, but it fixed the ghc-pkg one ("ghc-pkg couldn't decommit memory"
or something) - so we can be reasonably sure that the ghc package manager
provided the correct packages (and complete ones) now.
It's too bad that it didn't fix the big problem - I hoped it would...
>So, your analysis makes sense and is much appreciated, but the available
>evidence seems to suggest that something else might be going on.
ghc-pkg is saying:
Warning: This package indirectly depends on multiple versions of the same
package. This is highly likely to cause a compile failure.
package transformers-compat-0.5.1.4 requires transformers-0.5.2.0
package transformers-base-0.4.4 requires transformers-0.5.2.0
package primitive-0.6.3.0 requires transformers-0.5.2.0
package monad-control-1.0.1.0 requires transformers-0.5.2.0
package hspec-core-2.2.4 requires transformers-0.5.2.0
package hspec-2.2.4 requires transformers-0.5.2.0
package QuickCheck-2.10.1 requires transformers-0.5.2.0
package resourcet-1.1.7.5 requires transformers-0.5.2.0
package mtl-2.2.1 requires transformers-0.5.2.0
package mmorph-1.0.6 requires transformers-0.5.2.0
package exceptions-0.8.3 requires transformers-0.5.2.0
And we have
environment variable `GHC_PACKAGE_PATH' set to `
/gnu/store/34n70arzg4kc1yvckmz2sws38bbjxwsd-ghc-8.0.2/lib/ghc-8.0.2/package.conf.d:
/gnu/store/38m5h52cjr9mifpy1pjz84nvdqpkf0xa-ghc-lifted-base-0.2.3.8/lib/ghc-8.0.2/ghc-lifted-base-0.2.3.8.conf.d:
/gnu/store/kpz9xag4nmp3viwnxy3gvvfyj7050zsx-ghc-hspec-2.2.4/lib/ghc-8.0.2/ghc-hspec-2.2.4.conf.d:
/gnu/store/mgz1y98r11ys4mmskdq609r22w0f4cbz-ghc-transformers-base-0.4.4/lib/ghc-8.0.2/ghc-transformers-base-0.4.4.conf.d:
/gnu/store/bbhmvi5c51530happmcyqd7p2rfafyyi-ghc-monad-control-1.0.1.0/lib/ghc-8.0.2/ghc-monad-control-1.0.1.0.conf.d:
/gnu/store/sy3s3myih4wzc5m1g58411aam1hghrx8-ghc-transformers-compat-0.5.1.4/lib/ghc-8.0.2/ghc-transformers-compat-0.5.1.4.conf.d:
/gnu/store/smmh2iqqcdaxkzwr8dwkh9zqwb4nnx1q-ghc-mtl-2.2.1/lib/ghc-8.0.2/ghc-mtl-2.2.1.conf.d:
/gnu/store/pdgsa9gkvw9r8xw30gfg5502ysvdn0dz-ghc-mmorph-1.0.6/lib/ghc-8.0.2/ghc-mmorph-1.0.6.conf.d:
/gnu/store/an6vdcgn5366npr1kww3ycl1zv8q4vsd-ghc-exceptions-0.8.3/lib/ghc-8.0.2/ghc-exceptions-0.8.3.conf.d'
I don't see where the diamond depenency is...
But Timothy has a patchset that removes the customized "backported"
ghc-transformer
package dependency since ghc-transformer is now part of ghc proper.
Its subject is "gnu: Remove Haskell packages provided by GHC".
Maybe that helps...
Re: heads-up: Haskell updates
Hi Danny, Danny Milosavljevic writes: > On Thu, 15 Feb 2018 03:41:33 -0500 > Mark H Weaver wrote: > >> Given that it's using #ifdef to detect this, I'd expect the result to >> depend only on the _headers_ being used to compile, and not the actual >> kernel running the build. > > Yes, that's precisely the problem. > > * The [glibc or kernel] headers determine whether the ghc compilation detects > it > * The Linux kernel determines whether the functionality is actually present at > runtime when any ghc program (like ghc-pkg) runs. > > Linus takes backward compatibility seriously - the constants > or functionality are not going to vanish from Linux later. > > So the only problematic case is that the build process finds MADV_FREE > but the running Linux doesn't yet have it and a ghc program runs on it. > > Reading MADV_DONTNEED docs again, MADV_DONTNEED pretty much does the same > as MADV_FREE - but MADV_DONTNEED promises to make later accesses to the > range succeed (by providing a new zero-filled page if necessary) while > MADV_FREE promises to make them fail. > > So one could fall back to MADV_DONTNEED - should be fine, though a little > weird > for an allocator. > > If that's the case and the build still fails, let's just apply the Haskell > patch > to ghc (or update ghc if there's a newer release). Unfortunately, the patch didn't fix the problem on Hydra, as I mentioned in another message: https://hydra.gnu.org/build/2501179 https://hydra.gnu.org/build/2501155 >> why it works for Ricardo but not for Andreas and Hydra. > > At runtime it depends on the Linux kernel version present whether > the deallocation will work or not. > > So I suspect that Ricardo has a Linux >= 4.5 but Andreas and Hydra > have a Linux < 4.5. Is that correct? This turned out to be incorrect. Andreas said he was running a Debian kernel, version 4.9.x. So, your analysis makes sense and is much appreciated, but the available evidence seems to suggest that something else might be going on. Any more ideas? Mark
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > I’ll merge core-updates into master now. Is that premature? There are still over 500 newly failed jobs on core-updates compared to master. https://hydra.gnu.org/eval/109914?compare=109912 Mark
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > So let’s apply the patch. > Danny, could you please do this on master and core-updates? > > I’d like to merge core-updates this week and it would be great if we > could build all of the Haskell packages (and all those R packages that > depend on Pandoc) before the merge. Unfortunately, the patch applied by Danny didn't fix the problem on Hydra. I didn't look very closely, but from a glance the failure looks the same: https://hydra.gnu.org/build/2501179 (x86_64) https://hydra.gnu.org/build/2501155 (i686) Both of these builds were performed today, for evaluation 109914 which corresponds to commit f6cccefed599e06a814b702aa79b8a09f01ec41c on core-updates. That's Danny's commit: https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates&id=f6cccefed599e06a814b702aa79b8a09f01ec41c Mark
Re: heads-up: Haskell updates
Andreas Enge writes: > Hello Ricardo, > > On Sat, Feb 17, 2018 at 02:04:31PM +0100, Ricardo Wurmus wrote: >> I’ve built ghc-resourcet successfully with this change. I’d be happy if >> those of you who reported build failures could please retry building >> ghc-resourcet with current master or core-updates (the fix to ghc-mtl is >> on both branches now). > > it works for me on current core-updates. Thanks! Thank you for the feedback! I’ll merge core-updates into master now. Thanks everyone for working on this! -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Ricardo Wurmus writes: >> Based on your earlier suggestion, I played around with removing all the >> packages that GHC provides. I made the same change to ghc-mtl on >> core-updates, and it allows me to build ghc-resourcet. > […] >> Is this too drastic? I could rebase on top of your ghc-mtl changes and >> submit a patch if you think its OK. > > Not too drastic. This is exactly what I had hoped for :) > > Thank you for making the effort. A patch on top of latest core-updates > / master would be very welcome. I sent the patch. It’s bug #30501. (The most complicated part was the changelog; I hope it’s OK!) > Now, how do we prevent this in the future? Can we modify “guix lint” to > warn about these cases? Can we also change the hackage importer to > keep these packages out of the inputs of generated package definitions? I can think of two approaches. We could follow the style of Hackage and force everything to reference the base libraries. To make this work, we would have to make builds fail if they don’t reference a base library that they need. Maybe there’s a way to hide the base libraries in the Haskell build system, and use packages to expose them. The major downside to this is adding a “ghc-base” input to every Haskell package (and “ghc-binary” to a bunch, etc.). The upside is that it is more intuitive: the inputs look more like Hackage, and you could try new versions of the base libraries using standard Guix tools like: $ guix package -i ghc-pandoc \ --with-input=ghc-transformers=ghc-transformers-new (This would update all the dependencies, too, leaving the GHC-provided library hidden and only exposing the new library, thus avoiding all the conflicting version problems.) We would have to make sure that “guix package -i ghc” still provides the libraries, though, to save people a lot of confusion. The second approach would be to leave everything implicit, and add notes everywhere not to break things (in the docs, the linter, and the importer). I guess we would have to be careful when updating GHC in case it adopts new base libraries. The appeal of this approach is that it is basically what we just did, so it’s done modulo the changes to the linter and importer. Personally, I prefer the first approach even though it’s a bit of a radical departure. It feels like it fits more both with Guix and Haskell. I also think that most Haskell people coming to Guix would just understand it without any extra explanation. Updating every single Haskell package is a bit daunting :S. It’s something I’m willing to look at, though. What do you think? I’m happy to look at the linter and importer, too. I would like to package “git-annex”, so making the Haskell stuff a little nicer is only a minor detour towards that goal (last I checked, I will have to import a lot of packages yet). -- Tim
Re: heads-up: Haskell updates
Hello Ricardo, Ricardo Wurmus writes: > thanks for the extra information. Thank you for picking that up. > I find it very puzzling that I cannot reproduce these build failures on > my machine. > > I have just rebuilt ghc-resourcet with a modified ghc-mtl, which I > suspect is the source of the problem, because it pulls in a newer > version of ghc-transformers. I’m going to push this to core-updates and > master in a moment. This fixes the issue for me, thanks! Oleg. signature.asc Description: PGP signature
Re: heads-up: Haskell updates
Hello Ricardo, On Sat, Feb 17, 2018 at 02:04:31PM +0100, Ricardo Wurmus wrote: > I’ve built ghc-resourcet successfully with this change. I’d be happy if > those of you who reported build failures could please retry building > ghc-resourcet with current master or core-updates (the fix to ghc-mtl is > on both branches now). it works for me on current core-updates. Thanks! Andreas
Re: heads-up: Haskell updates
Hi Timothy, > Ricardo Wurmus writes: > >> I have just rebuilt ghc-resourcet with a modified ghc-mtl, which I >> suspect is the source of the problem, because it pulls in a newer >> version of ghc-transformers. I’m going to push this to core-updates and >> master in a moment. > > Based on your earlier suggestion, I played around with removing all the > packages that GHC provides. I made the same change to ghc-mtl on > core-updates, and it allows me to build ghc-resourcet. […] > Is this too drastic? I could rebase on top of your ghc-mtl changes and > submit a patch if you think its OK. Not too drastic. This is exactly what I had hoped for :) Thank you for making the effort. A patch on top of latest core-updates / master would be very welcome. Thank you very much! Now, how do we prevent this in the future? Can we modify “guix lint” to warn about these cases? Can we also change the hackage importer to keep these packages out of the inputs of generated package definitions? -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Hi Ricardo, Ricardo Wurmus writes: > I have just rebuilt ghc-resourcet with a modified ghc-mtl, which I > suspect is the source of the problem, because it pulls in a newer > version of ghc-transformers. I’m going to push this to core-updates and > master in a moment. Based on your earlier suggestion, I played around with removing all the packages that GHC provides. I made the same change to ghc-mtl on core-updates, and it allows me to build ghc-resourcet. I went a bit further and removed ghc-array, ghc-binary, ghc-bytestring, ghc-directory, ghc-haskeline, ghc-process, ghc-transformers, and ghc-transformers-0.4.2.0 from everywhere. As far as I can tell, only four packages use these as inputs: ghc-mtl, ghc-tar, ghc-hslogger, and darcs. After removing all the references, I tested building ghc-pandoc (which uses ghc-mtl and ghc-tar), ghc-hslogger, and darcs. All were successful. Is this too drastic? I could rebase on top of your ghc-mtl changes and submit a patch if you think its OK. -- Tim
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Timothy Sample writes: > >> Ricardo Wurmus writes: >> >>> I think that’s a misunderstanding. The cause for the error is earlier >>> when it complains that some packages depend on different versions of the >>> “transformers” package. “StateT” is a monad transformer. >> >> For what it’s worth, I fixed this error on my machine by adding >> “ghc-transformers” as an input to “ghc-transformers-compat”. (This also >> fixed “ghc-adjunctions”.) This suggests to me that what you wrote >> earlier accounts for the problem: >> >>> * many problems are caused by the fact that GHC includes a bunch of >>> packages that really shouldn’t be overridden by packages. Examples >>> are ghc-directory, ghc-binary, ghc-bytestring, etc. Since we still >>> have packages where these inputs are used, there can be conflicts down >>> the line, which are hard to fix. > > The solution might be to remove ghc-transformers from all packages that > currently have it as an input rather than add it wherever the > transformers is required. This seems to be only ghc-mtl. I’ve removed > it and am building ghc-mtl and ghc-resourcet now. I’ve built ghc-resourcet successfully with this change. I’d be happy if those of you who reported build failures could please retry building ghc-resourcet with current master or core-updates (the fix to ghc-mtl is on both branches now). Thanks in advance! -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Hi Oleg, thanks for the extra information. I find it very puzzling that I cannot reproduce these build failures on my machine. I have just rebuilt ghc-resourcet with a modified ghc-mtl, which I suspect is the source of the problem, because it pulls in a newer version of ghc-transformers. I’m going to push this to core-updates and master in a moment. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Oleg Pykhalov writes: > >> Ricardo Wurmus writes: >> >>> Marius Bakke writes: >>> Should we do a new merge to get the GHC patch, or just merge core-updates and let the problem "fix itself" on 'master'? >>> >>> I’d prefer building GHC and ghc-resourcet first. We don’t know if this >>> patch actually fixes our problems. We should merge master into >>> core-updates again. >> >> I've failed to build ‘ghc-pandoc’, because of dependency ‘ghc-resourcet’ >> failed to build on both branches >> >>- origin/master 369eee8763ca34b427860a86c8fe2db0963b52ae >>- origin/core-updates a124e4258ad911e1a65edb6c7d7d8f095249db5f > > Could you tell me more about the system on which you tried building it? > What kernel version did you use and how much memory does the system > have? GuixSD --8<---cut here---start->8--- $(guix build inxi)/bin/inxi -F -c 0 | epipe &>/dev/null System:Host: magnolia Kernel: 4.15.2-gnu x86_64 bits: 64 Desktop: N/A Distro: This is the GNU system. Welcome. Machine: Device: desktop Mobo: ASRock model: H67M-GE/HT serial: N/A UEFI: American Megatrends v: P2.10 date: 04/27/2012 CPU: Quad core Intel Core i5-3570K (-MCP-) cache: 6144 KB clock speeds: max: 3800 MHz 1: 2885 MHz 2: 1631 MHz 3: 3595 MHz 4: 1693 MHz … Info: Processes: 181 Uptime: 3 days 2:02 Memory: 6001.4/31878.0MB Client: Shell (bash) inxi: 2.3.56 --8<---cut here---end--->8--- > Does the log contain a message about ghc-pkg failing? No, 369eee8763ca34b427860a86c8fe2db0963b52ae --8<---cut here---start->8--- ./pre-inst-env env GUIX_PACKAGE_PATH= guix build --no-grafts ghc-resourcet The following derivation will be built: /gnu/store/fx5c5dhv0kh3s3qaxpapx3yqzrn1bh5m-ghc-resourcet-1.1.7.5.drv @ build-started /gnu/store/fx5c5dhv0kh3s3qaxpapx3yqzrn1bh5m-ghc-resourcet-1.1.7.5.drv - x86_64-linux /var/log/guix/drvs/fx//5c5dhv0kh3s3qaxpapx3yqzrn1bh5m-ghc-resourcet-1.1.7.5.drv.bz2 starting phase `set-SOURCE-DATE-EPOCH' phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds starting phase `set-paths' environment variable `PATH' set to `/gnu/store/pr4hycii84shm1sh0gvxixc12ws81cfx-ghc-8.0.2/bin:/gnu/store/zhhdii9mjckb6c2f7abmdsx39khv6dla-tar-1.29/bin:/gnu/store/4iqyh8xqjxazza3lx2iz5v39ipzifsfj-gzip-1.8/bin:/gnu/store/6hicns85s8ddp0y539wdspwx22ac2kmv-bzip2-1.0.6/bin:/gnu/store/g3nari57wcfnm00kv9bnpyzdzfq4h8pk-xz-5.2.2/bin:/gnu/store/msw2q7nr3vfmgwyxf15y0x7qbngs9y3d-file-5.30/bin:/gnu/store/7zbscp5r0djsalcnvfrm7g0mp70mhrkd-diffutils-3.6/bin:/gnu/store/9b0nh58q1dwxli80xj15gv2037az96xw-patch-2.7.5/bin:/gnu/store/7v8369lgnqvpphcw06hg59hb8hxmkr8x-sed-4.4/bin:/gnu/store/nzv180i3z33vnb9krmc73mazhf626384-findutils-4.6.0/bin:/gnu/store/9pq16kfcldqqcbd58mmfp37g3awhg4sd-gawk-4.1.4/bin:/gnu/store/sngyhm974sbmljknwb1xrni1ggzhpm4d-grep-3.0/bin:/gnu/store/42d5rjrdkln6nwvzwdc8dyd4w6iy3n5j-coreutils-8.27/bin:/gnu/store/6vps2jc0v6b4hr8ds98785xcf8061wz0-make-4.2.1/bin:/gnu/store/kpxi8h3669afr9r1bgvaf9ij3y4wdyyn-bash-minimal-4.4.12/bin:/gnu/store/qk79ck8gy1zppi4mbw4zw2y4z326wa4i-ld-wrapper-0/bin:/gnu/store/nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28/bin:/gnu/store/5sv5zy2kgg6iaqyv8zw49w4243j0xkd0-gcc-5.4.0/bin:/gnu/store/n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25/bin:/gnu/store/n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25/sbin' find-files: /gnu/store/bpjn8p93svqwhc8qa681c0j7nb1ssspv-resourcet-1.1.7.5.tar.gz/lib/ghc-8.0.2: Not a directory find-files: /gnu/store/zhhdii9mjckb6c2f7abmdsx39khv6dla-tar-1.29/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/4iqyh8xqjxazza3lx2iz5v39ipzifsfj-gzip-1.8/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/6hicns85s8ddp0y539wdspwx22ac2kmv-bzip2-1.0.6/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/g3nari57wcfnm00kv9bnpyzdzfq4h8pk-xz-5.2.2/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/msw2q7nr3vfmgwyxf15y0x7qbngs9y3d-file-5.30/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/7zbscp5r0djsalcnvfrm7g0mp70mhrkd-diffutils-3.6/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/9b0nh58q1dwxli80xj15gv2037az96xw-patch-2.7.5/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/7v8369lgnqvpphcw06hg59hb8hxmkr8x-sed-4.4/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/nzv180i3z33vnb9krmc73mazhf626384-findutils-4.6.0/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/9pq16kfcldqqcbd58mmfp37g3awhg4sd-gawk-4.1.4/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/sngyhm974sbmljknwb1xrni1ggzhpm4d-grep-3.0/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/42d5rjrdkln6nwvzwdc8dyd4w6iy3n5j-coreutils-8.27/lib/ghc-8.0.2: No such file or directory find-files: /gnu/store/6vps2jc0v6b4hr8ds98785xcf8061wz0-make-4.2.1/lib/ghc-8.0.2: No such file or directory f
Re: heads-up: Haskell updates
Timothy Sample writes: > Ricardo Wurmus writes: > >> I think that’s a misunderstanding. The cause for the error is earlier >> when it complains that some packages depend on different versions of the >> “transformers” package. “StateT” is a monad transformer. > > For what it’s worth, I fixed this error on my machine by adding > “ghc-transformers” as an input to “ghc-transformers-compat”. (This also > fixed “ghc-adjunctions”.) This suggests to me that what you wrote > earlier accounts for the problem: > >> * many problems are caused by the fact that GHC includes a bunch of >> packages that really shouldn’t be overridden by packages. Examples >> are ghc-directory, ghc-binary, ghc-bytestring, etc. Since we still >> have packages where these inputs are used, there can be conflicts down >> the line, which are hard to fix. The solution might be to remove ghc-transformers from all packages that currently have it as an input rather than add it wherever the transformers is required. This seems to be only ghc-mtl. I’ve removed it and am building ghc-mtl and ghc-resourcet now. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Oleg Pykhalov writes: > Ricardo Wurmus writes: > >> Marius Bakke writes: >> >>> Should we do a new merge to get the GHC patch, or just merge >>> core-updates and let the problem "fix itself" on 'master'? >> >> I’d prefer building GHC and ghc-resourcet first. We don’t know if this >> patch actually fixes our problems. We should merge master into >> core-updates again. > > I've failed to build ‘ghc-pandoc’, because of dependency ‘ghc-resourcet’ > failed to build on both branches > >- origin/master 369eee8763ca34b427860a86c8fe2db0963b52ae >- origin/core-updates a124e4258ad911e1a65edb6c7d7d8f095249db5f Could you tell me more about the system on which you tried building it? What kernel version did you use and how much memory does the system have? Does the log contain a message about ghc-pkg failing? -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Marius Bakke writes: > >> Should we do a new merge to get the GHC patch, or just merge >> core-updates and let the problem "fix itself" on 'master'? > > I’d prefer building GHC and ghc-resourcet first. We don’t know if this > patch actually fixes our problems. We should merge master into > core-updates again. I've failed to build ‘ghc-pandoc’, because of dependency ‘ghc-resourcet’ failed to build on both branches - origin/master 369eee8763ca34b427860a86c8fe2db0963b52ae - origin/core-updates a124e4258ad911e1a65edb6c7d7d8f095249db5f Oleg. signature.asc Description: PGP signature
Re: heads-up: Haskell updates
Mark H Weaver writes: > Ricardo Wurmus writes: > >> Marius Bakke writes: >> >>> Should we do a new merge to get the GHC patch, or just merge >>> core-updates and let the problem "fix itself" on 'master'? >> >> I’d prefer building GHC and ghc-resourcet first. We don’t know if this >> patch actually fixes our problems. We should merge master into >> core-updates again. > > I did the merge. Thank you, Mark. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Leo Famulari writes: > On Fri, Feb 16, 2018 at 07:12:47PM +0100, Marius Bakke wrote: >> Ricardo Wurmus writes: >> >> > Marius Bakke writes: >> > >> >> Should we do a new merge to get the GHC patch, or just merge >> >> core-updates and let the problem "fix itself" on 'master'? >> > >> > I’d prefer building GHC and ghc-resourcet first. We don’t know if this >> > patch actually fixes our problems. We should merge master into >> > core-updates again. >> >> As per the discussion on #guix, lfam cherry-picked the GHC patch and a >> new and hopefully final Hydra evaluation is underway. > > The Hydra web interface doesn't show that the previously pending > evaluation has been cancelled and a new evaluation started. Can we make > sure it's doing the right thing? All I did was to push the merge to core-updates. I was short on time and didn't look at Hydra at all. At present, there is a core-updates evaluation pending, but it's quite late in the process--the Scheme code has already finished running--so I'm reluctant to cancel this one. At this late phase of the process, I don't know how to discover which commit it corresponds to, but we'll find out when it's done. Mark
Re: heads-up: Haskell updates
On Fri, Feb 16, 2018 at 07:12:47PM +0100, Marius Bakke wrote: > Ricardo Wurmus writes: > > > Marius Bakke writes: > > > >> Should we do a new merge to get the GHC patch, or just merge > >> core-updates and let the problem "fix itself" on 'master'? > > > > I’d prefer building GHC and ghc-resourcet first. We don’t know if this > > patch actually fixes our problems. We should merge master into > > core-updates again. > > As per the discussion on #guix, lfam cherry-picked the GHC patch and a > new and hopefully final Hydra evaluation is underway. The Hydra web interface doesn't show that the previously pending evaluation has been cancelled and a new evaluation started. Can we make sure it's doing the right thing? signature.asc Description: PGP signature
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Marius Bakke writes: > >> Should we do a new merge to get the GHC patch, or just merge >> core-updates and let the problem "fix itself" on 'master'? > > I’d prefer building GHC and ghc-resourcet first. We don’t know if this > patch actually fixes our problems. We should merge master into > core-updates again. I did the merge. Mark
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Marius Bakke writes: > >> Should we do a new merge to get the GHC patch, or just merge >> core-updates and let the problem "fix itself" on 'master'? > > I’d prefer building GHC and ghc-resourcet first. We don’t know if this > patch actually fixes our problems. We should merge master into > core-updates again. As per the discussion on #guix, lfam cherry-picked the GHC patch and a new and hopefully final Hydra evaluation is underway. signature.asc Description: PGP signature
Re: heads-up: Haskell updates
Marius Bakke writes: > Should we do a new merge to get the GHC patch, or just merge > core-updates and let the problem "fix itself" on 'master'? I’d prefer building GHC and ghc-resourcet first. We don’t know if this patch actually fixes our problems. We should merge master into core-updates again. Would you like to give that a try, Marius? -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Danny Milosavljevic writes: > git am just failed because the index is not matching. > > So I'll not push my core-updates. > > Next, I tried git merge origin/master which gave me a number of merge > conflicts. > > One of those is gnu/local.mk: > > + %D%/packages/patches/gettext-multi-core.patch \ > + %D%/packages/patches/gettext-gnulib-multi-core.patch \ > > These referred files haven't been merged into core-updates by my action above. > > Why? If you do `git show a124e4258ad91`, which is the commit that adds "ghc-8.0-fall-back-to-madv_dontneed.patch" to local.mk, you'll see these files in the "context" of the change. However these files have been deleted on core-updates. So when git tries to merge a124e4258ad91, it can't find this "context" and instead adds it so that the commit applies cleanly. The solution in this case is to delete these unwanted lines from local.mk (again) and use "git commit" to conclude the merge. However there are two other conflicts. The emacs.scm one is easy, but the python.scm conflict requires adjusting the bpython context to be after the packages that were recently appended to python.scm on core-updates. Should we do a new merge to get the GHC patch, or just merge core-updates and let the problem "fix itself" on 'master'? signature.asc Description: PGP signature
Re: heads-up: Haskell updates
git am just failed because the index is not matching. So I'll not push my core-updates. Next, I tried git merge origin/master which gave me a number of merge conflicts. One of those is gnu/local.mk: + %D%/packages/patches/gettext-multi-core.patch \ + %D%/packages/patches/gettext-gnulib-multi-core.patch \ These referred files haven't been merged into core-updates by my action above. Why?
Re: heads-up: Haskell updates
Hi Danny, Danny Milosavljevic writes: >> Danny, could you please do this on master and core-updates? > > I've done it on master now. Thank you! > Maybe it's me being used to SVN, but can I git am the commit to > core-updates? Yes. > Wouldn't that cause a conflict on the next merge of master to > core-updates (because of the missing in-betweens) ? It should be fine. Every once in a while someone applies the same patch to both master and core-updates, so we have some experience with this. Thanks, Mark
Re: heads-up: Haskell updates
On Thu, Feb 15, 2018 at 06:03:56PM +0100, Danny Milosavljevic wrote: > Hi Ricardo, > > > Danny, could you please do this on master and core-updates? > > I've done it on master now. > > Maybe it's me being used to SVN, but can I git am the commit to core-updates? > > Wouldn't that cause a conflict on the next merge of master to core-updates > (because of the missing in-betweens) ? Git handles this case in my experience. If you try it and it becomes complicated, you could instead merge master into core-updates in order to get the change on core-updates. If you're not comfortable doing it with Git just let me know and I'll do it. signature.asc Description: PGP signature
Re: heads-up: Haskell updates
Danny Milosavljevic writes: > So the only problematic case is that the build process finds MADV_FREE > but the running Linux doesn't yet have it and a ghc program runs on it. > > Reading MADV_DONTNEED docs again, MADV_DONTNEED pretty much does the same > as MADV_FREE - but MADV_DONTNEED promises to make later accesses to the > range succeed (by providing a new zero-filled page if necessary) while > MADV_FREE promises to make them fail. > > So one could fall back to MADV_DONTNEED - should be fine, though a little > weird > for an allocator. > > If that's the case and the build still fails, let's just apply the Haskell > patch > to ghc (or update ghc if there's a newer release). There’s no newer release we may use. Of course, there’s GHC 8.2.x but the current version for LTSHaskell is 8.0.2. With 8.2.x who knows what other things are broken :) So let’s apply the patch. Danny, could you please do this on master and core-updates? I’d like to merge core-updates this week and it would be great if we could build all of the Haskell packages (and all those R packages that depend on Pandoc) before the merge. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Hi Ricardo, > Danny, could you please do this on master and core-updates? I've done it on master now. Maybe it's me being used to SVN, but can I git am the commit to core-updates? Wouldn't that cause a conflict on the next merge of master to core-updates (because of the missing in-betweens) ?
Re: heads-up: Haskell updates
Mark H Weaver writes: > So far, almost all of the new packages are building successfully on > Hydra, but I see one failure: ghc-resourcet, which in turn causes > r-bookdown to dep-fail: > > https://hydra.gnu.org/build/2495799/nixlog/1/raw Hmm, how odd. I did build ghc-resourcet before, but I may have broken it with later commits. Thanks for letting me know. I’ll try to fix it. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Timothy Sample writes: > Unfortunately, I don’t know enough about Haskell to comment on whether > or not these packages “really shouldn’t be overridden” or not. (It > makes sense to me, but OTOH, why are they all split up and on Hackage if > not for tinkering?) If they really shouldn’t be overridden, then we > should stop doing so! They shouldn’t be overridden, because of problems like this. They are offered on Hackage for developers to play with them, but there is no guarantee that all packages will work with the new versions. That’s precisely why there are efforts like Stackage / LTSHaskell. They agree on one version of GHC and harmonize package versions that work well together. There are much fewer packages that work with the latest version of some module than there are packages that would break. But worst of all is a mix of old and new modules, because that is very likely to cause problems. I have, in fact, already removed some packages from the inputs of Haskell packages as I prepared this big batch of updates. More work like that should be performed. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > I think that’s a misunderstanding. The cause for the error is earlier > when it complains that some packages depend on different versions of the > “transformers” package. “StateT” is a monad transformer. For what it’s worth, I fixed this error on my machine by adding “ghc-transformers” as an input to “ghc-transformers-compat”. (This also fixed “ghc-adjunctions”.) This suggests to me that what you wrote earlier accounts for the problem: > * many problems are caused by the fact that GHC includes a bunch of > packages that really shouldn’t be overridden by packages. Examples > are ghc-directory, ghc-binary, ghc-bytestring, etc. Since we still > have packages where these inputs are used, there can be conflicts down > the line, which are hard to fix. The “ghc-transformers-compat” package was using “transformers” from GHC, while “resourcet” and “adjunctions” used our “ghc-transformers” package from Hackage. Forcing “ghc-transformers-compat” to also use “transformers” from Hackage fixes the problems. I was trying to see if I could get Idris to build, and after fixing these two packages, ran into similar problems with “ghc-binary”. It looks like we might need some kind of policy here. Either we don’t tinker with (override) the built-in GHC packages or we somehow hide them during builds. If we decided not to override them, we should consider deleting all the packages that do so. Then all the dependent packages would pick up their dependencies from GHC (somewhat implicitly). If we hid the built-in packages, it would require that every Haskell package be explicit about which built-in GHC packages it uses (which is not currently the case as demonstrated above). Both of these approaches would probably cause a lot of errors right off the start. The first approach would cause compile-time errors (references to undefined variables in Guile), and the second, build errors. Unfortunately, I don’t know enough about Haskell to comment on whether or not these packages “really shouldn’t be overridden” or not. (It makes sense to me, but OTOH, why are they all split up and on Hackage if not for tinkering?) If they really shouldn’t be overridden, then we should stop doing so! I’m happy to work on this if there is a path forward. Thoughts? -- Tim
Re: heads-up: Haskell updates
On Thu, Feb 15, 2018 at 12:04:04PM +0100, Danny Milosavljevic wrote: > So I suspect that Ricardo has a Linux >= 4.5 but Andreas and Hydra > have a Linux < 4.5. Is that correct? Strangely not. I am running Guix on Debian with a kernel 4.9.0-5-amd64. Andreas
Re: heads-up: Haskell updates
> I think it just didn't update one of the packages or the package database > and now it got the wrong version where someone changed the type signature > on one side and the other side is stale. Or: Configuring resourcet-1.1.7.5... Warning: This package indirectly depends on multiple versions of the same package. This is highly likely to cause a compile failure.
Re: heads-up: Haskell updates
Hi Mark, Hi Ricardo, On Thu, 15 Feb 2018 03:41:33 -0500 Mark H Weaver wrote: > Given that it's using #ifdef to detect this, I'd expect the result to > depend only on the _headers_ being used to compile, and not the actual > kernel running the build. Yes, that's precisely the problem. * The [glibc or kernel] headers determine whether the ghc compilation detects it * The Linux kernel determines whether the functionality is actually present at runtime when any ghc program (like ghc-pkg) runs. Linus takes backward compatibility seriously - the constants or functionality are not going to vanish from Linux later. So the only problematic case is that the build process finds MADV_FREE but the running Linux doesn't yet have it and a ghc program runs on it. Reading MADV_DONTNEED docs again, MADV_DONTNEED pretty much does the same as MADV_FREE - but MADV_DONTNEED promises to make later accesses to the range succeed (by providing a new zero-filled page if necessary) while MADV_FREE promises to make them fail. So one could fall back to MADV_DONTNEED - should be fine, though a little weird for an allocator. If that's the case and the build still fails, let's just apply the Haskell patch to ghc (or update ghc if there's a newer release). > why it works for Ricardo but not for Andreas and Hydra. At runtime it depends on the Linux kernel version present whether the deallocation will work or not. So I suspect that Ricardo has a Linux >= 4.5 but Andreas and Hydra have a Linux < 4.5. Is that correct? Ricardo, you said you built on master - but in master's glibc the situation is the same: ~/x/glibc-2.25$ grep -r MADV_FREE bits ./bits/mman-linux.h:# define MADV_FREE8 /* Free pages only if memory pressure. */ Still would fail for Linux < 4.5. > Note that on our 'master' branch we are using Linux-libre-4.4 kernel > headers, but on 'core-updates' we're using 4.9 kernel headers. [...] > I suppose if one reads the error message literally: > > Control/Monad/Trans/Resource/Internal.hs:302:10: error: > • Could not deduce (MonadBase IO (Strict.StateT s m)) > arising from the superclasses of an instance declaration > from the context: MonadResource m > bound by the instance declaration > at Control/Monad/Trans/Resource/Internal.hs:302:10-63 > • In the instance declaration for > ‘MonadResource (Strict.StateT s m)’ I think it just didn't update one of the packages or the package database and now it got the wrong version where someone changed the type signature on one side and the other side is stale.
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Mark H Weaver writes: > >> I suppose if one reads the error message literally: >> >> Control/Monad/Trans/Resource/Internal.hs:302:10: error: >> • Could not deduce (MonadBase IO (Strict.StateT s m)) >> arising from the superclasses of an instance declaration >> from the context: MonadResource m >> bound by the instance declaration >> at Control/Monad/Trans/Resource/Internal.hs:302:10-63 >> • In the instance declaration for >> ‘MonadResource (Strict.StateT s m)’ >> >> "Could not deduce" might be interpreted to include the case where we >> failed for lack of sufficient memory. They might have written the code >> to catch any error at all, including out of memory errors, and report >> the failure with this error message. > > I think that’s a misunderstanding. The cause for the error is earlier > when it complains that some packages depend on different versions of the > “transformers” package. “StateT” is a monad transformer. > > I think that “ghc-pkg” fails, which leads to some other version of the > transformers package (maybe a default included with GHC) to be used. In > the case where ghc-pkg does not fail (as on my laptop) the proper > version is used, so there are no problems finding the expected > instances. Ah, okay, that sounds plausible. I confess that my previous message was somewhat rushed, without much investigation or thought behind it. Apologies for the noise. Regards, Mark
Re: heads-up: Haskell updates
Mark H Weaver writes: > Note that on our 'master' branch we are using Linux-libre-4.4 kernel > headers, but on 'core-updates' we're using 4.9 kernel headers. > > The recently created evaluation 109913 on Hydra is for core-updates with > the recent Haskell changes included. It'll be interesting to see if the > same failure happens there. I gave the 'ghc-resourcet' jobs a high > priority. Here are the 'ghc-resourcet' jobs for evaluation 109913: > > https://hydra.gnu.org/build/2497604 (x86_64) > https://hydra.gnu.org/build/2498874 (i686) The x86_64 build results are in: it failed with the same errors. Mark
Re: heads-up: Haskell updates
Mark H Weaver writes: > Given that it's using #ifdef to detect this, I'd expect the result to > depend only on the _headers_ being used to compile, and not the actual > kernel running the build. Is that right? If so, this doesn't explain > why it works for Ricardo but not for Andreas and Hydra. > > Note that on our 'master' branch we are using Linux-libre-4.4 kernel > headers, but on 'core-updates' we're using 4.9 kernel headers. FWIW, I only tried to build ghc-resourcet on the master branch, not on core-updates. > I suppose if one reads the error message literally: > > Control/Monad/Trans/Resource/Internal.hs:302:10: error: > • Could not deduce (MonadBase IO (Strict.StateT s m)) > arising from the superclasses of an instance declaration > from the context: MonadResource m > bound by the instance declaration > at Control/Monad/Trans/Resource/Internal.hs:302:10-63 > • In the instance declaration for > ‘MonadResource (Strict.StateT s m)’ > > "Could not deduce" might be interpreted to include the case where we > failed for lack of sufficient memory. They might have written the code > to catch any error at all, including out of memory errors, and report > the failure with this error message. I think that’s a misunderstanding. The cause for the error is earlier when it complains that some packages depend on different versions of the “transformers” package. “StateT” is a monad transformer. I think that “ghc-pkg” fails, which leads to some other version of the transformers package (maybe a default included with GHC) to be used. In the case where ghc-pkg does not fail (as on my laptop) the proper version is used, so there are no problems finding the expected instances. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Hi Danny, Danny Milosavljevic writes: > Hi Mark, > Hi Ricardo, > > On Wed, 14 Feb 2018 20:39:12 +0100 > Ricardo Wurmus wrote: > >> Nor do I see this message: >> >> ghc-pkg: unable to decommit memory: Invalid argument > > Which Linux kernel version does this run on? Last I knew, Hydra's x86 build slaves were all running kernels older than 4.5, but I'm not sure if that's still the case. >> I don’t know what this message means, but the messages about requiring a > > If there's large address space support [1], ghc 8 does its own allocation in a > 1 TB address space. That means it has to tell the kernel when it doesn't > need some chunk anymore - otherwise you're gonna run out of memory. > > It does that using madvise(2). There's two ways it tries to do that: > > (1) MADV_FREE: Signals that "I don't need that range at all anymore". > It usually means Linux will mark those pages free. > > (2) MADV_DONTNEED: Signals that "I don't need that range in the NEAR FUTURE". > It usually means Linux will swap those pages out. > > MADV_FREE was added in Linux 4.5. Haskell uses a #ifdef to detect it. Given that it's using #ifdef to detect this, I'd expect the result to depend only on the _headers_ being used to compile, and not the actual kernel running the build. Is that right? If so, this doesn't explain why it works for Ricardo but not for Andreas and Hydra. Note that on our 'master' branch we are using Linux-libre-4.4 kernel headers, but on 'core-updates' we're using 4.9 kernel headers. The recently created evaluation 109913 on Hydra is for core-updates with the recent Haskell changes included. It'll be interesting to see if the same failure happens there. I gave the 'ghc-resourcet' jobs a high priority. Here are the 'ghc-resourcet' jobs for evaluation 109913: https://hydra.gnu.org/build/2497604 (x86_64) https://hydra.gnu.org/build/2498874 (i686) > Newer glibc (such as the one in core-updates) unconditionally define MADV_FREE > to prevent programs from depending on a specific Linux kernel in this way [2]. > > There's a patch to ghc that falls back to (2) if (1) doesn't work: > > https://git.haskell.org/ghc.git/commitdiff/6576bf83cdf4eac05eb88a24aa934a736c91e3da > > ... but ghc 8.0.2 which we have on core-updates doesn't use it. > It uses either MADV_FREE or MADV_DONTNEED, determined at compile time. > > So if the Linux kernel is < 4.5 that's gonna end very badly. Thanks for the detailed analysis! I suppose if one reads the error message literally: Control/Monad/Trans/Resource/Internal.hs:302:10: error: • Could not deduce (MonadBase IO (Strict.StateT s m)) arising from the superclasses of an instance declaration from the context: MonadResource m bound by the instance declaration at Control/Monad/Trans/Resource/Internal.hs:302:10-63 • In the instance declaration for ‘MonadResource (Strict.StateT s m)’ "Could not deduce" might be interpreted to include the case where we failed for lack of sufficient memory. They might have written the code to catch any error at all, including out of memory errors, and report the failure with this error message. What do you think? Mark
Re: heads-up: Haskell updates
Hi Mark, Hi Ricardo, On Wed, 14 Feb 2018 20:39:12 +0100 Ricardo Wurmus wrote: > Nor do I see this message: > > ghc-pkg: unable to decommit memory: Invalid argument Which Linux kernel version does this run on? > I don’t know what this message means, but the messages about requiring a If there's large address space support [1], ghc 8 does its own allocation in a 1 TB address space. That means it has to tell the kernel when it doesn't need some chunk anymore - otherwise you're gonna run out of memory. It does that using madvise(2). There's two ways it tries to do that: (1) MADV_FREE: Signals that "I don't need that range at all anymore". It usually means Linux will mark those pages free. (2) MADV_DONTNEED: Signals that "I don't need that range in the NEAR FUTURE". It usually means Linux will swap those pages out. MADV_FREE was added in Linux 4.5. Haskell uses a #ifdef to detect it. Newer glibc (such as the one in core-updates) unconditionally define MADV_FREE to prevent programs from depending on a specific Linux kernel in this way [2]. There's a patch to ghc that falls back to (2) if (1) doesn't work: https://git.haskell.org/ghc.git/commitdiff/6576bf83cdf4eac05eb88a24aa934a736c91e3da ... but ghc 8.0.2 which we have on core-updates doesn't use it. It uses either MADV_FREE or MADV_DONTNEED, determined at compile time. So if the Linux kernel is < 4.5 that's gonna end very badly. For the record: https://ghc.haskell.org/trac/ghc/ticket/12865 Also https://github.com/NixOS/nixpkgs/issues/18118 mmap has a flag MAP_HUGETLB which would cause it to use a mounted hugetlbfs (the cgroup of which I advised to remove from GuixSD from the time being). ghc 8 does not use it so we are safe there. [1] use_large_address_space=no if test "$ac_cv_sizeof_void_p" -eq 8 ; then if test "x$EnableLargeAddressSpace" = "xyes" ; then if test "$ghc_host_os" = "darwin" ; then use_large_address_space=yes elif test "$ghc_host_os" = "openbsd" ; then # as of OpenBSD 5.8 (2015), OpenBSD does not support mmap with MAP_NORESERVE. # The flag MAP_NORESERVE is supported for source compatibility reasons, # but is completely ignored by OS mmap use_large_address_space=no else AC_CHECK_DECLS([MAP_NORESERVE, MADV_FREE, MADV_DONTNEED],[],[], [ #include #include #include #include ]) if test "$ac_cv_have_decl_MAP_NORESERVE" = "yes" && test "$ac_cv_have_decl_MADV_FREE" = "yes" || test "$ac_cv_have_decl_MADV_DONTNEED" = "yes" ; then use_large_address_space=yes fi fi fi fi if test "$use_large_address_space" = "yes" ; then AC_DEFINE([USE_LARGE_ADDRESS_SPACE], [1], [Enable single heap address space support]) fi madvise: EINVAL addr is not page-aligned or length is negative. EINVAL advice is not a valid. EINVAL advice is MADV_DONTNEED or MADV_REMOVE and the specified address range includes locked, Huge TLB pages, or VM_PFNMAP pages. EINVAL advice is MADV_MERGEABLE or MADV_UNMERGEABLE, but the kernel was not configured with CONFIG_KSM. EINVAL advice is MADV_FREE or MADV_WIPEONFORK but the specified address range includes file, Huge TLB, MAP_SHARED, or VM_PFNMAP ranges. [2] https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=981569c74cbb6bafa2ddcefa6dd9dbdc938ff1c8
Re: heads-up: Haskell updates
Andreas Enge writes: > On Wed, Feb 14, 2018 at 04:23:47PM -0500, Mark H Weaver wrote: >> Ricardo Wurmus writes: >> > I’m afraid I cannot reproduce this. I removed the successfully built >> > ghc-resourcet from the store and rebuilt it successfully. >> >> FWIW, on Hydra, the same failure happened on both x86_64 and i686: >> >> https://hydra.gnu.org/build/2495799/nixlog/1/raw (x86_64) >> https://hydra.gnu.org/build/2496364/nixlog/1/raw (i686) > > as an additional data point, it also fails on my x86_64 laptop. So the problem > is not specific to hydra. Which kernel version are you using? Here’s a mention of the “unable to decommit memory” bug: https://github.com/NixOS/nixpkgs/issues/18118 This suggests that it’s something to do with a new glibc and/or an older kernel, or maybe a GHC configuration problem. FWIW: I’m using Linux libre 4.15.2-gnu (x86_64) with 16GB of RAM. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Hello, On Wed, Feb 14, 2018 at 04:23:47PM -0500, Mark H Weaver wrote: > Ricardo Wurmus writes: > > I’m afraid I cannot reproduce this. I removed the successfully built > > ghc-resourcet from the store and rebuilt it successfully. > > FWIW, on Hydra, the same failure happened on both x86_64 and i686: > > https://hydra.gnu.org/build/2495799/nixlog/1/raw (x86_64) > https://hydra.gnu.org/build/2496364/nixlog/1/raw (i686) as an additional data point, it also fails on my x86_64 laptop. So the problem is not specific to hydra. Andreas
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Ricardo Wurmus writes: > >> Mark H Weaver writes: >> >>> So far, almost all of the new packages are building successfully on >>> Hydra, but I see one failure: ghc-resourcet, which in turn causes >>> r-bookdown to dep-fail: >>> >>> https://hydra.gnu.org/build/2495799/nixlog/1/raw >> >> Hmm, how odd. I did build ghc-resourcet before, but I may have broken >> it with later commits. >> >> Thanks for letting me know. I’ll try to fix it. > > I’m afraid I cannot reproduce this. I removed the successfully built > ghc-resourcet from the store and rebuilt it successfully. FWIW, on Hydra, the same failure happened on both x86_64 and i686: https://hydra.gnu.org/build/2495799/nixlog/1/raw (x86_64) https://hydra.gnu.org/build/2496364/nixlog/1/raw (i686) Mark
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > My local log looks rather different (see attached file). mlrjkywz6grnmf84gwmy3ggx1zglkd-ghc-resourcet-1.1.7.5.drv.bz2 Description: Binary data -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Ricardo Wurmus writes: > Mark H Weaver writes: > >> So far, almost all of the new packages are building successfully on >> Hydra, but I see one failure: ghc-resourcet, which in turn causes >> r-bookdown to dep-fail: >> >> https://hydra.gnu.org/build/2495799/nixlog/1/raw > > Hmm, how odd. I did build ghc-resourcet before, but I may have broken > it with later commits. > > Thanks for letting me know. I’ll try to fix it. I’m afraid I cannot reproduce this. I removed the successfully built ghc-resourcet from the store and rebuilt it successfully. There are a bunch of warnings, but I end up with a successful build of /gnu/store/y8rp418ynjb57hv824b7apih5bmpibba-ghc-resourcet-1.1.7.5. Thoughout the build it mentions /gnu/store/ngqrlni2iqs8lfbbjf1bd55bymc689d3-ghc-resourcet-1.1.7.5 as the output directory, which matches the build log on hydra. My local log looks rather different (see attached file). For one, I don’t see these messages: --8<---cut here---start->8--- Configuring resourcet-1.1.7.5... Warning: This package indirectly depends on multiple versions of the same package. This is highly likely to cause a compile failure. package transformers-compat-0.5.1.4 requires transformers-0.5.2.0 package transformers-base-0.4.4 requires transformers-0.5.2.0 package primitive-0.6.3.0 requires transformers-0.5.2.0 package monad-control-1.0.1.0 requires transformers-0.5.2.0 package hspec-core-2.2.4 requires transformers-0.5.2.0 package hspec-2.2.4 requires transformers-0.5.2.0 package QuickCheck-2.10.1 requires transformers-0.5.2.0 package resourcet-1.1.7.5 requires transformers-0.5.2.0 package mtl-2.2.1 requires transformers-0.5.2.0 package mmorph-1.0.6 requires transformers-0.5.2.0 package exceptions-0.8.3 requires transformers-0.5.2.0 --8<---cut here---end--->8--- Nor do I see this message: ghc-pkg: unable to decommit memory: Invalid argument I don’t know what this message means, but the messages about requiring a different version of the transformers package probably describe the problem on Hydra. For some reason the build environment I have on my machine is not the same as that on Hydra. I guess that this is related to ghc-pkg failing to set up the environment. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Hi Ricardo, Ricardo Wurmus writes: > I’ve just pushed a very large number of updates to Haskell packages and > switched to GHC 8 as the default. Wow, it's an impressive amount of work, kudos to you! So far, almost all of the new packages are building successfully on Hydra, but I see one failure: ghc-resourcet, which in turn causes r-bookdown to dep-fail: https://hydra.gnu.org/build/2495799/nixlog/1/raw There are many errors similar to these: --8<---cut here---start->8--- Control/Monad/Trans/Resource/Internal.hs:302:10: error: • Could not deduce (MonadBase IO (Strict.StateT s m)) arising from the superclasses of an instance declaration from the context: MonadResource m bound by the instance declaration at Control/Monad/Trans/Resource/Internal.hs:302:10-63 • In the instance declaration for ‘MonadResource (Strict.StateT s m)’ Control/Monad/Trans/Resource/Internal.hs:303:10: error: • Could not deduce (MonadBase IO (Strict.WriterT w m)) arising from the superclasses of an instance declaration from the context: (Monoid w, MonadResource m) bound by the instance declaration at Control/Monad/Trans/Resource/Internal.hs:303:10-75 • In the instance declaration for ‘MonadResource (Strict.WriterT w m)’ phase `build' failed after 1.5 seconds builder for `/gnu/store/r9mlrjkywz6grnmf84gwmy3ggx1zglkd-ghc-resourcet-1.1.7.5.drv' failed with exit code 1 @ build-failed /gnu/store/r9mlrjkywz6grnmf84gwmy3ggx1zglkd-ghc-resourcet-1.1.7.5.drv - 1 builder for `/gnu/store/r9mlrjkywz6grnmf84gwmy3ggx1zglkd-ghc-resourcet-1.1.7.5.drv' failed with exit code 1 --8<---cut here---end--->8--- Thanks for working on it! Mark
Re: heads-up: Haskell updates
Ludovic Courtès writes: >> * updating Haskell packages automatically is dangerous as not all >> packages work well together. When updating I often had to take a few >> steps back to reduce the version number. On Hackage I picked the LTS >> version where available. > > Does that mean that Hackage provides a package set that doesn’t work > well together? Or is it a defect in our updater? Packages on Hackage are not guaranteed to work well together. Having impossible version constraints in any larger set of packages has a long tradition in the Haskell world. That’s what led to Stackage and LTSHaskell, which both work on ensuring package compatibility. > I think it would be great if running “guix refresh -t hackage” would > give us a package set that works together, provided Hackage does the > necessary QA. (I thought it did because Nixpkgs imports all of it > wholesale AIUI.) Unfortunately, that’s not the case. Nixpkgs provides the latest versions (if the “Distributions” row on Hackage pages is to be believed), but we really want to provide the version that LTSHaskell or Stackage use. The latest version is only of interest to developers. We have a Stackage importer but I haven’t been able to make it work for this round of updates. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: heads-up: Haskell updates
Hello, Ricardo Wurmus skribis: > I’ve just pushed a very large number of updates to Haskell packages and > switched to GHC 8 as the default. Thanks for the heroic work! > * updating Haskell packages automatically is dangerous as not all > packages work well together. When updating I often had to take a few > steps back to reduce the version number. On Hackage I picked the LTS > version where available. Does that mean that Hackage provides a package set that doesn’t work well together? Or is it a defect in our updater? I think it would be great if running “guix refresh -t hackage” would give us a package set that works together, provided Hackage does the necessary QA. (I thought it did because Nixpkgs imports all of it wholesale AIUI.) > * this is based on my previous work from October 2016. Only few > packages had been updated since then, so most of my changes still > applied. When it wasn’t necessary I didn’t bother updating my > updates. This means that a second pass could be useful to update > packages that are below their LTS versions. > > In general I think we really need someone who feels responsible for > *all* the Haskell packages. It’s not okay to keep most of them at old > versions for over a year. Agreed, we need a Haskell champion to take care of this—not necessarily to do all the actual work themself, but rather to keep track of what’s lagging behind, what needs to be done, and to coordinate efforts. My impression is that people have been willing to help on this in the past but didn’t necessarily know what upgrading would entail. If you’re reading this and feel familiar with Haskell’s infrastructure, please don’t hesitate to chime in! Thanks, Ludo’.
heads-up: Haskell updates
Hi Guix, I’ve just pushed a very large number of updates to Haskell packages and switched to GHC 8 as the default. I have built almost all of these updated packages and some packages that depend on them, including r-rmarkdown, hisat, darcs, xmonad, and r-rcas. One notable exception is idris — I could not make it build at all. Neither the current version, nor any of the following versions up to 1.2.0. Help in this area is appreciated. Some notes: * updating Haskell packages automatically is dangerous as not all packages work well together. When updating I often had to take a few steps back to reduce the version number. On Hackage I picked the LTS version where available. * this is based on my previous work from October 2016. Only few packages had been updated since then, so most of my changes still applied. When it wasn’t necessary I didn’t bother updating my updates. This means that a second pass could be useful to update packages that are below their LTS versions. In general I think we really need someone who feels responsible for *all* the Haskell packages. It’s not okay to keep most of them at old versions for over a year. * many problems are caused by the fact that GHC includes a bunch of packages that really shouldn’t be overridden by packages. Examples are ghc-directory, ghc-binary, ghc-bytestring, etc. Since we still have packages where these inputs are used, there can be conflicts down the line, which are hard to fix. * I liberally added the “--allow-newer” configure flag to packages that have strict version constraints. In most cases that was to allow for a later version of QuickCheck. * If you find that anything is broken now that worked before, please coordinate updates and fixes on guix-devel@gnu.org. I hope this big blob of changes won’t inconvenience you too much. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
