Re: TLS certificates for NFS-over-TLS floating client
On Wed, 04 Mar 2020 13:35:15 +0900 (JST) Hiroki Sato h...@freebsd.org said Rick Macklem wrote in : rm> Hi, rm> rm> I am slowly trying to understand TLS certificates and am trying to figure rm> out how to do the following: rm> -> For an /etc/exports file with... rm> /home -tls -network 192.168.1.0 -mask 255.255.255.0 rm> /home -tlscert rm> rm> This syntax isn't implemented yet, but the thinking is that clients on the rm> 192.168.1 subnet would use TLS, but would not require a certificate. rm> For access from anywhere else, the client(s) would be required to have a rm> certificate. rm> rm> A typical client mounting from outside of the subnet might be my laptop, rm> which is using wifi and has no fixed IP/DNS name. rm> --> How do you create a certificate that the laptop can use, which the NFS rm> server can trust enough to allow the mount? rm> My thinking is that a "secret" value can be put in the certificate that the NFS rm> server can check for. I do not think it is a good idea to use a certificate with an embedded secret for authentication and/or authorization. In the case that the client offers a certificate upon establishing a TLS connection for authentication purpose, the authenticity will be checked on the server usually by using the CA certificate which was used to issue the client certificate. The CA cert must be put to somewhere the NFS server can read. The CA cert is secret. So if the NFS server can check the client certificate by the CA certificate, it means the NFS server can trust the client. I think no additional information is required. Authorization such as which mount point can be mounted by using the client cert can be implemented by using the CN field or other X.509 attributes, of course. It can be just a clear text. I think this is one of the most reliable and straightforward ways because in most cases both NFS servers and the clients are under the sysadmin's control. rm> Now, I'm not sure, but I don't think this certificate can be created via rm> a trust authority such that it would "verify". However, the server can rm> look for the "secret" in the certificate and allow the mount based on that. In the way described above, to use TLS client authentication, the NFS server admin has to have a certificate which allows to sign another certificate. This means that the admin must be a CA or trusted authority. In practice, one can generate a self-signed certificate by using openssl(1) and use it as its CA certificate. He can issue certificates signed by it for the NFS clients, and put his CA certificate to somewhere the NFS server can read. rm> Also, even if the NFS client/server have fixed IP addresses with well known rm> DNS names, it isn't obvious to me how signed certificates can be acquired rm> for them? rm> (Lets Encrypt expects the Acme protocol to work and that seems to be rm> web site/http specific?) TLS certificates do not always have (or do not need to have) a domain name as an attribute. Data in attributes are restricted depending on the purpose, so certificates issued by Let's Encrypt can have only domain names (CN or Subject Alternative Name), for example. An example which is not supported by Let's Encrypt is a certificate for S/MIME email encryption which has an email address. FWIW here's an example from the headers coming from this list. Received: from mx1.freebsd.org (mx1.freebsd.org [96.47.72.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx2.freebsd.org (Postfix) with ESMTPS id 1B07B7E9A8; Wed, 4 Mar 2020 04:37:12 + (UTC) (envelope-from owner-freebsd-curr...@freebsd.org) Not sure if it would help with your intent here. But there it is. :) --Chris -- Hiroki ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: TLS certificates for NFS-over-TLS floating client
Rick Macklem wrote in : rm> Hi, rm> rm> I am slowly trying to understand TLS certificates and am trying to figure rm> out how to do the following: rm> -> For an /etc/exports file with... rm> /home -tls -network 192.168.1.0 -mask 255.255.255.0 rm> /home -tlscert rm> rm> This syntax isn't implemented yet, but the thinking is that clients on the rm> 192.168.1 subnet would use TLS, but would not require a certificate. rm> For access from anywhere else, the client(s) would be required to have a rm> certificate. rm> rm> A typical client mounting from outside of the subnet might be my laptop, rm> which is using wifi and has no fixed IP/DNS name. rm> --> How do you create a certificate that the laptop can use, which the NFS rm>server can trust enough to allow the mount? rm> My thinking is that a "secret" value can be put in the certificate that the NFS rm> server can check for. I do not think it is a good idea to use a certificate with an embedded secret for authentication and/or authorization. In the case that the client offers a certificate upon establishing a TLS connection for authentication purpose, the authenticity will be checked on the server usually by using the CA certificate which was used to issue the client certificate. The CA cert must be put to somewhere the NFS server can read. The CA cert is secret. So if the NFS server can check the client certificate by the CA certificate, it means the NFS server can trust the client. I think no additional information is required. Authorization such as which mount point can be mounted by using the client cert can be implemented by using the CN field or other X.509 attributes, of course. It can be just a clear text. I think this is one of the most reliable and straightforward ways because in most cases both NFS servers and the clients are under the sysadmin's control. rm> Now, I'm not sure, but I don't think this certificate can be created via rm> a trust authority such that it would "verify". However, the server can rm> look for the "secret" in the certificate and allow the mount based on that. In the way described above, to use TLS client authentication, the NFS server admin has to have a certificate which allows to sign another certificate. This means that the admin must be a CA or trusted authority. In practice, one can generate a self-signed certificate by using openssl(1) and use it as its CA certificate. He can issue certificates signed by it for the NFS clients, and put his CA certificate to somewhere the NFS server can read. rm> Also, even if the NFS client/server have fixed IP addresses with well known rm> DNS names, it isn't obvious to me how signed certificates can be acquired rm> for them? rm> (Lets Encrypt expects the Acme protocol to work and that seems to be rm> web site/http specific?) TLS certificates do not always have (or do not need to have) a domain name as an attribute. Data in attributes are restricted depending on the purpose, so certificates issued by Let's Encrypt can have only domain names (CN or Subject Alternative Name), for example. An example which is not supported by Let's Encrypt is a certificate for S/MIME email encryption which has an email address. -- Hiroki pgp4efev_ZHYw.pgp Description: PGP signature
Re: Pkg repository is broken...
On Monday, 2 March 2020 at 17:58:01 +, marco wrote: > On Sun, Mar 01, 2020 at 04:50:59PM -0500, you (Brennan Vincent) sent the > following to [freebsd-current] : >> Apparently something has its ABI erroneously listed as FreeBSD:13.0:amd64 >> instead of FreeBSD:13:amd64. >> >> ``` >> $ sudo pkg update -f >> Updating FreeBSD repository catalogue... >> Fetching meta.conf: 100%163 B 0.2kB/s00:01 >> Fetching packagesite.txz: 100%6 MiB 6.4MB/s00:01 >> Processing entries: 72% >> pkg: wrong architecture: FreeBSD:13.0:amd64 instead of FreeBSD:13:amd64 >> pkg: repository FreeBSD contains packages with wrong ABI: FreeBSD:13.0:amd64 >> Processing entries: 100% >> Unable to update repository FreeBSD >> Error updating repositories! > > Ran into this very same problem today too. > Just learned on #freebsd that the repos are temporarily borked and > people are working hard to fix it. Any workarounds in the meantime? This must affect a lot of people, including those who use 12-: pkg: wrong architecture: FreeBSD:12.0:amd64 instead of FreeBSD:12:amd64 pkg: repository FreeBSD contains packages with wrong ABI: FreeBSD:12.0:amd64 Greg -- Sent from my desktop computer. Finger g...@freebsd.org for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA signature.asc Description: PGP signature
Re: TLS certificates for NFS-over-TLS floating client
On Wed, 4 Mar 2020 03:15:48 + Rick Macklem rmack...@uoguelph.ca said Hi, I am slowly trying to understand TLS certificates and am trying to figure out how to do the following: -> For an /etc/exports file with... /home -tls -network 192.168.1.0 -mask 255.255.255.0 /home -tlscert This syntax isn't implemented yet, but the thinking is that clients on the 192.168.1 subnet would use TLS, but would not require a certificate. For access from anywhere else, the client(s) would be required to have a certificate. A typical client mounting from outside of the subnet might be my laptop, which is using wifi and has no fixed IP/DNS name. --> How do you create a certificate that the laptop can use, which the NFS server can trust enough to allow the mount? My thinking is that a "secret" value can be put in the certificate that the NFS server can check for. The simplest way would be a fairly long list of random characters in the organizationName and/or organizationUnitName field(s) of the subject name. Alternately, it could be a newly defined extension for X509v3, I think? Now, I'm not sure, but I don't think this certificate can be created via a trust authority such that it would "verify". However, the server can look for the "secret" in the certificate and allow the mount based on that. Does this sound reasonable? Also, even if the NFS client/server have fixed IP addresses with well known DNS names, it isn't obvious to me how signed certificates can be acquired for them? (Lets Encrypt expects the Acme protocol to work and that seems to be web site/http specific?) I can only speak to the LetsEncrypt part of your question(s); There are additional verification methods available beyond www/httpd(s). But I found in the case of (e)mail; that the cert(s) issued by LetsEncrypt also work well for all my MXs. Hope this is helpful! Thanks for any help with this, rick --Chris ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: TLS certificates for NFS-over-TLS floating client
On Wed, Mar 04, 2020 at 03:15:48AM +, Rick Macklem wrote: > Hi, > > I am slowly trying to understand TLS certificates and am trying to figure > out how to do the following: > -> For an /etc/exports file with... > /home -tls -network 192.168.1.0 -mask 255.255.255.0 > /home -tlscert > > This syntax isn't implemented yet, but the thinking is that clients on the > 192.168.1 subnet would use TLS, but would not require a certificate. > For access from anywhere else, the client(s) would be required to have a > certificate. My gut reaction: that doesn't sound like a good idea. Trusting the local network to be secure is pretty risky, in general. > A typical client mounting from outside of the subnet might be my laptop, > which is using wifi and has no fixed IP/DNS name. > --> How do you create a certificate that the laptop can use, which the NFS >server can trust enough to allow the mount? You can give your laptop a certificate for an arbitrary name, provided that the NFS server knows to "validate" that name in an appropriate fashion. (I don't remember what draft-ietf-nfsv4-rpc-tls says about this validation.) > My thinking is that a "secret" value can be put in the certificate that the > NFS > server can check for. > The simplest way would be a fairly long list of random characters in the > organizationName and/or organizationUnitName field(s) of the subject name. > Alternately, it could be a newly defined extension for X509v3, I think? It would be better to just make a site-local CA and trust everything it issues (which, admittedly, is not the greatest option itself.) > Now, I'm not sure, but I don't think this certificate can be created via > a trust authority such that it would "verify". However, the server can > look for the "secret" in the certificate and allow the mount based on that. > > Does this sound reasonable? I'm not sure what goal you're trying to achieve by this "security through obscurity". > Also, even if the NFS client/server have fixed IP addresses with well known > DNS names, it isn't obvious to me how signed certificates can be acquired > for them? > (Lets Encrypt expects the Acme protocol to work and that seems to be > web site/http specific?) RFC 8738 specifies the ACME protocol for validating IP addresses. -Ben ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
TLS certificates for NFS-over-TLS floating client
Hi, I am slowly trying to understand TLS certificates and am trying to figure out how to do the following: -> For an /etc/exports file with... /home -tls -network 192.168.1.0 -mask 255.255.255.0 /home -tlscert This syntax isn't implemented yet, but the thinking is that clients on the 192.168.1 subnet would use TLS, but would not require a certificate. For access from anywhere else, the client(s) would be required to have a certificate. A typical client mounting from outside of the subnet might be my laptop, which is using wifi and has no fixed IP/DNS name. --> How do you create a certificate that the laptop can use, which the NFS server can trust enough to allow the mount? My thinking is that a "secret" value can be put in the certificate that the NFS server can check for. The simplest way would be a fairly long list of random characters in the organizationName and/or organizationUnitName field(s) of the subject name. Alternately, it could be a newly defined extension for X509v3, I think? Now, I'm not sure, but I don't think this certificate can be created via a trust authority such that it would "verify". However, the server can look for the "secret" in the certificate and allow the mount based on that. Does this sound reasonable? Also, even if the NFS client/server have fixed IP addresses with well known DNS names, it isn't obvious to me how signed certificates can be acquired for them? (Lets Encrypt expects the Acme protocol to work and that seems to be web site/http specific?) Thanks for any help with this, rick ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: NCURSES
On 03/03/20 15:12, Filippo Moretti wrote: > Dear Sir, I would like to update my CURRENT-AMD64.The ports that > depend from ncurses should be rebuilt.How do I find out which ports I need to > rebuild?Thank youFilippo check pkg-check(8) something like 'pkg check -Ba' should work. -- Guido Falsi ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
NCURSES
Dear Sir, I would like to update my CURRENT-AMD64.The ports that depend from ncurses should be rebuilt.How do I find out which ports I need to rebuild?Thank youFilippo ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Wired memory consumes almost all my memory recently
On 3/3/20 11:41 AM, Kirill Ponomarev wrote: On 03/03, Mateusz Piotrowski wrote: I'm experiencing some problems with memory on r358410. Please try r358505, it should be fixed. Thanks! I'm running r358571 now. I'll post again if the problem persists. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: can't build rust -- out of swap space
On Tue, 3 Mar 2020 15:31+0300, Yuri Pankov wrote: > On 03.03.2020 15:27, Trond Endrestøl wrote: > > On Tue, 3 Mar 2020 12:56+0300, Yuri Pankov wrote: > > > > > On 03.03.2020 12:44, Trond Endrestøl wrote: > > > > On Tue, 3 Mar 2020 12:40+0300, Yuri Pankov wrote: > > > > > > > > > On 03.03.2020 11:49, Yuri Pankov wrote: > > > > > > With recent pkg fallout, I'm trying to build rust myself first time > > > > > > ever > > > > > > (as > > > > > > far as I can remember), and it's failing running out of swap on the > > > > > > following step: > > > > > > > > > > > > Building stage0 std artifacts (x86_64-unknown-freebsd -> > > > > > > x86_64-unknown-freebsd) > > > > > > running: > > > > > > "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" > > > > > > "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" > > > > > > "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" > > > > > > "--features" > > > > > > "panic-unwind backtrace compiler-builtins-c" "--manifest-path" > > > > > > "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" > > > > > > "--message-format" "json-render-diagnostics" > > > > > > ^C^C^C > > > > > > Build completed unsuccessfully in 0:00:55 > > > > > > > > > > > > Here I pressed ^C as the build actually continues despite several > > > > > > rustdoc, > > > > > > python, and other processes being killed. > > > > > > > > > > > > swap_pager: out of swap space > > > > > > swp_pager_getswapspace(20): failed > > > > > > swap_pager: out of swap space > > > > > > swp_pager_getswapspace(11): failed > > > > > > > > > > > > The system has 32G of RAM and 2GB swap partition (as advised by > > > > > > zfs-auto > > > > > > installation option), top shows about 28G of memory free at that > > > > > > moment, > > > > > > so > > > > > > I'm wondering why the swap is being used, and if 2G should be enough > > > > > > to > > > > > > build rust. > > > > > > > > > > Looks like I got this wrong, adding a file-backed swap space I was > > > > > actually > > > > > able to run top, and seeing only 100M of memory being "Free", ~20G > > > > > memory > > > > > reported as "Active", and swap usage constantly growing being consumed > > > > > by > > > > > rustdoc process; something is really wrong here. > > > > > > > > Run top(1), hit the o key, type in size, and hit enter to have top > > > > sort the process list according to their virtual size. The culprit > > > > will eventually work its way to the top. > > > > > > Yes, it's rustdoc, and I'm seeing the same behavior as you described in > > > your > > > other reply. What's more interesting, having a little swap, processes get > > > killed almost immediately, and the build happily continues resulting > > > successful rust package. > > > > In my case, I see this behaviour twice during the build. I'll try and > > reduce my swap partition to 2 GiB and see if that makes a difference > > during the build. > > I wonder if you are seeing the problem for a long time, or it's something > recent for you? I just tried reinstalling the system from 20200227 snapshot, > and NOT seeing it, with or w/o the swap. Once I have everything installed, > I'll update to latest and re-check (note that I had WITH_CTF defined in > src.conf, though I doubt it's related). I have struggled for the past three weeks, with mad hypotheses ranging from ccache poisoning to faulty memory sticks, culminating in bootstrapping my localbase from scratch last weekend. One positive outcome is that I finally raised the size limit for my localbase ccache. -- Trond. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: can't build rust -- out of swap space
In message <993c159b-b9e0-cd47-5df6-73288748a...@fastmail.com>, Yuri Pankov wri tes: > On 03.03.2020 15:27, Trond Endrestøl wrote: > > On Tue, 3 Mar 2020 12:56+0300, Yuri Pankov wrote: > > > >> On 03.03.2020 12:44, Trond Endrestøl wrote: > >>> On Tue, 3 Mar 2020 12:40+0300, Yuri Pankov wrote: > >>> > On 03.03.2020 11:49, Yuri Pankov wrote: > > With recent pkg fallout, I'm trying to build rust myself first time eve > r > > (as > > far as I can remember), and it's failing running out of swap on the > > following step: > > > > Building stage0 std artifacts (x86_64-unknown-freebsd -> > > x86_64-unknown-freebsd) > > running: > > "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebs > d/stage0/bin/cargo" > > "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" > > "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" "--features > " > > "panic-unwind backtrace compiler-builtins-c" "--manifest-path" > > "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" > > "--message-format" "json-render-diagnostics" > > ^C^C^C > > Build completed unsuccessfully in 0:00:55 > > > > Here I pressed ^C as the build actually continues despite several > > rustdoc, > > python, and other processes being killed. > > > > swap_pager: out of swap space > > swp_pager_getswapspace(20): failed > > swap_pager: out of swap space > > swp_pager_getswapspace(11): failed > > > > The system has 32G of RAM and 2GB swap partition (as advised by zfs-aut > o > > installation option), top shows about 28G of memory free at that moment > , > > so > > I'm wondering why the swap is being used, and if 2G should be enough to > > build rust. > > Looks like I got this wrong, adding a file-backed swap space I was > actually > able to run top, and seeing only 100M of memory being "Free", ~20G memor > y > reported as "Active", and swap usage constantly growing being consumed b > y > rustdoc process; something is really wrong here. > >>> > >>> Run top(1), hit the o key, type in size, and hit enter to have top > >>> sort the process list according to their virtual size. The culprit > >>> will eventually work its way to the top. > >> > >> Yes, it's rustdoc, and I'm seeing the same behavior as you described in yo > ur > >> other reply. What's more interesting, having a little swap, processes get > >> killed almost immediately, and the build happily continues resulting > >> successful rust package. > > > > In my case, I see this behaviour twice during the build. I'll try and > > reduce my swap partition to 2 GiB and see if that makes a difference > > during the build. > > I wonder if you are seeing the problem for a long time, or it's > something recent for you? I just tried reinstalling the system from > 20200227 snapshot, and NOT seeing it, with or w/o the swap. Once I have > everything installed, I'll update to latest and re-check (note that I > had WITH_CTF defined in src.conf, though I doubt it's related). It's been a problem since rust-1.41 was committed. Your choices are, - revert back to rust-1.40, or - use rust-nightly for now. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: can't build rust -- out of swap space
On 03.03.2020 15:27, Trond Endrestøl wrote: On Tue, 3 Mar 2020 12:56+0300, Yuri Pankov wrote: On 03.03.2020 12:44, Trond Endrestøl wrote: On Tue, 3 Mar 2020 12:40+0300, Yuri Pankov wrote: On 03.03.2020 11:49, Yuri Pankov wrote: With recent pkg fallout, I'm trying to build rust myself first time ever (as far as I can remember), and it's failing running out of swap on the following step: Building stage0 std artifacts (x86_64-unknown-freebsd -> x86_64-unknown-freebsd) running: "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" "--message-format" "json-render-diagnostics" ^C^C^C Build completed unsuccessfully in 0:00:55 Here I pressed ^C as the build actually continues despite several rustdoc, python, and other processes being killed. swap_pager: out of swap space swp_pager_getswapspace(20): failed swap_pager: out of swap space swp_pager_getswapspace(11): failed The system has 32G of RAM and 2GB swap partition (as advised by zfs-auto installation option), top shows about 28G of memory free at that moment, so I'm wondering why the swap is being used, and if 2G should be enough to build rust. Looks like I got this wrong, adding a file-backed swap space I was actually able to run top, and seeing only 100M of memory being "Free", ~20G memory reported as "Active", and swap usage constantly growing being consumed by rustdoc process; something is really wrong here. Run top(1), hit the o key, type in size, and hit enter to have top sort the process list according to their virtual size. The culprit will eventually work its way to the top. Yes, it's rustdoc, and I'm seeing the same behavior as you described in your other reply. What's more interesting, having a little swap, processes get killed almost immediately, and the build happily continues resulting successful rust package. In my case, I see this behaviour twice during the build. I'll try and reduce my swap partition to 2 GiB and see if that makes a difference during the build. I wonder if you are seeing the problem for a long time, or it's something recent for you? I just tried reinstalling the system from 20200227 snapshot, and NOT seeing it, with or w/o the swap. Once I have everything installed, I'll update to latest and re-check (note that I had WITH_CTF defined in src.conf, though I doubt it's related). ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: can't build rust -- out of swap space
On Tue, 3 Mar 2020 12:56+0300, Yuri Pankov wrote: > On 03.03.2020 12:44, Trond Endrestøl wrote: > > On Tue, 3 Mar 2020 12:40+0300, Yuri Pankov wrote: > > > > > On 03.03.2020 11:49, Yuri Pankov wrote: > > > > With recent pkg fallout, I'm trying to build rust myself first time ever > > > > (as > > > > far as I can remember), and it's failing running out of swap on the > > > > following step: > > > > > > > > Building stage0 std artifacts (x86_64-unknown-freebsd -> > > > > x86_64-unknown-freebsd) > > > > running: > > > > "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" > > > > "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" > > > > "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" "--features" > > > > "panic-unwind backtrace compiler-builtins-c" "--manifest-path" > > > > "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" > > > > "--message-format" "json-render-diagnostics" > > > > ^C^C^C > > > > Build completed unsuccessfully in 0:00:55 > > > > > > > > Here I pressed ^C as the build actually continues despite several > > > > rustdoc, > > > > python, and other processes being killed. > > > > > > > > swap_pager: out of swap space > > > > swp_pager_getswapspace(20): failed > > > > swap_pager: out of swap space > > > > swp_pager_getswapspace(11): failed > > > > > > > > The system has 32G of RAM and 2GB swap partition (as advised by zfs-auto > > > > installation option), top shows about 28G of memory free at that moment, > > > > so > > > > I'm wondering why the swap is being used, and if 2G should be enough to > > > > build rust. > > > > > > Looks like I got this wrong, adding a file-backed swap space I was > > > actually > > > able to run top, and seeing only 100M of memory being "Free", ~20G memory > > > reported as "Active", and swap usage constantly growing being consumed by > > > rustdoc process; something is really wrong here. > > > > Run top(1), hit the o key, type in size, and hit enter to have top > > sort the process list according to their virtual size. The culprit > > will eventually work its way to the top. > > Yes, it's rustdoc, and I'm seeing the same behavior as you described in your > other reply. What's more interesting, having a little swap, processes get > killed almost immediately, and the build happily continues resulting > successful rust package. In my case, I see this behaviour twice during the build. I'll try and reduce my swap partition to 2 GiB and see if that makes a difference during the build. -- Trond. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Wired memory consumes almost all my memory recently
On 03/03, Mateusz Piotrowski wrote: > Hi current@, > > I'm experiencing some problems with memory on r358410. > > I'm building a lot of ports in poudriere for testing purposes, which is very > a resource-intensive operation. After a couple of hours of such a workload, > the machine starts to become much less responsive (e.g., it takes a couple > seconds to start a new tmux pane). When I run top I would always see that > almost all my memory is wired (15G), swap reports a couple of hundreds > megabytes being used, and the free memory is less than 100M. > > If I kill all my desktop processes like Firefox and Thunderbird (it takes a > couple of minutes to complete this task!) then the machine becomes > responsive again, still having maybe a couple hundreds of megabytes of free > memory for me. > > I thought that maybe this is an ARC problem (I'm using ZFS) but it didn't > help to restrict ARC to 8G by setting sysctl vfs.zfs.arc_max=8517738496. > > I think that I've never experienced similar problems before. Please try r358505, it should be fixed. signature.asc Description: PGP signature
Wired memory consumes almost all my memory recently
Hi current@, I'm experiencing some problems with memory on r358410. I'm building a lot of ports in poudriere for testing purposes, which is very a resource-intensive operation. After a couple of hours of such a workload, the machine starts to become much less responsive (e.g., it takes a couple seconds to start a new tmux pane). When I run top I would always see that almost all my memory is wired (15G), swap reports a couple of hundreds megabytes being used, and the free memory is less than 100M. If I kill all my desktop processes like Firefox and Thunderbird (it takes a couple of minutes to complete this task!) then the machine becomes responsive again, still having maybe a couple hundreds of megabytes of free memory for me. I thought that maybe this is an ARC problem (I'm using ZFS) but it didn't help to restrict ARC to 8G by setting sysctl vfs.zfs.arc_max=8517738496. I think that I've never experienced similar problems before. Cheers, Mateusz Piotrowski ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: can't build rust -- out of swap space
On 03.03.2020 12:44, Trond Endrestøl wrote: On Tue, 3 Mar 2020 12:40+0300, Yuri Pankov wrote: On 03.03.2020 11:49, Yuri Pankov wrote: With recent pkg fallout, I'm trying to build rust myself first time ever (as far as I can remember), and it's failing running out of swap on the following step: Building stage0 std artifacts (x86_64-unknown-freebsd -> x86_64-unknown-freebsd) running: "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" "--message-format" "json-render-diagnostics" ^C^C^C Build completed unsuccessfully in 0:00:55 Here I pressed ^C as the build actually continues despite several rustdoc, python, and other processes being killed. swap_pager: out of swap space swp_pager_getswapspace(20): failed swap_pager: out of swap space swp_pager_getswapspace(11): failed The system has 32G of RAM and 2GB swap partition (as advised by zfs-auto installation option), top shows about 28G of memory free at that moment, so I'm wondering why the swap is being used, and if 2G should be enough to build rust. Looks like I got this wrong, adding a file-backed swap space I was actually able to run top, and seeing only 100M of memory being "Free", ~20G memory reported as "Active", and swap usage constantly growing being consumed by rustdoc process; something is really wrong here. Run top(1), hit the o key, type in size, and hit enter to have top sort the process list according to their virtual size. The culprit will eventually work its way to the top. Yes, it's rustdoc, and I'm seeing the same behavior as you described in your other reply. What's more interesting, having a little swap, processes get killed almost immediately, and the build happily continues resulting successful rust package. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: can't build rust -- out of swap space
On Tue, 3 Mar 2020 12:40+0300, Yuri Pankov wrote: > On 03.03.2020 11:49, Yuri Pankov wrote: > > With recent pkg fallout, I'm trying to build rust myself first time ever (as > > far as I can remember), and it's failing running out of swap on the > > following step: > > > > Building stage0 std artifacts (x86_64-unknown-freebsd -> > > x86_64-unknown-freebsd) > > running: > > "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" > > "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" > > "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" "--features" > > "panic-unwind backtrace compiler-builtins-c" "--manifest-path" > > "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" > > "--message-format" "json-render-diagnostics" > > ^C^C^C > > Build completed unsuccessfully in 0:00:55 > > > > Here I pressed ^C as the build actually continues despite several rustdoc, > > python, and other processes being killed. > > > > swap_pager: out of swap space > > swp_pager_getswapspace(20): failed > > swap_pager: out of swap space > > swp_pager_getswapspace(11): failed > > > > The system has 32G of RAM and 2GB swap partition (as advised by zfs-auto > > installation option), top shows about 28G of memory free at that moment, so > > I'm wondering why the swap is being used, and if 2G should be enough to > > build rust. > > Looks like I got this wrong, adding a file-backed swap space I was actually > able to run top, and seeing only 100M of memory being "Free", ~20G memory > reported as "Active", and swap usage constantly growing being consumed by > rustdoc process; something is really wrong here. Run top(1), hit the o key, type in size, and hit enter to have top sort the process list according to their virtual size. The culprit will eventually work its way to the top. -- Trond. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: can't build rust -- out of swap space
On Tue, 3 Mar 2020 11:49+0300, Yuri Pankov wrote: > With recent pkg fallout, I'm trying to build rust myself first time ever (as > far as I can remember), and it's failing running out of swap on the following > step: > > Building stage0 std artifacts (x86_64-unknown-freebsd -> > x86_64-unknown-freebsd) > running: > "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" > "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" > "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" "--features" > "panic-unwind backtrace compiler-builtins-c" "--manifest-path" > "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" > "--message-format" "json-render-diagnostics" > ^C^C^C > Build completed unsuccessfully in 0:00:55 > > Here I pressed ^C as the build actually continues despite several rustdoc, > python, and other processes being killed. > > swap_pager: out of swap space > swp_pager_getswapspace(20): failed > swap_pager: out of swap space > swp_pager_getswapspace(11): failed > > The system has 32G of RAM and 2GB swap partition (as advised by zfs-auto > installation option), top shows about 28G of memory free at that moment, so > I'm wondering why the swap is being used, and if 2G should be enough to build > rust. > > Tried building both as root and user, DISABLE_MAKE_JOBS has no effect. > > Any hints? It's good to see that it's not only me. I suspect choice of hardware might play a role. My laptop's a Lenovo E590 with 32 GiB of memory and 16 GiB of swap space, running fairly recent -CURRENT, booted via UEFI. Activating the laptop's kernel dump partition as a secondary swap device, adding another 16 GiB of swap space, had no positive effect. In my case, I've noticed rustdoc being run as "rustdoc --crate-type proc-macro --help", and this process grows to a virtual size of 10T, grabbing all the memory and swap that it can find. My laptop even locked up on one such occasion. On Sunday, I switched to lang/rust-nightly and I managed to also build bat, exa and hexyl. I was not that lucky with firefox and ripgrep, they both caused rustc to crash in the same fashion. I've tried building lang/rust on a couple of VMs running the latest -CURRENT, only to have these builds succeed. I even added the contents of /var/db/ports from my laptop, and the build still succeeded. Building lang/rust on stable/12 is successful atm. Maybe it's a combination of hardware and changes to the FreeBSD VM subsystem. -- Trond. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: can't build rust -- out of swap space
On 03.03.2020 11:49, Yuri Pankov wrote: With recent pkg fallout, I'm trying to build rust myself first time ever (as far as I can remember), and it's failing running out of swap on the following step: Building stage0 std artifacts (x86_64-unknown-freebsd -> x86_64-unknown-freebsd) running: "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" "--message-format" "json-render-diagnostics" ^C^C^C Build completed unsuccessfully in 0:00:55 Here I pressed ^C as the build actually continues despite several rustdoc, python, and other processes being killed. swap_pager: out of swap space swp_pager_getswapspace(20): failed swap_pager: out of swap space swp_pager_getswapspace(11): failed The system has 32G of RAM and 2GB swap partition (as advised by zfs-auto installation option), top shows about 28G of memory free at that moment, so I'm wondering why the swap is being used, and if 2G should be enough to build rust. Looks like I got this wrong, adding a file-backed swap space I was actually able to run top, and seeing only 100M of memory being "Free", ~20G memory reported as "Active", and swap usage constantly growing being consumed by rustdoc process; something is really wrong here. Tried building both as root and user, DISABLE_MAKE_JOBS has no effect. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
can't build rust -- out of swap space
With recent pkg fallout, I'm trying to build rust myself first time ever (as far as I can remember), and it's failing running out of swap on the following step: Building stage0 std artifacts (x86_64-unknown-freebsd -> x86_64-unknown-freebsd) running: "/usr/ports/lang/rust/work/rustc-1.41.1-src/build/x86_64-unknown-freebsd/stage0/bin/cargo" "build" "-Zconfig-profile" "--target" "x86_64-unknown-freebsd" "-Zbinary-dep-depinfo" "-j" "1" "-v" "--release" "--frozen" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/usr/ports/lang/rust/work/rustc-1.41.1-src/src/libtest/Cargo.toml" "--message-format" "json-render-diagnostics" ^C^C^C Build completed unsuccessfully in 0:00:55 Here I pressed ^C as the build actually continues despite several rustdoc, python, and other processes being killed. swap_pager: out of swap space swp_pager_getswapspace(20): failed swap_pager: out of swap space swp_pager_getswapspace(11): failed The system has 32G of RAM and 2GB swap partition (as advised by zfs-auto installation option), top shows about 28G of memory free at that moment, so I'm wondering why the swap is being used, and if 2G should be enough to build rust. Tried building both as root and user, DISABLE_MAKE_JOBS has no effect. Any hints? ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
