Re: Problem reports for version control systems

[matthew sporleder]

On Fri, Apr 30, 2021 at 5:50 AM Joerg Sonnenberger  wrote:
>
> On Fri, Apr 30, 2021 at 05:31:53PM +1200, Lloyd Parkes wrote:
> > ceph4% hg --version
> > Mercurial Distributed SCM (version 5.3.2)
>
> Please note that this is quite an old version and a lot of work on
> improving both CPU time and memory use has been spend since then.
>
> Joerg

I confirmed this was the case - btw.  mercurial 5.7 actually works for
a clone and should stay under 1G of memory but will eat about 8GB of
space (git is 2GB)

These older versions all failed for me 9/10 times.

Re: regarding the changes to kernel entropy gathering

[matthew sporleder]

On Wed, Apr 7, 2021 at 7:10 AM Martin Husemann  wrote:
>
> On Wed, Apr 07, 2021 at 07:05:12AM -0400, matthew sporleder wrote:
> > Is the issue gaw saw exclusive to xen first boots?  Are there other
> > ways to end up in his situation?
>
> It happens on all new installations for machines with no RNG, which is
> the far majority of everything but "newish" amd64 and a few arm and mips
> boards/SoC.
>
> It is unrelated to Xen.
>
> Martin

So on a brand new installation/first boot why isn't the clock a
sufficiently random thing? (anymore?)

Hung and unusable systems are a big problem.  Happening on the first
boot is not a good first impression. :)

Re: regarding the changes to kernel entropy gathering

[matthew sporleder]

> On Apr 6, 2021, at 8:09 AM, Taylor R Campbell  wrote:
> 
> 
>> Date: Mon, 05 Apr 2021 10:58:58 +0700
>> From: Robert Elz 
>> I understand that some people desire highly secure systems (I'm not
>> convinced that anyone running NetBSD can really justify that desire,
>> but that's beside the point) and that's fine - make the system be able
>> to be as secure as possible, just don't require me to enable it, and
>> don't make it impossible or even difficuly to disable it - and allow
>> some kind of middle ground, just just "perfectly secure" and "hopeless".
> 
> The main issue that hits people is that the traditional mechanism by
> which the OS reports a potential security problem with entropy is for
> it to make applications silently hang -- and the issue is getting
> worse now that getrandom() is more widely used, e.g. in Python when
> you do `import multiprocessing'.
> 
> Based on experience over the past year with a meaningful criterion for
> _detecting_ potential problems, I don't think that's a useful
> mechanism for _reporting_ them, which is why I added several other
> mechanisms -- a line in the /etc/security report, an `entropy' knob in
> /etc/rc.conf to wait or fail to single-user (default: neither) -- and
> proposed to remove the blocking behaviour of getrandom() in favour of
> focusing on feedback in system integration:
> 
> https://mail-index.netbsd.org/tech-userlevel/2021/01/11/msg012807.html
> 
> (main discussion after all the noise starts here:
> https://mail-index.netbsd.org/tech-userlevel/2021/01/15/msg012846.html)
> 
> But I ran out of steam to continue the discussion at the time.


Is the issue gaw saw exclusive to xen first boots?  Are there other ways to end 
up in his situation?

Re: sponsor NetBSD for 2020 https://github.com/sponsors/NetBSD

[matthew sporleder]

Indeed -- casting a wide net is in our interest.  I hope you are able
to use one of our many potential donation offerings -- paypal, stripe,
amazon smile, github sponsorship.. any I am missing?

On Tue, Nov 10, 2020 at 7:55 AM Sad Clouds  wrote:
>
> On Tue, 10 Nov 2020 12:31:26 +0100
> Matthias Petermann  wrote:
>
> > Hallo Matthew,
> >
> > Am 10.11.2020 um 05:35 schrieb matthew sporleder:
> > > Hey -- the end of the year is coming up fast.  Wouldn't you feel
> > > better about yourself if you added a github sponsorship to balance
> > > out your incredible year? :)
> > How does this type of donation compare to a Paypal Monthly
> > Subscription? Is it just a different way of transport, or are there
> > advantages / disadvantages to Paypal?
> >
> > Kind regards
> > Matthias
>
> It looks to me like github sponsorship is geared towards small
> developers who don't have their own project web page, with
> payment submission links, etc. The irritating thing about github is
> that they don't allow you to submit bug reports or donations unless you
> setup an account on their platform. Last thing I want to do is to create
> various social media accounts on competing platform - GitHub, GitLab,
> SourceForge, Bitbucket, GNU Savannah, ... and the list goes on. Life is
> too short for that nonsense. Luckily NetBSD accept bug reports and
> donations on their official web site, and you don't have to sign up for
> anything.

Re: sponsor NetBSD for 2020 https://github.com/sponsors/NetBSD

[matthew sporleder]

On Tue, Nov 10, 2020 at 6:31 AM Matthias Petermann  wrote:
>
> Hallo Matthew,
>
> Am 10.11.2020 um 05:35 schrieb matthew sporleder:
> > Hey -- the end of the year is coming up fast.  Wouldn't you feel
> > better about yourself if you added a github sponsorship to balance out
> > your incredible year? :)
> How does this type of donation compare to a Paypal Monthly Subscription?
> Is it just a different way of transport, or are there advantages /
> disadvantages to Paypal?
>
> Kind regards
> Matthias

It is my understanding, although it hasn't been confirmed by
finance-exec@, that microsoft is absorbing the fees for us.

I am also intrigued by the potential network effects of sponsors
encouraging more sponsors, etc.

sponsor NetBSD for 2020 https://github.com/sponsors/NetBSD

[matthew sporleder]

Hey -- the end of the year is coming up fast.  Wouldn't you feel
better about yourself if you added a github sponsorship to balance out
your incredible year? :)

Do you live in one of these places?

Australia
Austria
Belgium
Canada
Cyprus
Czech Republic
Denmark
Estonia
Finland
France
Germany
Greece
Hong Kong SAR
Ireland
Italy
Japan
Latvia
Lithuania
Luxembourg
Malta
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Singapore
Slovakia
Slovenia
Spain
Sweden
Switzerland
United Kingdom
United States of America

then sign up: https://github.com/sponsors/NetBSD

A little while ago we added some very low tiers so it's not a lot of
money at all!

Re: github.com/NetBSD/src 5 days old?

[matthew sporleder]

On Fri, May 22, 2020 at 5:57 PM Greg A. Woods  wrote:
>
> At Thu, 21 May 2020 15:11:41 -0400, Andrew Cagney  
> wrote:
> Subject: Re: github.com/NetBSD/src 5 days old?
> >
> > The details are all found here:
> > https://mail-index.netbsd.org/tech-repository/2020/02/17/msg000685.html
>
> That just says what might happen (and what could/should happen at the
> same time), not why (nor how the decision was arrived at).
>
> > > I've never found anything there explaining the actual rationale for Hg.
>
> --

Joerg is the one doing all of the work and he wants to land on hg.

Every other "justification" or benchmark or whatever is pretty much a
lie.  Especially now, five years later, when git has gotten better and
better at big repos.

Matt

p.s. this whole thing reached a head (Core statement on version
control systems) in Jan 2015
https://mail-index.netbsd.org/tech-repository/2015/01/04/msg000497.html

Re: github.com/NetBSD/src 5 days old?

[matthew sporleder]

On Sun, May 17, 2020 at 8:08 PM Constantine A. Murenin  wrote:
>
> On Thu, 14 May 2020 at 09:23, Hauke Fath  
> wrote:
>>
>> [re-directing to tech-repository, which was created precisely to keep
>> debates like this one off the other lists...]
>>
>> On Thu, 14 May 2020 14:47:02 +0200, Jens Rehsack wrote:
>> > I doubt that you'll find a modern solution running fine on any 4M computer.
>> > Network filesystems, cross compilers etc. where invented to support 
>> > machines
>> > which can't provide all required resources for a job on their own.
>>
>> Unfortunately, the VCS equivalent to your list would be a client
>> connecting to a beefy local DVCS instance, which to the best of my
>> knowledge has not been invented, yet.
>
>
> Actually, it has already been invented.  GitHub has links to download the 
> checkout as a zip archive from any branch.
>
> E.g., https://github.com/NetBSD/src/archive/netbsd-9.zip has the checkout 
> from `netbsd-9`.
>
> I've just tried how it works, and am getting 5MB/s on my 12.6MB/s connection 
> through the WiFi in the office, so, it seems to be working good enough.  I 
> believe they archive it on the go, as a stream, because there's no file size 
> upfront when you first download it; I've tried downloading it a second time 
> right after completing the first one, and I did get the size then (Length: 
> 548765520 (523M) [application/zip]), so, they are smart enough to cache it at 
> least for some time.
>
> Of course, the biggest issue is that there's no way to ignore any specific 
> parts of the tree, so, you're stuck with downloading a 0.5GB archive of a 
> 2.4GB checkout.  I'm still of the opinion that it might be a good idea to 
> split the `src` repository into several sub-repositories like syssrc, gnusrc 
> and src, as per 
> http://mail-index.netbsd.org/tech-repository/2020/02/21/msg000698.html.  Or 
> maybe at least provide such a setup as an option, especially to just get the 
> kernel?
>
> Cheers,
> Constantine.

This is a built-in git feature:
https://git-scm.com/book/en/v2/Git-Tools-Bundling  (hg archive is the
same, I think)

If you want small and fast you can use shallow clone and, although you
get the entire tree's bundle, it is small and fast.
You can then use --sparse to build a "sparse" (kernel only or
whatever) limited checkout (aka working dir) -- (new git feature--
https://git-scm.com/docs/git-sparse-checkout  /
https://git-scm.com/docs/git-read-tree#_sparse_checkout  ) / I don't
know about mercurial's version of this

Re: github.com/NetBSD/src 5 days old?

[matthew sporleder]

> On May 13, 2020, at 10:11 PM, John Franklin  wrote:
> 
> On Apr 30, 2020, at 21:28, bch  wrote:
>> 
>> I thought the plan to move to HG hasn't been finalised yet, am I missing 
>> something?  Plus, why HG and not Fossil, if the end-result consumption is 
>> via Git anyways?
>> 
>> Last I heard fossil had scaling issues due to the large number of artifacts 
>> that needed to be tracked. I may be able to trawl notes and find some 
>> particulars, or Joerg may be able to comment from memory on the technical 
>> aspects.
>> 
>> 
>> I was really hopeful for fossil as a solution as it seems really sane for 
>> many reasons:
>> 1) good user interface(s)
>> 2) good, novel ticket handling
>> 3) sane architecture
>> 4) portable C implementation
>> 5) BSD license 
>> 
>> I think in the end though Joerg reckoned the scalability issue was too much.
> 
> There are scalability issues with Mercurial, too.  I cloned NetBSD src on a 
> 1GB RAM, 1GB swap, 4 CPU VM (Debian Buster) using git from the GitHub project 
> and from anonhg.netbsd.org.
> 
> Git consumed 675MB of memory at its peak, and took 4m38s.  
> 
> Cloning with hg from anonhg.netbsd.org consumes all RAM and all swap before 
> the OOM killer takes it out.  
> 
> Upping the memory to 2GB RAM (still 1GB swap) gets further along, to the 
> point where hg forks into $(CPU_COUNT) processes for “updating to bookmark @ 
> on branch trunk” before the OOM killer takes it out.  
> 
> Finally, 2GB RAM and 1GB swap, and enabling vm.overcommit_memory was enough 
> to let hg finish in 17m52s.
> 
> jf
> -- 
> John Franklin
> frank...@elfie.org

This argument does not work. I went through the same goalpost moving exercise 
years ago and martin@ even got some efficiency patches into git as a result, 
but the super low memory shallow clone is not even good enough. 
> 

Re: tar extract changed since netbsd-8? (extracting sets over running system)

[matthew sporleder]

On Tue, Nov 12, 2019 at 8:51 AM Joerg Sonnenberger  wrote:
>
> On Tue, Nov 12, 2019 at 08:25:46AM -0500, Greg Troxel wrote:
> > David Brownlee  writes:
> >
> > > On Tue, 12 Nov 2019 at 11:33, Jaromír Doleček  
> > > wrote:
> > >>
> > >> Le mar. 12 nov. 2019 à 12:05, Martin Husemann  a 
> > >> écrit :
> > >>>
> > >>> Not seen this locally, but that would be the switch to bsd/libarchive 
> > >>> tar.
> > >>> Maybe it does not unlink files before extraction and replaces them 
> > >>> in-space?
> > >>>
> > >>> I do the same kind of updates, but usually on a very quiet system.
> > >>>
> > >>> The crashes you see are from other running processes? Joerg would likely
> > >>> say: "don't do that" ;-)
> > >>>
> > >>
> > >> I thought we do unlink by default, I think I remember a commit to that 
> > >> effect in past. The unlink is very useful default behaviour of GNU tar. 
> > >> In-place overwrite is very rarely the wanted behaviour.
> > >
> > > Aha thanks!
> > >
> > > I would argue the switch to unlink no longer being the default is a
> > > regression. If its mandated by standards or we're the only outlier
> > > then it probably makes sense to switch, but otherwise its sprinkling
> > > occasional non deterministic breakage across a bunch of scripts which
> > > previously ran fine on NetBSD.
>
> It does not force an unlink first, but will unlink if there is a
> conflict. So the question would be why open(2) with O_CREAT|O_EXCL
> doesn't fail.
>
> > I'm not quite clear on 'unlink by default', but it seems to me the
> > replacement files should be written to a temporary name and then
> > renamed() into place so the file is either the old version or the new
> > version.
>
> That would dramatically increase the time it takes for an untar for
> little to no gain.
>
> Joerg

If you try the same thing on FreeBSD does the same error happen?

Re: Autoexpand of root file system after image copy - need something better

[matthew sporleder]

On Fri, Dec 22, 2017 at 4:14 PM, Thomas Goldthorpe  wrote:
> Hi Folks,
>
> Chuck Silvers suggested to send to port-arm and current-users for this:
>
> I may or may not stand alone on this, but, the autoexpand of the root file
> system to full flash size that is in the images (i'm thinking the PI, but
> others too), is detrimental and a pain to work around.  I find it very rare
> I ever want the root filesystem to be the full size of the flash.  Larger,
> yes, but, not that much.
>
> Is there any reason that this can't instead be done such that it asks how
> much to expand it to, only the first boot, rather than just expanding it?
> If doing something attached to the console during boot is too much of an
> issue, then perhaps a script that the user can run if they so desire, along
> with a boot time message saying they can do so if they want.  An option to
> the user could then be max-size for those who want to do so, otherwise just
> enter the size you want into the script input.  The workaround is time
> consuming:  take flash to machine with fat support, change boot to -s, back
> to evb and boot, edit out resize, mount fat and remove -s, reboot...  There
> must be a better way.
>
> Just asking.
>
> Thank you
>
> Tom

Seems easy to add to boot.cfg or if you interrupt the boot and take a
param, but I think expanding is smart since (most?) first boots are
non-interactive?

Fwd: nyftp.netbsd.org outage 2017-07-24 through 2017-07-25

[matthew sporleder]

In case anyone was wondering why the various ny* servers are down right now..


-- Forwarded message --
From: Thor Lancelot Simon 
Date: Mon, Jul 24, 2017 at 1:38 AM
Subject: nyftp.netbsd.org outage 2017-07-24 through 2017-07-25


Due to urgent construction, nyftp.netbsd.org and other NetBSD resources
hosted at Columbia University will be offline beginning mid-day (New York
time) today, the 24th of July, and will come back online some time late in
the day tomorrow, the 25th.

Re: Identifying the NetBSD shell

[matthew sporleder]

On Mon, Mar 21, 2016 at 2:14 PM, Greg Troxel  wrote:
>
> Robert Elz  writes:
>
>> What I am thinking for this, is that we create 3 segment version numbers,
>> N.M.P where "N.M" is the netbsd release the shell started at (so the
>> NetBSD 7.0 shell would have had N=7 and M=0 had this scheme existed when
>> it was released.  P is to be a patchlevel counter that gets incremented
>> whenever there are changes made that are significant enough to warrant
>> calling it a new version - and certainly every time a new vresion is
>> released to pkgsrc).   Just like NetBSD version numbers, M==99 implies
>> "current" on he way to version N+1.   Note that P in the shell version
>> numbering would have no relationship at all with the third value in
>> NetBSD version numbers, so NetBSD 7.0.1 7.0.2 ... would all just have a
>> "7.0.P" shell version number, with whatever value is appropriate for P
>> (possibly all just 0).
>>
>> For now, I am using 7.99.1 as my version number - as I am doing all of
>> this to the NetBSD current (7.99) shell, and 7.99.0 would be the shell
>> that is there immediately before this version number shceme is implemented
>> (and all previous versions... there have been many recently.)
>
> I find using 7.99.X awkward, as that's a version that means something
> for the kernel (and userland more or less), and this is really something
> quite different.
>
> I'd be tempted to call in 1.0.0 arbitrarily, and bump micro on bugfixes
> (or just prior to export of a new portable version), minor on feature
> additions, and major on significant compat breaks (perhaps never).
> That will keep people from inferring things that aren't true.

I think the responses here have shown that you should just use your
best judgement and stick with it.  :)

Re: rsync to anoncvs.NetBSD.org::cvsroot missing dirs?

[matthew sporleder]

On Tue, Dec 1, 2015 at 11:03 PM, Paul Ripke  wrote:
> I've been rsyncing against anoncvs (and/or mirrors) for ~years, without
> issues. I've just noticed build issues, and tracked it down to the
> following dirs missing from my rsync of the anoncvs repo:
>
> src/external/gpl3/gcc.old
> src/external/gpl3/gdb.old
>
> Running a diff between a cvs checkout of the local rsync repo and a
> cvs checkout direct from anoncvs seems to show quite a lot of stuff
> missing from the rsync'ed repo (1854 dirs/files).
> Permissions issue, perhaps?
>
> FYI: I'm currently rsyncing against anoncvs.NetBSD.org::cvsroot
> I assume this is still a somewhat supported method of keeping up
> to date sources?
>
> Cheers,
> --
> Paul Ripke
> "Great minds discuss ideas, average minds discuss events, small minds
>  discuss people."
> -- Disputed: Often attributed to Eleanor Roosevelt. 1948.


I'm not sure if it's happened yet but a lot of our systems (anoncvs
included) are migrating.

I'm confident things will clear up over the next little bit.

Re: READ ME: updating mpc, mpfr, and eventually gmp

[matthew sporleder]

On Sun, Dec 15, 2013 at 8:05 PM, Thomas Mueller
mueller6...@bellsouth.net wrote:

 Matt Sporleder asks:

  What is your build.sh command?

  Did the build ever work?

 Latest build.sh command was

 === build.sh command:./build.sh -m amd64 -M ../obj.amd64.llvm -B 
 nb20131214-llvm -T ../tooldir.amd64.llvm -V MKLLVM=yes -V HAVE_LLVM=yes -V 
 MKLIBCXX=yes -U -j 9 distribution kernel=GENERIC

 I don't use llvm/clang all the time.

 I was able to build 6.99.23 and before that.

 There was something in the original post on this thread about cleaning both 
 the tools and normal build directories.

 So maybe I need to clean out the tools directory 
 (/BETA1/netbsd-HEAD/usr/src/tools) except for the CVS subdirectory?

 Or maybe clear the entire /BETA1/netbsd-HEAD/usr/src directory and build 
 directories in /BETA1/netbsd-HEAD/usr but preserving log files and CVS 
 subdirectory?


 Tom


Can you take a step back and try using default tools (gcc) and a clean
dir (build.sh -r should do it, or just completely delete your obj and
tool dir first)?

Re: PF netbsd 6.1.2

[matthew sporleder]

On Thu, Oct 10, 2013 at 7:03 AM, Guilherme Covolo smy...@youare.not.br wrote:
 How to disable pf.boot or open ssh port on netbsd 6.1?

 thanks


NetBSD 6.1 has two options for firewalls: ipf and pf

there are plenty of tutorials on each, but here's one with ipf:
https://wiki.netbsd.org/nsps/

also check man rc.conf on how to enable the firewalls
 ipfilter`YES' or `NO'.  Runs ipf(8) to load in packet filter
 specifications from /etc/ipf.conf at network boot time,
 before any interfaces are configured.  Passes
 ipfilter_flags.  See ipf.conf(5).

 pf  Boolean value.  Enable pf(4) at network boot time: Load
 the initial configuration pf.boot.conf(5) before the net-
 work is up.  After the network has been configured, then
 load the final ruleset pf.conf(5).

Hope this helps,
Matt