Re: reposync out of memory

2020-05-09 Thread Lucas 
Stuart Henderson  wrote:
> I can add something to the readme after the ports tree has unlocked.
> 
> I think you're seeing this now due to the churn because every file in
> the repositories was touched when the tree was tagged with OPENBSD_6_7.
> I haven't seen it on the 2 machines I have running reposync, but they
> update more frequently so probably didn't have to deal with updating
> all of ports+src+xenocara together, which is probably what tipped it
> over the edge.

For the archive, creating a new login class bumping datasize-cur and
adding cvs to that new login class did the trick. I doubt that's the
magic number tho, as I went over half of ports repo before that change,
so ymmv.

#
# reposync might choke during releases with default datasize
#
reposync:\
:datasize-cur=1024M:\
:tc=default:

Re: reposync out of memory

2020-05-09 Thread Stuart Henderson 
On 2020-05-09, Lucas  wrote:
> Hello misc@,
>
> Starting today, reposync is running out of memory for me. Happened 3
> times in a row already, in different stages. It looks like this when it
> happens:
>
>>f.st.. ports/net/megatools/pkg/PLIST,v
> ERROR: out of memory in flist_expand [receiver]
> rsync error: error allocating core memory buffers (code 22) at util2.c(105) 
> [receiver=3.1.3]
> rsync: [generator] write error: Broken pipe (32)
> rsync error: error in socket IO (code 10) at io.c(820) [generator=3.1.3]
> rsync: [receiver] write error: Broken pipe (32)
> reposync: rsync failed
>
> I'm issuing the following command:
>
> oolong$ doas -u cvs /usr/local/bin/reposync 
> rsync://obsdacvs.cs.toronto.edu/obsdcvs/ /home/cvs
>
> I set up cvs user as described in the pkg-readme:
>
> oolong$ getent passwd cvs; id -c cvs
> cvs:*:1001:1001::/nonexistent:/sbin/nologin
> default
>
> default class is unaltered in the system:
>
> default:\
>   :path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin /usr/local/bin 
> /usr/local/sbin:\
>   :umask=022:\
>   :datasize-max=768M:\
>   :datasize-cur=768M:\
>   :maxproc-max=256:\
>   :maxproc-cur=128:\
>   :openfiles-max=1024:\
>   :openfiles-cur=512:\
>   :stacksize-cur=4M:\
>   :localcipher=blowfish,a:\
>   :tc=auth-defaults:\
>   :tc=auth-ftp-defaults:
>
> FTR, I'm using reposync since it came out and this is the first time it
> happens. I update once a week, if that matters.
>
> Is anyone experiencing this too? I guess I can work it around bumping
> datasize for cvs user, but if this is happening to other ones, maybe
> it's worht to add something to the pkg-readme.

I can add something to the readme after the ports tree has unlocked.

I think you're seeing this now due to the churn because every file in
the repositories was touched when the tree was tagged with OPENBSD_6_7.
I haven't seen it on the 2 machines I have running reposync, but they
update more frequently so probably didn't have to deal with updating
all of ports+src+xenocara together, which is probably what tipped it
over the edge.

> Just in case, dmesg can be found after the email. Thanks in advance.

Thank you for the nice clear report.

reposync out of memory

2020-05-09 Thread Lucas 
Hello misc@,

Starting today, reposync is running out of memory for me. Happened 3
times in a row already, in different stages. It looks like this when it
happens:

>f.st.. ports/net/megatools/pkg/PLIST,v
ERROR: out of memory in flist_expand [receiver]
rsync error: error allocating core memory buffers (code 22) at util2.c(105) 
[receiver=3.1.3]
rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820) [generator=3.1.3]
rsync: [receiver] write error: Broken pipe (32)
reposync: rsync failed

I'm issuing the following command:

oolong$ doas -u cvs /usr/local/bin/reposync 
rsync://obsdacvs.cs.toronto.edu/obsdcvs/ /home/cvs

I set up cvs user as described in the pkg-readme:

oolong$ getent passwd cvs; id -c cvs
cvs:*:1001:1001::/nonexistent:/sbin/nologin
default

default class is unaltered in the system:

default:\
:path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin /usr/local/bin 
/usr/local/sbin:\
:umask=022:\
:datasize-max=768M:\
:datasize-cur=768M:\
:maxproc-max=256:\
:maxproc-cur=128:\
:openfiles-max=1024:\
:openfiles-cur=512:\
:stacksize-cur=4M:\
:localcipher=blowfish,a:\
:tc=auth-defaults:\
:tc=auth-ftp-defaults:

FTR, I'm using reposync since it came out and this is the first time it
happens. I update once a week, if that matters.

Is anyone experiencing this too? I guess I can work it around bumping
datasize for cvs user, but if this is happening to other ones, maybe
it's worht to add something to the pkg-readme.

Just in case, dmesg can be found after the email. Thanks in advance.

-Lucas


OpenBSD 6.7 (GENERIC.MP) #182: Thu May  7 11:11:58 MDT 2020
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 3944083456 (3761MB)
avail mem = 3811930112 (3635MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xdae9d000 (68 entries)
bios0: vendor LENOVO version "G2ET33WW (1.13 )" date 07/24/2012
bios0: LENOVO 2325BG4
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SLIC TCPA SSDT SSDT SSDT HPET APIC MCFG ECDT FPDT ASF! 
UEFI UEFI POAT SSDT SSDT DMAR UEFI
acpi0: wakeup devices LID_(S4) SLPB(S3) IGBE(S4) EXP3(S4) XHCI(S3) EHC1(S3) 
EHC2(S3) HDEF(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz, 1197.52 MHz, 06-3a-09
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz, 1197.29 MHz, 06-3a-09
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xf800, bus 0-63
acpiec0 at acpi0
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PEG_)
acpiprt2 at acpi0: bus 2 (EXP1)
acpiprt3 at acpi0: bus 3 (EXP2)
acpiprt4 at acpi0: bus 4 (EXP3)
acpicpu0 at acpi0: C3(200@87 mwait.1@0x30), C2(500@59 mwait.1@0x10), C1(1000@1 
mwait.1), PSS
acpicpu1 at acpi0: C3(200@87 mwait.1@0x30), C2(500@59 mwait.1@0x10), C1(1000@1 
mwait.1), PSS
acpipwrres0 at acpi0: PUBS, resource for XHCI, EHC1, EHC2
acpitz0 at acpi0: critical temperature is 103 degC
acpibtn0 at acpi0: LID_
acpibtn1 at acpi0: SLPB
acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001
acpicmos0 at acpi0
tpm0 at acpi0: TPM_ addr 0xfed4/0x5000, device 0x104a rev 0x4e
acpibat0 at acpi0: BAT0 model "45N1029" serial 15304 type LION oem "LGC"
acpiac0 at acpi0: AC unit offline
acpithinkpad0 at acpi0: version 1.0
"PNP0C14" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpidock0 at acpi0: GDCK not docked (0)
acpivideo0 at acpi0: VID_
acpivout0 at acpivideo0: LCD0
acpivideo1 at acpi0: VID_
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 1197 MHz: speeds: 2601, 2600, 2500, 2400, 2300,