Re: openrsync out of memory
Olivier Antoine(olivier.anto...@gmail.com) on 2019.08.19 11:34:06 +0200: > Hi, > On i386: before patch: > $ dd if=/dev/urandom of=in bs=1M count=2k > $ openrsync --rsync-path=/usr/bin/openrsync -av in localhost:out > Transfer starting: 1 files > sender.c:551: error: in: mmap: Cannot allocate memory > client.c:85: error: rsync_sender > receiver.c:345: error: poll: hangup > server.c:145: error: rsync_receiver > > With your patch: > $ patch -p0 < /tmp/1 > Hmm... Looks like a unified diff to me... > The text leading up to this was: > -- > |diff --git usr.bin/rsync/uploader.c usr.bin/rsync/uploader.c > |index fd07b22caeb..cce8b47a4c9 100644 > |--- usr.bin/rsync/uploader.c > |+++ usr.bin/rsync/uploader.c > -- > Patching file usr.bin/rsync/uploader.c using Plan A... > Hunk #1 succeeded at 158. > Hunk #2 succeeded at 741. > Hunk #3 succeeded at 910. > Hmm... Ignoring the trailing garbage. > done > > $ cd usr.bin/rsync/ && make -j3 && doas make install > ??? > > $ openrsync --rsync-path=/usr/bin/openrsync -av in localhost:out > Transfer starting: 1 files > sender.c:551: error: in: mmap: Cannot allocate memory > client.c:85: error: rsync_sender > receiver.c:345: error: poll: hangup > server.c:145: error: rsync_receiver > > From what I see, the mmap problem is on sender.c Thanks, i'll work on that one too. /Benno
Re: openrsync out of memory
Hi,
On i386: before patch:
$ dd if=/dev/urandom of=in bs=1M count=2k
$ openrsync --rsync-path=/usr/bin/openrsync -av in localhost:out
Transfer starting: 1 files
sender.c:551: error: in: mmap: Cannot allocate memory
client.c:85: error: rsync_sender
receiver.c:345: error: poll: hangup
server.c:145: error: rsync_receiver
With your patch:
$ patch -p0 < /tmp/1
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--
|diff --git usr.bin/rsync/uploader.c usr.bin/rsync/uploader.c
|index fd07b22caeb..cce8b47a4c9 100644
|--- usr.bin/rsync/uploader.c
|+++ usr.bin/rsync/uploader.c
--
Patching file usr.bin/rsync/uploader.c using Plan A...
Hunk #1 succeeded at 158.
Hunk #2 succeeded at 741.
Hunk #3 succeeded at 910.
Hmm... Ignoring the trailing garbage.
done
$ cd usr.bin/rsync/ && make -j3 && doas make install
…
$ openrsync --rsync-path=/usr/bin/openrsync -av in localhost:out
Transfer starting: 1 files
sender.c:551: error: in: mmap: Cannot allocate memory
client.c:85: error: rsync_sender
receiver.c:345: error: poll: hangup
server.c:145: error: rsync_receiver
>From what I see, the mmap problem is on sender.c
Cheers,
On Sat, Aug 17, 2019 at 4:53 PM Sebastian Benoit wrote:
>
> Joe Davis(m...@jo.ie) on 2019.08.16 12:26:36 +0100:
> > By the looks of it, openrsync does attempt to map the entire file, from
> > usr.bin/rsync/uploader.c:
> >
> > mapsz = st.st_size;
> > map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
> >
> > The likely reason for your out of memory error is the default datasize
> > in login.conf. IIRC on some arches it's set to 768MB by default, which
> > would allow your 300MB file to transfer, but would cause mmap to fail
> > upon attempting to map the 1.6GB one.
> >
> > Increasing the default limits in /etc/login.conf should fix the problem.
> >
> > Note that rsync (not openrsync), doesn't use mmap for other reasons,
> > from rsync-3.1.3/fileio.c:
> >
> > /* This provides functionality somewhat similar to mmap() but using read().
> > * It gives sliding window access to a file. mmap() is not used because of
> > * the possibility of another program (such as a mailer) truncating the
> > * file thus giving us a SIGBUS. */
> >
> > Cheers,
> > Joe
>
> Hi,
>
> this replaces the mmap() with pread(), please try it out.
>
> I dont much like the error handling here, but its a start.
>
> ok?
>
>
> diff --git usr.bin/rsync/uploader.c usr.bin/rsync/uploader.c
> index fd07b22caeb..cce8b47a4c9 100644
> --- usr.bin/rsync/uploader.c
> +++ usr.bin/rsync/uploader.c
> @@ -158,8 +158,8 @@ init_blk(struct blk *p, const struct blkset *set, off_t
> offs,
> p->len = idx < set->blksz - 1 ? set->len : set->rem;
> p->offs = offs;
>
> - p->chksum_short = hash_fast(map + offs, p->len);
> - hash_slow(map + offs, p->len, p->chksum_long, sess);
> + p->chksum_short = hash_fast(map, p->len);
> + hash_slow(map, p->len, p->chksum_long, sess);
> }
>
> /*
> @@ -741,8 +741,9 @@ rsync_uploader(struct upload *u, int *fileinfd,
> {
> struct blkset blk;
> struct stat st;
> - void *map, *bufp;
> - size_t i, mapsz, pos, sz;
> + void *mbuf, *bufp;
> + ssize_t msz;
> + size_t i, pos, sz;
> off_t offs;
> int c;
> const struct flist *f;
> @@ -909,35 +910,46 @@ rsync_uploader(struct upload *u, int *fileinfd,
> blk.csum = u->csumlen;
>
> if (*fileinfd != -1 && st.st_size > 0) {
> - mapsz = st.st_size;
> - map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
> - if (map == MAP_FAILED) {
> - ERR("%s: mmap", u->fl[u->idx].path);
> - close(*fileinfd);
> - *fileinfd = -1;
> - return -1;
> - }
> -
> init_blkset(&blk, st.st_size);
> assert(blk.blksz);
>
> blk.blks = calloc(blk.blksz, sizeof(struct blk));
> if (blk.blks == NULL) {
> ERR("calloc");
> - munmap(map, mapsz);
> + close(*fileinfd);
> + *fileinfd = -1;
> + return -1;
> + }
> +
> + if ((mbuf = calloc(1, blk.len)) == NULL) {
> + ERR("calloc");
> close(*fileinfd);
> *fileinfd = -1;
> return -1;
> }
>
> offs = 0;
> - for (i = 0; i < blk.blksz; i++) {
> - init_blk(&blk.blks[i],
> - &blk, offs, i, map, sess);
> + i = 0;
> + do {
> + msz = pread(*fileinfd, mbuf, blk.len, offs);
> +
Re: openrsync out of memory
Joe Davis(m...@jo.ie) on 2019.08.16 12:26:36 +0100:
> By the looks of it, openrsync does attempt to map the entire file, from
> usr.bin/rsync/uploader.c:
>
> mapsz = st.st_size;
> map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
>
> The likely reason for your out of memory error is the default datasize
> in login.conf. IIRC on some arches it's set to 768MB by default, which
> would allow your 300MB file to transfer, but would cause mmap to fail
> upon attempting to map the 1.6GB one.
>
> Increasing the default limits in /etc/login.conf should fix the problem.
>
> Note that rsync (not openrsync), doesn't use mmap for other reasons,
> from rsync-3.1.3/fileio.c:
>
> /* This provides functionality somewhat similar to mmap() but using read().
> * It gives sliding window access to a file. mmap() is not used because of
> * the possibility of another program (such as a mailer) truncating the
> * file thus giving us a SIGBUS. */
>
> Cheers,
> Joe
Hi,
this replaces the mmap() with pread(), please try it out.
I dont much like the error handling here, but its a start.
ok?
diff --git usr.bin/rsync/uploader.c usr.bin/rsync/uploader.c
index fd07b22caeb..cce8b47a4c9 100644
--- usr.bin/rsync/uploader.c
+++ usr.bin/rsync/uploader.c
@@ -158,8 +158,8 @@ init_blk(struct blk *p, const struct blkset *set, off_t
offs,
p->len = idx < set->blksz - 1 ? set->len : set->rem;
p->offs = offs;
- p->chksum_short = hash_fast(map + offs, p->len);
- hash_slow(map + offs, p->len, p->chksum_long, sess);
+ p->chksum_short = hash_fast(map, p->len);
+ hash_slow(map, p->len, p->chksum_long, sess);
}
/*
@@ -741,8 +741,9 @@ rsync_uploader(struct upload *u, int *fileinfd,
{
struct blkset blk;
struct stat st;
- void *map, *bufp;
- size_t i, mapsz, pos, sz;
+ void *mbuf, *bufp;
+ ssize_t msz;
+ size_t i, pos, sz;
off_t offs;
int c;
const struct flist *f;
@@ -909,35 +910,46 @@ rsync_uploader(struct upload *u, int *fileinfd,
blk.csum = u->csumlen;
if (*fileinfd != -1 && st.st_size > 0) {
- mapsz = st.st_size;
- map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0);
- if (map == MAP_FAILED) {
- ERR("%s: mmap", u->fl[u->idx].path);
- close(*fileinfd);
- *fileinfd = -1;
- return -1;
- }
-
init_blkset(&blk, st.st_size);
assert(blk.blksz);
blk.blks = calloc(blk.blksz, sizeof(struct blk));
if (blk.blks == NULL) {
ERR("calloc");
- munmap(map, mapsz);
+ close(*fileinfd);
+ *fileinfd = -1;
+ return -1;
+ }
+
+ if ((mbuf = calloc(1, blk.len)) == NULL) {
+ ERR("calloc");
close(*fileinfd);
*fileinfd = -1;
return -1;
}
offs = 0;
- for (i = 0; i < blk.blksz; i++) {
- init_blk(&blk.blks[i],
- &blk, offs, i, map, sess);
+ i = 0;
+ do {
+ msz = pread(*fileinfd, mbuf, blk.len, offs);
+ if (msz < 0) {
+ ERR("pread");
+ close(*fileinfd);
+ *fileinfd = -1;
+ return -1;
+ }
+ if ((size_t)msz != blk.len && (size_t)msz != blk.rem) {
+ /* short read, try again */
+ continue;
+ }
+ init_blk(&blk.blks[i], &blk, offs, i, mbuf, sess);
offs += blk.len;
- }
+ LOG3(
+ "i=%ld, offs=%lld, msz=%ld, blk.len=%lu,
blk.rem=%lu",
+ i, offs, msz, blk.len, blk.rem);
+ i++;
+ } while (i < blk.blksz);
- munmap(map, mapsz);
close(*fileinfd);
*fileinfd = -1;
LOG3("%s: mapped %jd B with %zu blocks",
Re: openrsync out of memory
Christian Weisgerber wrote: > On 2019-08-16, Jan Stary wrote: > > >> Does that mean openrsync tries to mmap() the entire file? > >> The machine only has 256MB of memory, but it does transfer > >> a test file of 300MB, so that can't be it. > > > > I forgot about 1GB swap, so that's why it works > > for files up to around 1.2G, but not larger. > > Why would the size of physical memory + swap matter? > mmap() doesn't copy a file into memory, it maps it into the address > space. Yes, but when rsync calculates the hash, before exchanging the hash to decide if it should ship the file, it touches every page of the file, which pulls it into the process.
Re: openrsync out of memory
On 2019-08-16, Jan Stary wrote: >> Does that mean openrsync tries to mmap() the entire file? >> The machine only has 256MB of memory, but it does transfer >> a test file of 300MB, so that can't be it. > > I forgot about 1GB swap, so that's why it works > for files up to around 1.2G, but not larger. Why would the size of physical memory + swap matter? mmap() doesn't copy a file into memory, it maps it into the address space. -- Christian "naddy" Weisgerber na...@mips.inka.de
Re: openrsync out of memory
On Aug 16 10:43:41, h...@stare.cz wrote: > Does that mean openrsync tries to mmap() the entire file? > The machine only has 256MB of memory, but it does transfer > a test file of 300MB, so that can't be it. I forgot about 1GB swap, so that's why it works for files up to around 1.2G, but not larger. Jan
Re: openrsync out of memory
Joe Davis wrote: > By the looks of it, openrsync does attempt to map the entire file, from > usr.bin/rsync/uploader.c: > > mapsz = st.st_size; > map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0); > > The likely reason for your out of memory error is the default datasize > in login.conf. IIRC on some arches it's set to 768MB by default, which > would allow your 300MB file to transfer, but would cause mmap to fail > upon attempting to map the 1.6GB one. > > Increasing the default limits in /etc/login.conf should fix the problem. > > Note that rsync (not openrsync), doesn't use mmap for other reasons, > from rsync-3.1.3/fileio.c: > > /* This provides functionality somewhat similar to mmap() but using read(). > * It gives sliding window access to a file. mmap() is not used because of > * the possibility of another program (such as a mailer) truncating the > * file thus giving us a SIGBUS. */ It should be rewritten to not use mmap. mmap is not needed in this case, since it is a network speaker. Meanwhile, fallback code isn't present. Pretty ridiculous..
Re: openrsync out of memory
By the looks of it, openrsync does attempt to map the entire file, from usr.bin/rsync/uploader.c: mapsz = st.st_size; map = mmap(NULL, mapsz, PROT_READ, MAP_SHARED, *fileinfd, 0); The likely reason for your out of memory error is the default datasize in login.conf. IIRC on some arches it's set to 768MB by default, which would allow your 300MB file to transfer, but would cause mmap to fail upon attempting to map the 1.6GB one. Increasing the default limits in /etc/login.conf should fix the problem. Note that rsync (not openrsync), doesn't use mmap for other reasons, from rsync-3.1.3/fileio.c: /* This provides functionality somewhat similar to mmap() but using read(). * It gives sliding window access to a file. mmap() is not used because of * the possibility of another program (such as a mailer) truncating the * file thus giving us a SIGBUS. */ Cheers, Joe
openrsync out of memory
This is 6.5-current on an old ALIX (dmesg bellow).
I am syncing its backups to a remote machine with openrsync.
It works on all files except one that is big:
196M/backup/gw.stare.cz/dump.home.0
32.0K /backup/gw.stare.cz/dump.home.1
32.0K /backup/gw.stare.cz/dump.home.2
32.0K /backup/gw.stare.cz/dump.home.3
64.0K /backup/gw.stare.cz/dump.home.4
75.3M /backup/gw.stare.cz/dump.root.0
64.0K /backup/gw.stare.cz/dump.root.1
64.0K /backup/gw.stare.cz/dump.root.2
64.0K /backup/gw.stare.cz/dump.root.3
64.0K /backup/gw.stare.cz/dump.root.4
10.2M /backup/gw.stare.cz/dump.var.0
2.1M/backup/gw.stare.cz/dump.var.1
2.1M/backup/gw.stare.cz/dump.var.2
2.1M/backup/gw.stare.cz/dump.var.3
2.2M/backup/gw.stare.cz/dump.var.4
1.6G/backup/gw.stare.cz/dump.var.log.0
$ openrsync -av --del /backup/gw.stare.cz/ remote:/backup/gw.stare.cz
(sits there for a long time once it gets to the big file)
/usr/src/usr.bin/rsync/sender.c:550: error: /backup/gw.stare.cz/dump.var.log.0:
mmap: Cannot allocate memory
/usr/src/usr.bin/rsync/client.c:86: error: rsync_sender
According to ktrace,
66400 openrsync STRU struct stat { dev=9, ino=16224, mode=-rw--- , nlink=1
, uid=1000<"hans">, gid=0<"wheel">, rdev=115088, atime=1562283014<"Jul 5 01:30:
14 2019">.166197875, mtime=1565912129<"Aug 16 01:35:29 2019">.689322159, ctime=1
565912129<"Aug 16 01:35:29 2019">.689322159, size=1692231680, blocks=3305600, bl
ksize=65536, flags=0x0, gen=0x31cdf9ee }
66400 openrsync RET fstat 0
66400 openrsync CALL kbind(0xcf7d08b0,12,0xd5a85aa662cb95c1)
66400 openrsync RET kbind 0
66400 openrsync CALL mmap(0,0x64dd6800,0x1,0x1,4,0)
66400 openrsync RET mmap -1 errno 12 Cannot allocate memory
Does that mean openrsync tries to mmap() the entire file?
The machine only has 256MB of memory, but it does transfer
a test file of 300MB, so that can't be it.
Jan
OpenBSD 6.5-current (GENERIC) #6: Sat May 4 19:43:01 MDT 2019
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
real mem = 267931648 (255MB)
avail mem = 247394304 (235MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 11/05/08, BIOS32 rev. 0 @ 0xfd088
pcibios0 at bios0: rev 2.1 @ 0xf/0x1
pcibios0: pcibios_get_intr_routing - function not supported
pcibios0: PCI IRQ Routing information unavailable.
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xe/0xa800
cpu0 at mainbus0: (uniprocessor)
cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD" 586-class) 499
MHz, 05-0a-02
cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW
mtrr: K6-family MTRR support (2 registers)
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x33
glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES
vr0 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 10, address
00:0d:b9:1a:a4:10
ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI 0x004063,
model 0x0034
vr1 at pci0 dev 10 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11, address
00:0d:b9:1a:a4:11
ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI 0x004063,
model 0x0034
vr2 at pci0 dev 11 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 15, address
00:0d:b9:1a:a4:12
ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI 0x004063,
model 0x0034
athn0 at pci0 dev 12 function 0 "Atheros AR9280" rev 0x01: irq 9
athn0: AR9280 rev 2 (2T2R), ROM rev 22, address 04:f0:21:01:d6:86
glxpcib0 at pci0 dev 15 function 0 "AMD CS5536 ISA" rev 0x03: rev 3, 32-bit
3579545Hz timer, watchdog, gpio, i2c
gpio0 at glxpcib0: 32 pins
iic0 at glxpcib0
maxtmp0 at iic0 addr 0x4c: lm86
pciide0 at pci0 dev 15 function 2 "AMD CS5536 IDE" rev 0x01: DMA, channel 0
wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0:
wd0: 1-sector PIO, LBA48, 15279MB, 31293360 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
ohci0 at pci0 dev 15 function 4 "AMD CS5536 USB" rev 0x02: irq 12, version 1.0,
legacy support
ehci0 at pci0 dev 15 function 5 "AMD CS5536 USB" rev 0x02: irq 12
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00
addr 1
isa0 at glxpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00
addr 1
nvram: invalid checksum
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (9cd0e5ba033bd225.a) swap on wd0b dump on wd0b
clock: unknown CMOS layout
