[
https://issues.apache.org/jira/browse/ARROW-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470504#comment-17470504
]
Jeroen van Straten commented on ARROW-7051:
-------------------------------------------
I'll look at this for one of my first issues.
Idea: can we use mmap for this? On my machine anyway, mmap'ing a read-only,
anonymous, and private region will give fully zeroed virtual memory where (I
guess) all pages are mapped to the same physical page, or maybe some x86 MMU
tricks are done to avoid even that. In any case, the following code results in
no physical memory being allocated at all (not even a single 4kiB page,
apparently):
{code:java|title=zero.c|borderStyle=solid}
#include <sys/mman.h>
#include <stdio.h>
int main() {
char *addr;
const size_t length = 1024ull*1024*1024*64; // 64 GiB, more memory than my
machine has
addr = (char*)mmap(0, length, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS |
MAP_NORESERVE, -1, 0);
if (!addr) {
perror("mmap");
return 1;
}
// Just to force the kernel to populate the page table by having the CPU
read from every page in the mapping:
size_t pos = 0;
char total = 0;
while (pos < length) {
total += addr[pos];
pos += 4096; // page size
}
printf("total: %d (should be 0)\n", (int)total);
// Wait for input so resource usage can be inspected.
getchar();
munmap(addr, length);
}
{code}
Also:
- CPU cache layers using physical addressing will not be filled with useless
zeros.
- If anything tries to write to the zeroed region, the application will
immediately crash, rather than continuing with broken invariants.
The only problem I see is that I don't know how portable this is, if at all.
But of course a fallback to some form of malloc + memset can always be done at
compile time.
> [C++] Improve MakeArrayOfNull to support creation of multiple arrays
> --------------------------------------------------------------------
>
> Key: ARROW-7051
> URL: https://issues.apache.org/jira/browse/ARROW-7051
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Affects Versions: 0.14.0
> Reporter: Ben Kietzman
> Assignee: Jeroen van Straten
> Priority: Minor
> Labels: beginner, good-first-issue
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> MakeArrayOfNull reuses a single buffer of {{0}} for all buffers in the array
> it creates. It could be extended to reuse that same buffer for all buffers in
> multiple arrays. This optimization will make RecordBatchProjector and
> ConcatenateTablesWithPromotion more memory efficient
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
