[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 closed https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 updated
https://github.com/llvm/llvm-project/pull/84410
>From 1cb6905e726b1f6c6cb71ea1e4d9ab4405aedeaf Mon Sep 17 00:00:00 2001
From: Eduard Zingerman
Date: Fri, 26 Jan 2024 04:18:32 +0200
Subject: [PATCH] [BPF] Add addr_space_cast BPF instruction
addr_space_cast instruction
===
This commit aims to support BPF arena kernel side feature [0]:
- arena is a memory region accessible from both
BPF program and userspace;
- base pointers for this memory region differ between
kernel and user spaces;
- `dst_reg = addr_space_cast(src_reg, dst_addr_space, src_addr_space)`
translates src_reg, a pointer in src_addr_space
to dst_reg, equivalent pointer in dst_addr_space,
{src,dst}_addr_space are immediate constants;
- number 0 is assigned to kernel address space;
- number 1 is assigned to user address space.
[0]
https://lore.kernel.org/bpf/20240206220441.38311-1-alexei.starovoi...@gmail.com/
On the LLVM side, the goal is to make load and store operations on
arena pointers "transparent" for BPF programs:
- assume that pointers with non-zero address space are pointers to
arena memory;
- assume that arena is identified by address space number;
- assume that address space zero corresponds to kernel address space;
- assume that every BPF-side load or store from arena is done via
pointer in user address space, thus convert base pointers using
`addr_space_cast(src_reg, 0, 1)`;
Only load, store, cmpxchg and atomicrmw IR instructions are handled by
this transformation.
For example, the following C code:
#define __as __attribute__((address_space(1)))
void copy(int __as *from, int __as *to) { *to = *from; }
Compiled to the following IR:
define void @copy(ptr addrspace(1) %from, ptr addrspace(1) %to) {
entry:
%0 = load i32, ptr addrspace(1) %from, align 4
store i32 %0, ptr addrspace(1) %to, align 4
ret void
}
Is transformed to:
%to2 = addrspacecast ptr addrspace(1) %to to ptr ;; !
%from1 = addrspacecast ptr addrspace(1) %from to ptr ;; !
%0 = load i32, ptr %from1, align 4, !tbaa !3
store i32 %0, ptr %to2, align 4, !tbaa !3
ret void
And compiled as:
r2 = addr_space_cast(r2, 0, 1)
r1 = addr_space_cast(r1, 0, 1)
r1 = *(u32 *)(r1 + 0)
*(u32 *)(r2 + 0) = r1
exit
Internally:
- piggy-back `BPFCheckAndAdjustIR` pass to insert address space casts
for base pointer of memory access instructions, when base pointer
has non-zero address space;
- modify `BPFInstrInfo.td` and `BPFIselLowering.cpp` to allow
translation of `addrspacecast` instruction:
- define new machine instruction: `ADDR_SPACE_CAST`;
- define pattern to select `ADDR_SPACE_CAST` for `addrspacecast` ISD nodes.
Global variables in .arena section
==
Make it so that all globals within same address space reside in
section with name ".arena.N", where N is number of the address space.
E.g. for the following C program:
```c
__as const char a[2] = {1,2};
__as char b[2] = {3,4};
__as char c[2];
...
```
Generate the following layout:
```
$ clang -O2 --target=bpf t.c -c -o - \
| llvm-readelf --sections --symbols -
...
Section Headers:
[Nr] Name TypeAddress OffSize ES Flg
Lk Inf Al
...
[ 4] .arena.272PROGBITS e8 18 00 WA
0 0 4
...
Symbol table '.symtab' contains 8 entries:
Num:Value Size TypeBind Vis Ndx Name
...
3: 8 OBJECT GLOBAL DEFAULT 4 a
4: 0008 8 OBJECT GLOBAL DEFAULT 4 b
5: 0010 8 OBJECT GLOBAL DEFAULT 4 c
... ^^^
Note section index
```
Convert 'addrcast M->N -> GEP -> addrcast N->M' to just GEP
===
For BPF GEP would adjust pointer using same offset in any address
space, thus transformation of form:
%inner = addrspacecast N->M %ptr
%gep = getelementptr %inner, ...
%outer = addrspacecast M->N %gep
to just:
%gep = getelementptr %ptr, ...
is valid.
Applying such transformation helps with C patterns that use e.g.
(void *) casts to offsets w/o actual memory access:
#define container_of(ptr, type, member) \
({ \
void __arena *__mptr = (void *)(ptr); \
((type *)(__mptr - offsetof(type, member)));\
})
(Note the address space cast on first body line)
---
clang/lib/Basic/Targets/BPF.cpp | 3 +
.../test/Preprocessor/bpf-predefined-macros.c | 8 ++
.../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 1 +
llvm/lib/Target/BPF/BPF.h | 8 ++
.../Target/BPF/BPFASpaceCastSimplifyPass.cpp | 92 ++
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
UTC_ARGS: --version 4
+; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s
+
+; Generated from the following C code:
+;
+; extern int __uptr *magic1();
+; extern int __uptr *magic2();
+;
+; void test(long i) {
+; int __uptr *a;
+;
+; if (i > 42)
+; a = magic1();
+; else
+; a = magic2();
+; a[5] = 7;
+; }
+;
+; Using the following command:
+;
+; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c
+
+; Function Attrs: nounwind
eddyz87 wrote:
Done.
https://github.com/llvm/llvm-project/pull/84410
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 updated
https://github.com/llvm/llvm-project/pull/84410
>From 503c1abc8dd63ec1500d1ed867a4bfefc1aed062 Mon Sep 17 00:00:00 2001
From: Eduard Zingerman
Date: Fri, 26 Jan 2024 04:18:32 +0200
Subject: [PATCH 1/4] [BPF] Add addr_space_cast BPF instruction
This commit aims to support BPF arena kernel side feature [0]:
- arena is a memory region accessible from both
BPF program and userspace;
- base pointers for this memory region differ between
kernel and user spaces;
- `dst_reg = addr_space_cast(src_reg, dst_addr_space, src_addr_space)`
translates src_reg, a pointer in src_addr_space
to dst_reg, equivalent pointer in dst_addr_space,
{src,dst}_addr_space are immediate constants;
- number 0 is assigned to kernel address space;
- number 1 is assigned to user address space.
On the LLVM side, the goal is to make load and store operations on
arena pointers "transparent" for BPF programs:
- assume that pointers with non-zero address space are pointers to
arena memory;
- assume that arena is identified by address space number;
- assume that address space zero corresponds to kernel address space;
- assume that every BPF-side load or store from arena is done via
pointer in user address space, thus convert base pointers using
`addr_space_cast(src_reg, 0, 1)`;
Only load, store, cmpxchg and atomicrmw IR instructions are handled by
this transformation.
For example, the following C code:
#define __as __attribute__((address_space(1)))
void copy(int __as *from, int __as *to) { *to = *from; }
Compiled to the following IR:
define void @copy(ptr addrspace(1) %from, ptr addrspace(1) %to) {
entry:
%0 = load i32, ptr addrspace(1) %from, align 4
store i32 %0, ptr addrspace(1) %to, align 4
ret void
}
Is transformed to:
%to2 = addrspacecast ptr addrspace(1) %to to ptr ;; !
%from1 = addrspacecast ptr addrspace(1) %from to ptr ;; !
%0 = load i32, ptr %from1, align 4, !tbaa !3
store i32 %0, ptr %to2, align 4, !tbaa !3
ret void
And compiled as:
r2 = addr_space_cast(r2, 0, 1)
r1 = addr_space_cast(r1, 0, 1)
r1 = *(u32 *)(r1 + 0)
*(u32 *)(r2 + 0) = r1
exit
Internally:
- piggy-back `BPFCheckAndAdjustIR` pass to insert address space casts
for base pointer of memory access instructions, when base pointer
has non-zero address space;
- modify `BPFInstrInfo.td` and `BPFIselLowering.cpp` to allow
translation of `addrspacecast` instruction:
- define new machine instruction: `ADDR_SPACE_CAST`;
- define pattern to select `ADDR_SPACE_CAST` for `addrspacecast` ISD nodes.
[0]
https://lore.kernel.org/bpf/20240206220441.38311-1-alexei.starovoi...@gmail.com/
---
.../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 1 +
llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp | 104 ++
llvm/lib/Target/BPF/BPFInstrInfo.td | 29 +
.../test/CodeGen/BPF/addr-space-auto-casts.ll | 78 +
llvm/test/CodeGen/BPF/addr-space-cast.ll | 22
llvm/test/CodeGen/BPF/addr-space-gep-chain.ll | 25 +
llvm/test/CodeGen/BPF/addr-space-phi.ll | 53 +
.../test/CodeGen/BPF/assembler-disassembler.s | 7 ++
8 files changed, 319 insertions(+)
create mode 100644 llvm/test/CodeGen/BPF/addr-space-auto-casts.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-cast.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-phi.ll
diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
index 0d1eef60c3b550..3145bc3d19f5dc 100644
--- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
+++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
@@ -271,6 +271,7 @@ struct BPFOperand : public MCParsedAsmOperand {
.Case("xchg32_32", true)
.Case("cmpxchg_64", true)
.Case("cmpxchg32_32", true)
+.Case("addr_space_cast", true)
.Default(false);
}
};
diff --git a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
index 81effc9b1db46c..d39a6b57aafed4 100644
--- a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
+++ b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
@@ -14,6 +14,8 @@
// optimizations are done and those builtins can be removed.
// - remove llvm.bpf.getelementptr.and.load builtins.
// - remove llvm.bpf.getelementptr.and.store builtins.
+// - for loads and stores with base addresses from non-zero address space
+// cast base address to zero address space (support for BPF arenas).
//
//===--===//
@@ -55,6 +57,7 @@ class BPFCheckAndAdjustIR final : public ModulePass {
bool removeCompareBuiltin(Module );
bool sinkMinMax(Module );
bool removeGEPBuiltins(Module );
+ bool insertASpaceCasts(Module );
};
} // End anonymous namespace
@@ -416,11 +419,112 @@ bool
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
UTC_ARGS: --version 4
+; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s
+
+; Generated from the following C code:
+;
+; extern int __uptr *magic1();
+; extern int __uptr *magic2();
+;
+; void test(long i) {
+; int __uptr *a;
+;
+; if (i > 42)
+; a = magic1();
+; else
+; a = magic2();
+; a[5] = 7;
+; }
+;
+; Using the following command:
+;
+; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c
+
+; Function Attrs: nounwind
eddyz87 wrote:
Oh, I see what you mean. I cleaned up the attributes by hand because those were
not necessary for the test and have not adjusted the comment. Will change,
thank you.
https://github.com/llvm/llvm-project/pull/84410
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/inclyc edited https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
UTC_ARGS: --version 4
+; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s
+
+; Generated from the following C code:
+;
+; extern int __uptr *magic1();
+; extern int __uptr *magic2();
+;
+; void test(long i) {
+; int __uptr *a;
+;
+; if (i > 42)
+; a = magic1();
+; else
+; a = magic2();
+; a[5] = 7;
+; }
+;
+; Using the following command:
+;
+; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c
+
+; Function Attrs: nounwind
inclyc wrote:
No, it do has 'nounwind' attr because it is attributed with `#0`, however, in
our testing ir I see `#0` is omitted
https://github.com/llvm/llvm-project/pull/84410
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 approved this pull request. https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 edited https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 updated
https://github.com/llvm/llvm-project/pull/84410
>From 0575ca506badd58d870ce03b1f2e26e7a3d6168b Mon Sep 17 00:00:00 2001
From: Eduard Zingerman
Date: Fri, 26 Jan 2024 04:18:32 +0200
Subject: [PATCH 1/4] [BPF] Add addr_space_cast BPF instruction
This commit aims to support BPF arena kernel side feature [0]:
- arena is a memory region accessible from both
BPF program and userspace;
- base pointers for this memory region differ between
kernel and user spaces;
- `dst_reg = addr_space_cast(src_reg, dst_addr_space, src_addr_space)`
translates src_reg, a pointer in src_addr_space
to dst_reg, equivalent pointer in dst_addr_space,
{src,dst}_addr_space are immediate constants;
- number 0 is assigned to kernel address space;
- number 1 is assigned to user address space.
On the LLVM side, the goal is to make load and store operations on
arena pointers "transparent" for BPF programs:
- assume that pointers with non-zero address space are pointers to
arena memory;
- assume that arena is identified by address space number;
- assume that address space zero corresponds to kernel address space;
- assume that every BPF-side load or store from arena is done via
pointer in user address space, thus convert base pointers using
`addr_space_cast(src_reg, 0, 1)`;
Only load, store, cmpxchg and atomicrmw IR instructions are handled by
this transformation.
For example, the following C code:
#define __as __attribute__((address_space(1)))
void copy(int __as *from, int __as *to) { *to = *from; }
Compiled to the following IR:
define void @copy(ptr addrspace(1) %from, ptr addrspace(1) %to) {
entry:
%0 = load i32, ptr addrspace(1) %from, align 4
store i32 %0, ptr addrspace(1) %to, align 4
ret void
}
Is transformed to:
%to2 = addrspacecast ptr addrspace(1) %to to ptr ;; !
%from1 = addrspacecast ptr addrspace(1) %from to ptr ;; !
%0 = load i32, ptr %from1, align 4, !tbaa !3
store i32 %0, ptr %to2, align 4, !tbaa !3
ret void
And compiled as:
r2 = addr_space_cast(r2, 0, 1)
r1 = addr_space_cast(r1, 0, 1)
r1 = *(u32 *)(r1 + 0)
*(u32 *)(r2 + 0) = r1
exit
Internally:
- piggy-back `BPFCheckAndAdjustIR` pass to insert address space casts
for base pointer of memory access instructions, when base pointer
has non-zero address space;
- modify `BPFInstrInfo.td` and `BPFIselLowering.cpp` to allow
translation of `addrspacecast` instruction:
- define new machine instruction: `ADDR_SPACE_CAST`;
- define pattern to select `ADDR_SPACE_CAST` for `addrspacecast` ISD nodes.
[0]
https://lore.kernel.org/bpf/20240206220441.38311-1-alexei.starovoi...@gmail.com/
---
.../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 1 +
llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp | 104 ++
llvm/lib/Target/BPF/BPFInstrInfo.td | 29 +
.../test/CodeGen/BPF/addr-space-auto-casts.ll | 78 +
llvm/test/CodeGen/BPF/addr-space-cast.ll | 22
llvm/test/CodeGen/BPF/addr-space-gep-chain.ll | 25 +
llvm/test/CodeGen/BPF/addr-space-phi.ll | 54 +
.../test/CodeGen/BPF/assembler-disassembler.s | 7 ++
8 files changed, 320 insertions(+)
create mode 100644 llvm/test/CodeGen/BPF/addr-space-auto-casts.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-cast.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-phi.ll
diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
index 0d1eef60c3b550..3145bc3d19f5dc 100644
--- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
+++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
@@ -271,6 +271,7 @@ struct BPFOperand : public MCParsedAsmOperand {
.Case("xchg32_32", true)
.Case("cmpxchg_64", true)
.Case("cmpxchg32_32", true)
+.Case("addr_space_cast", true)
.Default(false);
}
};
diff --git a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
index 81effc9b1db46c..d39a6b57aafed4 100644
--- a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
+++ b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
@@ -14,6 +14,8 @@
// optimizations are done and those builtins can be removed.
// - remove llvm.bpf.getelementptr.and.load builtins.
// - remove llvm.bpf.getelementptr.and.store builtins.
+// - for loads and stores with base addresses from non-zero address space
+// cast base address to zero address space (support for BPF arenas).
//
//===--===//
@@ -55,6 +57,7 @@ class BPFCheckAndAdjustIR final : public ModulePass {
bool removeCompareBuiltin(Module );
bool sinkMinMax(Module );
bool removeGEPBuiltins(Module );
+ bool insertASpaceCasts(Module );
};
} // End anonymous namespace
@@ -416,11 +419,112 @@ bool
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,92 @@ +//=== BPFIRPeephole.cpp - IR Peephole Transformation --===// eddyz87 wrote: My bad. Thank you for taking a look at this MR, will push updates with fixes soon. https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
UTC_ARGS: --version 4
+; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s
+
+; Generated from the following C code:
+;
+; extern int __uptr *magic1();
+; extern int __uptr *magic2();
+;
+; void test(long i) {
+; int __uptr *a;
+;
+; if (i > 42)
+; a = magic1();
+; else
+; a = magic2();
+; a[5] = 7;
+; }
+;
+; Using the following command:
+;
+; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c
+
+; Function Attrs: nounwind
eddyz87 wrote:
Retested this, the attrs are like in the test:
```c
$ cat test.c
#define __uptr __attribute__((address_space(1)))
extern int __uptr *magic1();
extern int __uptr *magic2();
void test(long i) {
int __uptr *a;
if (i > 42)
a = magic1();
else
a = magic2();
a[5] = 7;
}
$ clang --target=bpf -O2 -S -emit-llvm -o - test.c | head -n10
; ModuleID = 'test.c'
source_filename = "test.c"
target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
target triple = "bpf"
; Function Attrs: nounwind
define dso_local void @test(i64 noundef %i) local_unnamed_addr #0 {
entry:
%cmp = icmp sgt i64 %i, 42
br i1 %cmp, label %if.then, label %if.else
```
https://github.com/llvm/llvm-project/pull/84410
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -31,6 +31,9 @@ def SDT_BPFMEMCPY : SDTypeProfile<0, 4, [SDTCisVT<0,
i64>,
SDTCisVT<1, i64>,
SDTCisVT<2, i64>,
SDTCisVT<3, i64>]>;
+def SDT_BPFAddrSpaceCast : SDTypeProfile<0, 3, [SDTCisPtrTy<0>,
eddyz87 wrote:
That works, thanks for the tip. I ended up with the following `td` definitions:
```llvm
def ADDR_SPACE_CAST
: ALU_RR {
bits<64> dst_as;
bits<64> src_as;
let Inst{47-32} = 1;
let Inst{31-16} = dst_as{15-0};
let Inst{15-0} = src_as{15-0};
}
def SrcAddrSpace : SDNodeXFormgetTargetConstant(
cast(N)->getSrcAddressSpace(),
SDLoc(N), MVT::i64);
}]>;
def DstAddrSpace : SDNodeXFormgetTargetConstant(
cast(N)->getDestAddressSpace(),
SDLoc(N), MVT::i64);
}]>;
def : Pat<(addrspacecast:$this GPR:$src),
(ADDR_SPACE_CAST $src, (DstAddrSpace $this), (SrcAddrSpace $this))>;
```
https://github.com/llvm/llvm-project/pull/84410
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -638,6 +643,31 @@ SDValue BPFTargetLowering::LowerDYNAMIC_STACKALLOC(SDValue
Op,
return DAG.getMergeValues(Ops, SDLoc());
}
+SDValue BPFTargetLowering::LowerADDRSPACECAST(SDValue Op,
+ SelectionDAG ) const {
+ auto *ACast = dyn_cast(Op);
+ const SDValue = ACast->getOperand(0);
+ unsigned SrcAS = ACast->getSrcAddressSpace();
+ unsigned DstAS = ACast->getDestAddressSpace();
+ SDLoc DL(Op);
+
+ if (SrcAS > 0 && DstAS > 0) {
+SmallString<64> Msg;
+raw_svector_ostream OS(Msg);
+OS << "Can't cast address space " << SrcAS << " to " << DstAS
+ << ": either source or destination address space has to be zero";
+fail(DL, DAG, Msg);
eddyz87 wrote:
Actually, I think this check is no longer necessary. It remains from a previous
version, where `cast_kern` and `cast_user` instructions were used, modeled as
an address space cast at IR level. There either source or destination address
space had to be zero. Sorry for confusion, I should have removed this when
switching MR to `addr_space_cast` instruction.
https://github.com/llvm/llvm-project/pull/84410
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -31,6 +31,9 @@ def SDT_BPFMEMCPY : SDTypeProfile<0, 4, [SDTCisVT<0, i64>, SDTCisVT<1, i64>, SDTCisVT<2, i64>, SDTCisVT<3, i64>]>; +def SDT_BPFAddrSpaceCast : SDTypeProfile<0, 3, [SDTCisPtrTy<0>, inclyc wrote: (question) Why not just use the common "addrspacecast" ISD node? https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,92 @@ +//=== BPFIRPeephole.cpp - IR Peephole Transformation --===// inclyc wrote: nit https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -638,6 +643,31 @@ SDValue BPFTargetLowering::LowerDYNAMIC_STACKALLOC(SDValue
Op,
return DAG.getMergeValues(Ops, SDLoc());
}
+SDValue BPFTargetLowering::LowerADDRSPACECAST(SDValue Op,
+ SelectionDAG ) const {
+ auto *ACast = dyn_cast(Op);
+ const SDValue = ACast->getOperand(0);
+ unsigned SrcAS = ACast->getSrcAddressSpace();
+ unsigned DstAS = ACast->getDestAddressSpace();
+ SDLoc DL(Op);
+
+ if (SrcAS > 0 && DstAS > 0) {
+SmallString<64> Msg;
+raw_svector_ostream OS(Msg);
+OS << "Can't cast address space " << SrcAS << " to " << DstAS
+ << ": either source or destination address space has to be zero";
+fail(DL, DAG, Msg);
inclyc wrote:
Our `fail` method is used for `DiagnosticInfoUnsupported` diagnostic, should we
create a new diagnostic kind?
https://github.com/llvm/llvm-project/pull/84410
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
UTC_ARGS: --version 4
+; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s
+
+; Generated from the following C code:
+;
+; extern int __uptr *magic1();
+; extern int __uptr *magic2();
+;
+; void test(long i) {
+; int __uptr *a;
+;
+; if (i > 42)
+; a = magic1();
+; else
+; a = magic2();
+; a[5] = 7;
+; }
+;
+; Using the following command:
+;
+; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c
+
+; Function Attrs: nounwind
inclyc wrote:
this function attrs looks outdated
https://github.com/llvm/llvm-project/pull/84410
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
llvmbot wrote:
@llvm/pr-subscribers-clang
Author: None (4ast)
Changes
---
Patch is 34.73 KiB, truncated to 20.00 KiB below, full version:
https://github.com/llvm/llvm-project/pull/84410.diff
23 Files Affected:
- (modified) clang/lib/Basic/Targets/BPF.cpp (+3)
- (modified) clang/test/Preprocessor/bpf-predefined-macros.c (+8)
- (modified) llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp (+1)
- (modified) llvm/lib/Target/BPF/BPF.h (+8)
- (added) llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp (+92)
- (modified) llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp (+116)
- (modified) llvm/lib/Target/BPF/BPFISelLowering.cpp (+32)
- (modified) llvm/lib/Target/BPF/BPFISelLowering.h (+3-1)
- (modified) llvm/lib/Target/BPF/BPFInstrInfo.td (+21)
- (modified) llvm/lib/Target/BPF/BPFTargetMachine.cpp (+5)
- (modified) llvm/lib/Target/BPF/CMakeLists.txt (+1)
- (added) llvm/test/CodeGen/BPF/addr-space-auto-casts.ll (+78)
- (added) llvm/test/CodeGen/BPF/addr-space-cast.ll (+22)
- (added) llvm/test/CodeGen/BPF/addr-space-gep-chain.ll (+25)
- (added) llvm/test/CodeGen/BPF/addr-space-globals.ll (+30)
- (added) llvm/test/CodeGen/BPF/addr-space-globals2.ll (+25)
- (added) llvm/test/CodeGen/BPF/addr-space-phi.ll (+52)
- (added) llvm/test/CodeGen/BPF/addr-space-simplify-1.ll (+19)
- (added) llvm/test/CodeGen/BPF/addr-space-simplify-2.ll (+21)
- (added) llvm/test/CodeGen/BPF/addr-space-simplify-3.ll (+26)
- (added) llvm/test/CodeGen/BPF/addr-space-simplify-4.ll (+21)
- (added) llvm/test/CodeGen/BPF/addr-space-simplify-5.ll (+25)
- (modified) llvm/test/CodeGen/BPF/assembler-disassembler.s (+7)
``diff
diff --git a/clang/lib/Basic/Targets/BPF.cpp b/clang/lib/Basic/Targets/BPF.cpp
index e3fbbb720d0694..26a54f631fcfc4 100644
--- a/clang/lib/Basic/Targets/BPF.cpp
+++ b/clang/lib/Basic/Targets/BPF.cpp
@@ -35,6 +35,9 @@ void BPFTargetInfo::getTargetDefines(const LangOptions ,
Builder.defineMacro("__BPF_CPU_VERSION__", "0");
return;
}
+
+ Builder.defineMacro("__BPF_FEATURE_ARENA_CAST");
+
if (CPU.empty() || CPU == "generic" || CPU == "v1") {
Builder.defineMacro("__BPF_CPU_VERSION__", "1");
return;
diff --git a/clang/test/Preprocessor/bpf-predefined-macros.c
b/clang/test/Preprocessor/bpf-predefined-macros.c
index ff4d00ac3bcfcc..fea24d1ea0ff7b 100644
--- a/clang/test/Preprocessor/bpf-predefined-macros.c
+++ b/clang/test/Preprocessor/bpf-predefined-macros.c
@@ -61,6 +61,9 @@ int r;
#ifdef __BPF_FEATURE_ST
int s;
#endif
+#ifdef __BPF_FEATURE_ARENA_CAST
+int t;
+#endif
// CHECK: int b;
// CHECK: int c;
@@ -90,6 +93,11 @@ int s;
// CPU_V4: int r;
// CPU_V4: int s;
+// CPU_V1: int t;
+// CPU_V2: int t;
+// CPU_V3: int t;
+// CPU_V4: int t;
+
// CPU_GENERIC: int g;
// CPU_PROBE: int f;
diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
index 90697c6645be2f..3fbd14facaf804 100644
--- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
+++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
@@ -270,6 +270,7 @@ struct BPFOperand : public MCParsedAsmOperand {
.Case("xchg32_32", true)
.Case("cmpxchg_64", true)
.Case("cmpxchg32_32", true)
+.Case("addr_space_cast", true)
.Default(false);
}
};
diff --git a/llvm/lib/Target/BPF/BPF.h b/llvm/lib/Target/BPF/BPF.h
index 5c77d183e1ef3d..bbdbdbbde53228 100644
--- a/llvm/lib/Target/BPF/BPF.h
+++ b/llvm/lib/Target/BPF/BPF.h
@@ -66,6 +66,14 @@ class BPFIRPeepholePass : public
PassInfoMixin {
static bool isRequired() { return true; }
};
+class BPFASpaceCastSimplifyPass
+: public PassInfoMixin {
+public:
+ PreservedAnalyses run(Function , FunctionAnalysisManager );
+
+ static bool isRequired() { return true; }
+};
+
class BPFAdjustOptPass : public PassInfoMixin {
public:
PreservedAnalyses run(Module , ModuleAnalysisManager );
diff --git a/llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp
b/llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp
new file mode 100644
index 00..d32cecc073fab9
--- /dev/null
+++ b/llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp
@@ -0,0 +1,92 @@
+//=== BPFIRPeephole.cpp - IR Peephole Transformation
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "BPF.h"
+#include
+
+#define DEBUG_TYPE "bpf-aspace-simplify"
+
+using namespace llvm;
+
+namespace {
+
+struct CastGEPCast {
+ AddrSpaceCastInst *OuterCast;
+
+ // Match chain of instructions:
+ // %inner = addrspacecast N->M
+ // %gep = getelementptr %inner, ...
+ // %outer = addrspacecast M->N %gep
+ // Where I is %outer.
+ static std::optional match(Value *I) {
+auto *OuterCast = dyn_cast(I);
+if (!OuterCast)
+ return
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/4ast created https://github.com/llvm/llvm-project/pull/84410
None
>From dd19f019066852dba15259cd201ce1b94fa5b6e6 Mon Sep 17 00:00:00 2001
From: Eduard Zingerman
Date: Fri, 26 Jan 2024 04:18:32 +0200
Subject: [PATCH 1/4] [BPF] Add addr_space_cast BPF instruction
This commit aims to support BPF arena kernel side feature [0]:
- arena is a memory region accessible from both
BPF program and userspace;
- base pointers for this memory region differ between
kernel and user spaces;
- `dst_reg = addr_space_cast(src_reg, dst_addr_space, src_addr_space)`
translates src_reg, a pointer in src_addr_space
to dst_reg, equivalent pointer in dst_addr_space,
{src,dst}_addr_space are immediate constants;
- number 0 is assigned to kernel address space;
- number 1 is assigned to user address space.
On the LLVM side, the goal is to make load and store operations on
arena pointers "transparent" for BPF programs:
- assume that pointers with non-zero address space are pointers to
arena memory;
- assume that arena is identified by address space number;
- assume that address space zero corresponds to kernel address space;
- assume that every BPF-side load or store from arena is done via
pointer in user address space, thus convert base pointers using
`addr_space_cast(src_reg, 0, 1)`;
Only load, store, cmpxchg and atomicrmw IR instructions are handled by
this transformation.
For example, the following C code:
#define __as __attribute__((address_space(1)))
void copy(int __as *from, int __as *to) { *to = *from; }
Compiled to the following IR:
define void @copy(ptr addrspace(1) %from, ptr addrspace(1) %to) {
entry:
%0 = load i32, ptr addrspace(1) %from, align 4
store i32 %0, ptr addrspace(1) %to, align 4
ret void
}
Is transformed to:
%to2 = addrspacecast ptr addrspace(1) %to to ptr ;; !
%from1 = addrspacecast ptr addrspace(1) %from to ptr ;; !
%0 = load i32, ptr %from1, align 4, !tbaa !3
store i32 %0, ptr %to2, align 4, !tbaa !3
ret void
And compiled as:
r2 = addr_space_cast(r2, 0, 1)
r1 = addr_space_cast(r1, 0, 1)
r1 = *(u32 *)(r1 + 0)
*(u32 *)(r2 + 0) = r1
exit
Internally:
- piggy-back `BPFCheckAndAdjustIR` pass to insert address space casts
for base pointer of memory access instructions, when base pointer
has non-zero address space;
- modify `BPFInstrInfo.td` and `BPFIselLowering.cpp` to allow
translation of `addrspacecast` instruction:
- define new SDNode type `BPFAddrSpaceCast`;
- lower `addrspacecast` as such SDNode;
- define new machine instruction: `ADDR_SPACE_CAST`;
- define pattern to select `ADDR_SPACE_CAST` for `BPFAddrSpaceCast` nodes.
[0]
https://lore.kernel.org/bpf/20240206220441.38311-1-alexei.starovoi...@gmail.com/
---
.../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 1 +
llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp | 104 ++
llvm/lib/Target/BPF/BPFISelLowering.cpp | 32 ++
llvm/lib/Target/BPF/BPFISelLowering.h | 4 +-
llvm/lib/Target/BPF/BPFInstrInfo.td | 21
.../test/CodeGen/BPF/addr-space-auto-casts.ll | 78 +
llvm/test/CodeGen/BPF/addr-space-cast.ll | 22
llvm/test/CodeGen/BPF/addr-space-gep-chain.ll | 25 +
llvm/test/CodeGen/BPF/addr-space-phi.ll | 52 +
.../test/CodeGen/BPF/assembler-disassembler.s | 7 ++
10 files changed, 345 insertions(+), 1 deletion(-)
create mode 100644 llvm/test/CodeGen/BPF/addr-space-auto-casts.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-cast.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll
create mode 100644 llvm/test/CodeGen/BPF/addr-space-phi.ll
diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
index 90697c6645be2f..3fbd14facaf804 100644
--- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
+++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp
@@ -270,6 +270,7 @@ struct BPFOperand : public MCParsedAsmOperand {
.Case("xchg32_32", true)
.Case("cmpxchg_64", true)
.Case("cmpxchg32_32", true)
+.Case("addr_space_cast", true)
.Default(false);
}
};
diff --git a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
index 81effc9b1db46c..d39a6b57aafed4 100644
--- a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
+++ b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
@@ -14,6 +14,8 @@
// optimizations are done and those builtins can be removed.
// - remove llvm.bpf.getelementptr.and.load builtins.
// - remove llvm.bpf.getelementptr.and.store builtins.
+// - for loads and stores with base addresses from non-zero address space
+// cast base address to zero address space (support for BPF arenas).
//
//===--===//
@@ -55,6 +57,7 @@ class BPFCheckAndAdjustIR final : public ModulePass {
bool
