Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[David Miller]

From: Ingo Molnar 
Date: Sat, 11 Nov 2017 09:16:00 +0100

> Please don't apply it yet as the series is still under active
> discussion - for now

Fine, reverted.

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[David Miller]

From: Ingo Molnar 
Date: Sat, 11 Nov 2017 09:16:00 +0100

> Please don't apply it yet as the series is still under active
> discussion - for now

Fine, reverted.

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[Ingo Molnar]

* David Miller  wrote:

> From: Josef Bacik 
> Date: Tue,  7 Nov 2017 15:28:41 -0500
> 
> > I'm sending this through Dave since it'll conflict with other BPF changes 
> > in his
> > tree, but since it touches tracing as well Dave would like a review from
> > somebody on the tracing side.
>  ...
> > A lot of our error paths are not well tested because we have no good way of
> > injecting errors generically.  Some subystems (block, memory) have ways to
> > inject errors, but they are random so it's hard to get reproduceable 
> > results.
> > 
> > With BPF we can add determinism to our error injection.  We can use kprobes 
> > and
> > other things to verify we are injecting errors at the exact case we are 
> > trying
> > to test.  This patch gives us the tool to actual do the error injection 
> > part.
> > It is very simple, we just set the return value of the pt_regs we're given 
> > to
> > whatever we provide, and then override the PC with a dummy function that 
> > simply
> > returns.
> > 
> > Right now this only works on x86, but it would be simple enough to expand to
> > other architectures.  Thanks,
> 
> Series applied, thanks Josef.

Please don't apply it yet as the series is still under active discussion - for 
now 
I'm NAK-ing the x86 bits because I have second thoughts about the whole premise 
of 
the feature being added here.

Thanks,

Ingo

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[Ingo Molnar]

* David Miller  wrote:

> From: Josef Bacik 
> Date: Tue,  7 Nov 2017 15:28:41 -0500
> 
> > I'm sending this through Dave since it'll conflict with other BPF changes 
> > in his
> > tree, but since it touches tracing as well Dave would like a review from
> > somebody on the tracing side.
>  ...
> > A lot of our error paths are not well tested because we have no good way of
> > injecting errors generically.  Some subystems (block, memory) have ways to
> > inject errors, but they are random so it's hard to get reproduceable 
> > results.
> > 
> > With BPF we can add determinism to our error injection.  We can use kprobes 
> > and
> > other things to verify we are injecting errors at the exact case we are 
> > trying
> > to test.  This patch gives us the tool to actual do the error injection 
> > part.
> > It is very simple, we just set the return value of the pt_regs we're given 
> > to
> > whatever we provide, and then override the PC with a dummy function that 
> > simply
> > returns.
> > 
> > Right now this only works on x86, but it would be simple enough to expand to
> > other architectures.  Thanks,
> 
> Series applied, thanks Josef.

Please don't apply it yet as the series is still under active discussion - for 
now 
I'm NAK-ing the x86 bits because I have second thoughts about the whole premise 
of 
the feature being added here.

Thanks,

Ingo

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[David Miller]

From: Josef Bacik 
Date: Tue,  7 Nov 2017 15:28:41 -0500

> I'm sending this through Dave since it'll conflict with other BPF changes in 
> his
> tree, but since it touches tracing as well Dave would like a review from
> somebody on the tracing side.
 ...
> A lot of our error paths are not well tested because we have no good way of
> injecting errors generically.  Some subystems (block, memory) have ways to
> inject errors, but they are random so it's hard to get reproduceable results.
> 
> With BPF we can add determinism to our error injection.  We can use kprobes 
> and
> other things to verify we are injecting errors at the exact case we are trying
> to test.  This patch gives us the tool to actual do the error injection part.
> It is very simple, we just set the return value of the pt_regs we're given to
> whatever we provide, and then override the PC with a dummy function that 
> simply
> returns.
> 
> Right now this only works on x86, but it would be simple enough to expand to
> other architectures.  Thanks,

Series applied, thanks Josef.

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[David Miller]

From: Josef Bacik 
Date: Tue,  7 Nov 2017 15:28:41 -0500

> I'm sending this through Dave since it'll conflict with other BPF changes in 
> his
> tree, but since it touches tracing as well Dave would like a review from
> somebody on the tracing side.
 ...
> A lot of our error paths are not well tested because we have no good way of
> injecting errors generically.  Some subystems (block, memory) have ways to
> inject errors, but they are random so it's hard to get reproduceable results.
> 
> With BPF we can add determinism to our error injection.  We can use kprobes 
> and
> other things to verify we are injecting errors at the exact case we are trying
> to test.  This patch gives us the tool to actual do the error injection part.
> It is very simple, we just set the return value of the pt_regs we're given to
> whatever we provide, and then override the PC with a dummy function that 
> simply
> returns.
> 
> Right now this only works on x86, but it would be simple enough to expand to
> other architectures.  Thanks,

Series applied, thanks Josef.

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[Peter Zijlstra]

On Wed, Nov 08, 2017 at 06:43:25AM +0900, Alexei Starovoitov wrote:
> On 11/8/17 5:28 AM, Josef Bacik wrote:
> > I'm sending this through Dave since it'll conflict with other BPF changes 
> > in his
> > tree, but since it touches tracing as well Dave would like a review from
> > somebody on the tracing side.
> > 
> > v4->v5:
> > - disallow kprobe_override programs from being put in the prog map array so 
> > we
> >   don't tail call into something we didn't check.  This allows us to make 
> > the
> >   normal path still fast without a bunch of percpu operations.
> 
> looks great to me.
> Peter,
> could you please review x86 bits?

I did not in fact get the patch... But after digging it out of LKML I
don't immediately see something broken, but this is not stuff I
typically know well.

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[Peter Zijlstra]

On Wed, Nov 08, 2017 at 06:43:25AM +0900, Alexei Starovoitov wrote:
> On 11/8/17 5:28 AM, Josef Bacik wrote:
> > I'm sending this through Dave since it'll conflict with other BPF changes 
> > in his
> > tree, but since it touches tracing as well Dave would like a review from
> > somebody on the tracing side.
> > 
> > v4->v5:
> > - disallow kprobe_override programs from being put in the prog map array so 
> > we
> >   don't tail call into something we didn't check.  This allows us to make 
> > the
> >   normal path still fast without a bunch of percpu operations.
> 
> looks great to me.
> Peter,
> could you please review x86 bits?

I did not in fact get the patch... But after digging it out of LKML I
don't immediately see something broken, but this is not stuff I
typically know well.

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[David Miller]

From: Alexei Starovoitov 
Date: Wed, 8 Nov 2017 06:43:25 +0900

> looks great to me.
> Peter,
> could you please review x86 bits?

I'm still waiting for this.

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[David Miller]

From: Alexei Starovoitov 
Date: Wed, 8 Nov 2017 06:43:25 +0900

> looks great to me.
> Peter,
> could you please review x86 bits?

I'm still waiting for this.

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[Alexei Starovoitov]

On 11/8/17 5:28 AM, Josef Bacik wrote:

I'm sending this through Dave since it'll conflict with other BPF changes in his
tree, but since it touches tracing as well Dave would like a review from
somebody on the tracing side.

v4->v5:
- disallow kprobe_override programs from being put in the prog map array so we
  don't tail call into something we didn't check.  This allows us to make the
  normal path still fast without a bunch of percpu operations.


looks great to me.
Peter,
could you please review x86 bits?


v3->v4:
- fix a build error found by kbuild test bot (I didn't wait long enough
  apparently.)
- Added a warning message as per Daniels suggestion.

v2->v3:
- added a ->kprobe_override flag to bpf_prog.
- added some sanity checks to disallow attaching bpf progs that have
  ->kprobe_override set that aren't for ftrace kprobes.
- added the trace_kprobe_ftrace helper to check if the trace_event_call is a
  ftrace kprobe.
- renamed bpf_kprobe_state to bpf_kprobe_override, fixed it so we only read this
  value in the kprobe path, and thus only write to it if we're overriding or
  clearing the override.

v1->v2:
- moved things around to make sure that bpf_override_return could really only be
  used for an ftrace kprobe.
- killed the special return values from trace_call_bpf.
- renamed pc_modified to bpf_kprobe_state so bpf_override_return could tell if
  it was being called from an ftrace kprobe context.
- reworked the logic in kprobe_perf_func to take advantage of bpf_kprobe_state.
- updated the test as per Alexei's review.

- Original message -

A lot of our error paths are not well tested because we have no good way of
injecting errors generically.  Some subystems (block, memory) have ways to
inject errors, but they are random so it's hard to get reproduceable results.

With BPF we can add determinism to our error injection.  We can use kprobes and
other things to verify we are injecting errors at the exact case we are trying
to test.  This patch gives us the tool to actual do the error injection part.
It is very simple, we just set the return value of the pt_regs we're given to
whatever we provide, and then override the PC with a dummy function that simply
returns.

Right now this only works on x86, but it would be simple enough to expand to
other architectures.  Thanks,

Josef

Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection

[Alexei Starovoitov]

On 11/8/17 5:28 AM, Josef Bacik wrote:

I'm sending this through Dave since it'll conflict with other BPF changes in his
tree, but since it touches tracing as well Dave would like a review from
somebody on the tracing side.

v4->v5:
- disallow kprobe_override programs from being put in the prog map array so we
  don't tail call into something we didn't check.  This allows us to make the
  normal path still fast without a bunch of percpu operations.


looks great to me.
Peter,
could you please review x86 bits?


v3->v4:
- fix a build error found by kbuild test bot (I didn't wait long enough
  apparently.)
- Added a warning message as per Daniels suggestion.

v2->v3:
- added a ->kprobe_override flag to bpf_prog.
- added some sanity checks to disallow attaching bpf progs that have
  ->kprobe_override set that aren't for ftrace kprobes.
- added the trace_kprobe_ftrace helper to check if the trace_event_call is a
  ftrace kprobe.
- renamed bpf_kprobe_state to bpf_kprobe_override, fixed it so we only read this
  value in the kprobe path, and thus only write to it if we're overriding or
  clearing the override.

v1->v2:
- moved things around to make sure that bpf_override_return could really only be
  used for an ftrace kprobe.
- killed the special return values from trace_call_bpf.
- renamed pc_modified to bpf_kprobe_state so bpf_override_return could tell if
  it was being called from an ftrace kprobe context.
- reworked the logic in kprobe_perf_func to take advantage of bpf_kprobe_state.
- updated the test as per Alexei's review.

- Original message -

A lot of our error paths are not well tested because we have no good way of
injecting errors generically.  Some subystems (block, memory) have ways to
inject errors, but they are random so it's hard to get reproduceable results.

With BPF we can add determinism to our error injection.  We can use kprobes and
other things to verify we are injecting errors at the exact case we are trying
to test.  This patch gives us the tool to actual do the error injection part.
It is very simple, we just set the return value of the pt_regs we're given to
whatever we provide, and then override the PC with a dummy function that simply
returns.

Right now this only works on x86, but it would be simple enough to expand to
other architectures.  Thanks,

Josef

[PATCH 0/2][v5] Add the ability to do BPF directed error injection

[Josef Bacik]

I'm sending this through Dave since it'll conflict with other BPF changes in his
tree, but since it touches tracing as well Dave would like a review from
somebody on the tracing side.

v4->v5:
- disallow kprobe_override programs from being put in the prog map array so we
  don't tail call into something we didn't check.  This allows us to make the
  normal path still fast without a bunch of percpu operations.

v3->v4:
- fix a build error found by kbuild test bot (I didn't wait long enough
  apparently.)
- Added a warning message as per Daniels suggestion.

v2->v3:
- added a ->kprobe_override flag to bpf_prog.
- added some sanity checks to disallow attaching bpf progs that have
  ->kprobe_override set that aren't for ftrace kprobes.
- added the trace_kprobe_ftrace helper to check if the trace_event_call is a
  ftrace kprobe.
- renamed bpf_kprobe_state to bpf_kprobe_override, fixed it so we only read this
  value in the kprobe path, and thus only write to it if we're overriding or
  clearing the override.

v1->v2:
- moved things around to make sure that bpf_override_return could really only be
  used for an ftrace kprobe.
- killed the special return values from trace_call_bpf.
- renamed pc_modified to bpf_kprobe_state so bpf_override_return could tell if
  it was being called from an ftrace kprobe context.
- reworked the logic in kprobe_perf_func to take advantage of bpf_kprobe_state.
- updated the test as per Alexei's review.

- Original message -

A lot of our error paths are not well tested because we have no good way of
injecting errors generically.  Some subystems (block, memory) have ways to
inject errors, but they are random so it's hard to get reproduceable results.

With BPF we can add determinism to our error injection.  We can use kprobes and
other things to verify we are injecting errors at the exact case we are trying
to test.  This patch gives us the tool to actual do the error injection part.
It is very simple, we just set the return value of the pt_regs we're given to
whatever we provide, and then override the PC with a dummy function that simply
returns.

Right now this only works on x86, but it would be simple enough to expand to
other architectures.  Thanks,

Josef

[PATCH 0/2][v5] Add the ability to do BPF directed error injection

[Josef Bacik]

I'm sending this through Dave since it'll conflict with other BPF changes in his
tree, but since it touches tracing as well Dave would like a review from
somebody on the tracing side.

v4->v5:
- disallow kprobe_override programs from being put in the prog map array so we
  don't tail call into something we didn't check.  This allows us to make the
  normal path still fast without a bunch of percpu operations.

v3->v4:
- fix a build error found by kbuild test bot (I didn't wait long enough
  apparently.)
- Added a warning message as per Daniels suggestion.

v2->v3:
- added a ->kprobe_override flag to bpf_prog.
- added some sanity checks to disallow attaching bpf progs that have
  ->kprobe_override set that aren't for ftrace kprobes.
- added the trace_kprobe_ftrace helper to check if the trace_event_call is a
  ftrace kprobe.
- renamed bpf_kprobe_state to bpf_kprobe_override, fixed it so we only read this
  value in the kprobe path, and thus only write to it if we're overriding or
  clearing the override.

v1->v2:
- moved things around to make sure that bpf_override_return could really only be
  used for an ftrace kprobe.
- killed the special return values from trace_call_bpf.
- renamed pc_modified to bpf_kprobe_state so bpf_override_return could tell if
  it was being called from an ftrace kprobe context.
- reworked the logic in kprobe_perf_func to take advantage of bpf_kprobe_state.
- updated the test as per Alexei's review.

- Original message -

A lot of our error paths are not well tested because we have no good way of
injecting errors generically.  Some subystems (block, memory) have ways to
inject errors, but they are random so it's hard to get reproduceable results.

With BPF we can add determinism to our error injection.  We can use kprobes and
other things to verify we are injecting errors at the exact case we are trying
to test.  This patch gives us the tool to actual do the error injection part.
It is very simple, we just set the return value of the pt_regs we're given to
whatever we provide, and then override the PC with a dummy function that simply
returns.

Right now this only works on x86, but it would be simple enough to expand to
other architectures.  Thanks,

Josef