Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Wed, Mar 27, 2019 at 01:19:29AM -0500, Scott Wood wrote: > On Wed, 2019-03-27 at 13:10 +0800, Wu Hao wrote: > > On Mon, Mar 25, 2019 at 05:58:36PM -0500, Scott Wood wrote: > > > On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote: > > > > On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote: > > > > > In early partial reconfiguration private feature, it only > > > > > supports 32bit data width when writing data to hardware for > > > > > PR. 512bit data width PR support is an important optimization > > > > > for some specific solutions (e.g. XEON with FPGA integrated), > > > > > it allows driver to use AVX512 instruction to improve the > > > > > performance of partial reconfiguration. e.g. programming one > > > > > 100MB bitstream image via this 512bit data width PR hardware > > > > > only takes ~300ms, but 32bit revision requires ~3s per test > > > > > result. > > > > > > > > > > Please note now this optimization is only done on revision 2 > > > > > of this PR private feature which is only used in integrated > > > > > solution that AVX512 is always supported. > > > > > > > > > > Signed-off-by: Ananda Ravuri > > > > > Signed-off-by: Xu Yilun > > > > > Signed-off-by: Wu Hao > > > > > --- > > > > > drivers/fpga/dfl-fme-main.c | 3 ++ > > > > > drivers/fpga/dfl-fme-mgr.c | 75 > > > > > +- > > > > > -- > > > > > - > > > > > drivers/fpga/dfl-fme-pr.c | 45 --- > > > > > drivers/fpga/dfl-fme.h | 2 ++ > > > > > drivers/fpga/dfl.h | 5 +++ > > > > > 5 files changed, 99 insertions(+), 31 deletions(-) > > > > > > > > > > diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme- > > > > > main.c > > > > > index 086ad24..076d74f 100644 > > > > > --- a/drivers/fpga/dfl-fme-main.c > > > > > +++ b/drivers/fpga/dfl-fme-main.c > > > > > @@ -21,6 +21,8 @@ > > > > > #include "dfl.h" > > > > > #include "dfl-fme.h" > > > > > > > > > > +#define DRV_VERSION "0.8" > > > > > > > > What is this going to be used for? Under what circumstances will the > > > > driver version be bumped? What does it have to do with 512-bit > > > > writes? > > > > This patchset adds more features to this driver, so i would like to add > > a DRV_VERSION there as an initial one. In the future, if some new features > > or extensions for existing features (e.g. new revision of a private > > feature) > > are added we need to bump this version. > > This doesn't seem like a good way of advertising API availability... Besides > being awkward to query, what happens if a distro kernel has backported some > features but not others that came before? What does it advertise? DRV_VERSION here is not used for API availablity. :) > I'd suggest some sort of feature flag mechanism that can be queried via > ioctl (e.g. along the lines of KVM capabilities), if "try the API and fall > back if it fails" is unsatisfactory. > > Plus, if it's about new APIs being exposed, this doesn't seem like the right > patch for it to be in... Actually this patch doesn't introduce new APIs, I am trying to make this transparent to endusers. That means users don't need to know it's a 32bit PR or a faster 512bit one, they still use the same IOCTL interface for PR. the API_VERSION and CHECK_EXTENSION ioctls have been defined, but I think at least we don't need to bump them for this change. How do you think? > > > > Sorry, I missed the comment about revision 2 only being on integrated > > > devices -- but will that always be the case? Seems worthwhile to check > > > for > > > AVX512 support anyway. And there's still the possibility of being built > > > with an old binutils such that CONFIG_AS_AVX512 is not set, or running > > > on a > > > kernel where avx512 was disabled via a boot option. > > > > > > What about future revisions >= 2? Currently the driver will treat them > > > as > > > if they were revision < 2. Is that intended? > > > > Yes, it's intended. Currently we don't have any hardware with revisions > > > 2, > > and support new revisions may need new code. :) e.g. currently revision > > is > > used to tell 32bit vs 512bit PR, but in future revisions, it may have new > > capability registers for this purpose. > > The driver should refuse to bind to unrecognized revisions, if they're not > expected to be compatible. Yes, agree. Thanks Hao
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Wed, 2019-03-27 at 13:10 +0800, Wu Hao wrote: > On Mon, Mar 25, 2019 at 05:58:36PM -0500, Scott Wood wrote: > > On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote: > > > On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote: > > > > In early partial reconfiguration private feature, it only > > > > supports 32bit data width when writing data to hardware for > > > > PR. 512bit data width PR support is an important optimization > > > > for some specific solutions (e.g. XEON with FPGA integrated), > > > > it allows driver to use AVX512 instruction to improve the > > > > performance of partial reconfiguration. e.g. programming one > > > > 100MB bitstream image via this 512bit data width PR hardware > > > > only takes ~300ms, but 32bit revision requires ~3s per test > > > > result. > > > > > > > > Please note now this optimization is only done on revision 2 > > > > of this PR private feature which is only used in integrated > > > > solution that AVX512 is always supported. > > > > > > > > Signed-off-by: Ananda Ravuri > > > > Signed-off-by: Xu Yilun > > > > Signed-off-by: Wu Hao > > > > --- > > > > drivers/fpga/dfl-fme-main.c | 3 ++ > > > > drivers/fpga/dfl-fme-mgr.c | 75 > > > > +- > > > > -- > > > > - > > > > drivers/fpga/dfl-fme-pr.c | 45 --- > > > > drivers/fpga/dfl-fme.h | 2 ++ > > > > drivers/fpga/dfl.h | 5 +++ > > > > 5 files changed, 99 insertions(+), 31 deletions(-) > > > > > > > > diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme- > > > > main.c > > > > index 086ad24..076d74f 100644 > > > > --- a/drivers/fpga/dfl-fme-main.c > > > > +++ b/drivers/fpga/dfl-fme-main.c > > > > @@ -21,6 +21,8 @@ > > > > #include "dfl.h" > > > > #include "dfl-fme.h" > > > > > > > > +#define DRV_VERSION"0.8" > > > > > > What is this going to be used for? Under what circumstances will the > > > driver version be bumped? What does it have to do with 512-bit > > > writes? > > This patchset adds more features to this driver, so i would like to add > a DRV_VERSION there as an initial one. In the future, if some new features > or extensions for existing features (e.g. new revision of a private > feature) > are added we need to bump this version. This doesn't seem like a good way of advertising API availability... Besides being awkward to query, what happens if a distro kernel has backported some features but not others that came before? What does it advertise? I'd suggest some sort of feature flag mechanism that can be queried via ioctl (e.g. along the lines of KVM capabilities), if "try the API and fall back if it fails" is unsatisfactory. Plus, if it's about new APIs being exposed, this doesn't seem like the right patch for it to be in... > > Sorry, I missed the comment about revision 2 only being on integrated > > devices -- but will that always be the case? Seems worthwhile to check > > for > > AVX512 support anyway. And there's still the possibility of being built > > with an old binutils such that CONFIG_AS_AVX512 is not set, or running > > on a > > kernel where avx512 was disabled via a boot option. > > > > What about future revisions >= 2? Currently the driver will treat them > > as > > if they were revision < 2. Is that intended? > > Yes, it's intended. Currently we don't have any hardware with revisions > > 2, > and support new revisions may need new code. :) e.g. currently revision > is > used to tell 32bit vs 512bit PR, but in future revisions, it may have new > capability registers for this purpose. The driver should refuse to bind to unrecognized revisions, if they're not expected to be compatible. -Scott
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Wed, Mar 27, 2019 at 01:10:31AM -0500, Scott Wood wrote:
> On Wed, 2019-03-27 at 12:37 +0800, Wu Hao wrote:
> > On Tue, Mar 26, 2019 at 04:22:34PM -0500, Scott Wood wrote:
> > > On Tue, 2019-03-26 at 14:33 -0500, Alan Tull wrote:
> > > > On Mon, Mar 25, 2019 at 5:58 PM Scott Wood wrote:
> > > > > >
> > > > Hi Scott,
> > > >
> > > > > On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote:
> > > > > > On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> > > > > > > +#else
> > > > > > > +static inline void copy512(void *src, void __iomem *dst)
> > > > > > > +{
> > > > > > > + WARN_ON_ONCE(1);
> > > > > > > +}
> > > > > > > +#endif
> > > > > >
> Likewise, this will be called if a revision 2 device is used on non-
> > > > > > x86
> > > > > > (or on x86 with an old binutils). The driver should fall back to
> > > > > > 32-
> > > > > > bit
> > > > > > in such cases.
> >
> > Unfortunately revision 2 is only for integrated FPGA solution, and it
> > doesn't
> > support any fallback solution (original 32bit data partial reconfiguration
> > is
> > not supported any more), so driver has to WARN in such path.
>
> >From the commit message it seemed like this was just an optimization, not
> something necessary to support revision 2.
>
> If there's no way to program the device without AVX512, then printing an
> error message and returning an error to userspace would be better than
> WARN_ON, since it's not actually a kernel bug.
Fair enough. Will do. Thanks for the suggestion.
Hao
>
> -Scott
>
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Wed, 2019-03-27 at 12:37 +0800, Wu Hao wrote:
> On Tue, Mar 26, 2019 at 04:22:34PM -0500, Scott Wood wrote:
> > On Tue, 2019-03-26 at 14:33 -0500, Alan Tull wrote:
> > > On Mon, Mar 25, 2019 at 5:58 PM Scott Wood wrote:
> > > > >
> > > Hi Scott,
> > >
> > > > On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote:
> > > > > On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> > > > > > +#else
> > > > > > +static inline void copy512(void *src, void __iomem *dst)
> > > > > > +{
> > > > > > + WARN_ON_ONCE(1);
> > > > > > +}
> > > > > > +#endif
> > > > >
Likewise, this will be called if a revision 2 device is used on non-
> > > > > x86
> > > > > (or on x86 with an old binutils). The driver should fall back to
> > > > > 32-
> > > > > bit
> > > > > in such cases.
>
> Unfortunately revision 2 is only for integrated FPGA solution, and it
> doesn't
> support any fallback solution (original 32bit data partial reconfiguration
> is
> not supported any more), so driver has to WARN in such path.
>From the commit message it seemed like this was just an optimization, not
something necessary to support revision 2.
If there's no way to program the device without AVX512, then printing an
error message and returning an error to userspace would be better than
WARN_ON, since it's not actually a kernel bug.
-Scott
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Mon, Mar 25, 2019 at 05:53:50PM -0500, Scott Wood wrote:
> On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> > @@ -200,21 +228,32 @@ static int fme_mgr_write(struct fpga_manager *mgr,
> > pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT,
> > pr_status);
> > }
> >
> > - if (count < 4) {
> > + if (count < priv->pr_datawidth) {
> > dev_err(dev, "Invalid PR bitstream size\n");
> > return -EINVAL;
>
> Shouldn't this have become a WARN_ON in patch 2 given that the kernel
> already pads the buffer?
Thanks a lot for the review and comments.
I agree. it's better to use WARN_ON this place.
>
> > }
> >
> > - pr_data = 0;
> > - pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > - *(((u32 *)buf) + i));
> > - writeq(pr_data, fme_pr + FME_PR_DATA);
> > - count -= 4;
> > + switch (priv->pr_datawidth) {
> > + case 4:
> > + pr_data = 0;
> > + pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > + *((u32 *)buf));
>
> I know it's not new, but why not just "pr_data = FIELD..."? Const should
> also be preserved in the cast, and you can drop one set of parentheses.
Yes, agree, will fix this.
>
> > + writeq(pr_data, fme_pr + FME_PR_DATA);
> > + break;
> > + case 64:
> > + copy512((void *)buf, fme_pr + FME_PR_512_DATA);
> > + break;
>
> Unnecessary cast.
Will fix this.
>
> > + default:
> > + ret = -EFAULT;
> > + goto done;
>
> How is it EFAULT? Any other value for pr_datawidth should be WARN_ON
> since it's set by kernel code.
Agree, will fix this in the next version.
>
> > @@ -159,13 +161,10 @@ static int fme_pr(struct platform_device *pdev,
> > unsigned long arg)
> > fpga_bridges_put(>bridge_list);
> >
> > put_device(>dev);
> > -unlock_exit:
> > - mutex_unlock(>lock);
> > free_exit:
> > vfree(buf);
> > - if (copy_to_user((void __user *)arg, _pr, minsz))
> > - return -EFAULT;
> > -
>
> Why is the copy_to_user being removed?
This code is not needed at all but added by mistake i think.
Sorry, i should move these code into a separated patch with proper comments
to avoid confusion.
Thanks
Hao
>
> -Scott
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Mon, Mar 25, 2019 at 05:58:36PM -0500, Scott Wood wrote:
> On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote:
> > On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> > > In early partial reconfiguration private feature, it only
> > > supports 32bit data width when writing data to hardware for
> > > PR. 512bit data width PR support is an important optimization
> > > for some specific solutions (e.g. XEON with FPGA integrated),
> > > it allows driver to use AVX512 instruction to improve the
> > > performance of partial reconfiguration. e.g. programming one
> > > 100MB bitstream image via this 512bit data width PR hardware
> > > only takes ~300ms, but 32bit revision requires ~3s per test
> > > result.
> > >
> > > Please note now this optimization is only done on revision 2
> > > of this PR private feature which is only used in integrated
> > > solution that AVX512 is always supported.
> > >
> > > Signed-off-by: Ananda Ravuri
> > > Signed-off-by: Xu Yilun
> > > Signed-off-by: Wu Hao
> > > ---
> > > drivers/fpga/dfl-fme-main.c | 3 ++
> > > drivers/fpga/dfl-fme-mgr.c | 75 +-
> > > --
> > > -
> > > drivers/fpga/dfl-fme-pr.c | 45 ---
> > > drivers/fpga/dfl-fme.h | 2 ++
> > > drivers/fpga/dfl.h | 5 +++
> > > 5 files changed, 99 insertions(+), 31 deletions(-)
> > >
> > > diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
> > > index 086ad24..076d74f 100644
> > > --- a/drivers/fpga/dfl-fme-main.c
> > > +++ b/drivers/fpga/dfl-fme-main.c
> > > @@ -21,6 +21,8 @@
> > > #include "dfl.h"
> > > #include "dfl-fme.h"
> > >
> > > +#define DRV_VERSION "0.8"
> >
> > What is this going to be used for? Under what circumstances will the
> > driver version be bumped? What does it have to do with 512-bit writes?
This patchset adds more features to this driver, so i would like to add
a DRV_VERSION there as an initial one. In the future, if some new features
or extensions for existing features (e.g. new revision of a private feature)
are added we need to bump this version.
> >
> > > +#if defined(CONFIG_X86) && defined(CONFIG_AS_AVX512)
> > > +
> > > +#include
> > > +
> > > +static inline void copy512(void *src, void __iomem *dst)
> > > +{
> > > + kernel_fpu_begin();
> > > +
> > > + asm volatile("vmovdqu64 (%0), %%zmm0;"
> > > + "vmovntdq %%zmm0, (%1);"
> > > + :
> > > + : "r"(src), "r"(dst));
> > > +
> > > + kernel_fpu_end();
> > > +}
> >
> > Shouldn't there be some sort of check that AVX512 is actually supported
> > on the running system?
> >
> > Also, src should be const, and the asm statement should have a memory
> > clobber.
> >
> > > +#else
> > > +static inline void copy512(void *src, void __iomem *dst)
> > > +{
> > > + WARN_ON_ONCE(1);
> > > +}
> > > +#endif
> >
> > Likewise, this will be called if a revision 2 device is used on non-x86
> > (or on x86 with an old binutils). The driver should fall back to 32-bit
> > in such cases.
>
> Sorry, I missed the comment about revision 2 only being on integrated
> devices -- but will that always be the case? Seems worthwhile to check for
> AVX512 support anyway. And there's still the possibility of being built
> with an old binutils such that CONFIG_AS_AVX512 is not set, or running on a
> kernel where avx512 was disabled via a boot option.
>
> What about future revisions >= 2? Currently the driver will treat them as
> if they were revision < 2. Is that intended?
Yes, it's intended. Currently we don't have any hardware with revisions > 2,
and support new revisions may need new code. :) e.g. currently revision is
used to tell 32bit vs 512bit PR, but in future revisions, it may have new
capability registers for this purpose.
Thanks
Hao
>
> -Scott
>
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Tue, Mar 26, 2019 at 04:22:34PM -0500, Scott Wood wrote:
> On Tue, 2019-03-26 at 14:33 -0500, Alan Tull wrote:
> > On Mon, Mar 25, 2019 at 5:58 PM Scott Wood wrote:
> >
> > Hi Scott,
> >
> > > On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote:
> > > > On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> > > > > +#if defined(CONFIG_X86) && defined(CONFIG_AS_AVX512)
> > > > > +
> > > > > +#include
> > > > > +
> > > > > +static inline void copy512(void *src, void __iomem *dst)
> > > > > +{
> > > > > + kernel_fpu_begin();
> > > > > +
> > > > > + asm volatile("vmovdqu64 (%0), %%zmm0;"
> > > > > +"vmovntdq %%zmm0, (%1);"
> > > > > +:
> > > > > +: "r"(src), "r"(dst));
> > > > > +
> > > > > + kernel_fpu_end();
> > > > > +}
> > > >
> > > > Shouldn't there be some sort of check that AVX512 is actually
> > > > supported
> > > > on the running system?
> > > >
> > > > Also, src should be const, and the asm statement should have a memory
> > > > clobber.
Yes, I will fix this in the next version.
> > > >
> > > > > +#else
> > > > > +static inline void copy512(void *src, void __iomem *dst)
> > > > > +{
> > > > > + WARN_ON_ONCE(1);
> > > > > +}
> > > > > +#endif
> > > >
> > > > Likewise, this will be called if a revision 2 device is used on non-
> > > > x86
> > > > (or on x86 with an old binutils). The driver should fall back to 32-
> > > > bit
> > > > in such cases.
Unfortunately revision 2 is only for integrated FPGA solution, and it doesn't
support any fallback solution (original 32bit data partial reconfiguration is
not supported any more), so driver has to WARN in such path.
> > >
> > > Sorry, I missed the comment about revision 2 only being on integrated
> > > devices -- but will that always be the case? Seems worthwhile to check
> > > for
> > > AVX512 support anyway. And there's still the possibility of being built
> > > with an old binutils such that CONFIG_AS_AVX512 is not set, or running
> > > on a
> > > kernel where avx512 was disabled via a boot option.
> >
> > The code checks for CONFIG_AS_AVX512 above.
>
> That just indicates that binutils supports it. Plus, the code does not
> check for CONFIG_AS_AVX512 when deciding whether to set pr_datawidth to 64
> (and thus call copy512), so you'll get a WARN_ON rather than falling back to
> 32-bit.
>
> > What boot option are you referring to?
>
> clearcpuid=304
Just tried it, my system was down after running above AVX512 with this option.
I agree that it needs to add some check code to make sure it's safe to run
such instructions. I will add some cpu_feature_enabled() check in the next
version.
Thanks a lot for the review and comments.
Hao
>
> -Scott
>
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Tue, 2019-03-26 at 14:33 -0500, Alan Tull wrote:
> On Mon, Mar 25, 2019 at 5:58 PM Scott Wood wrote:
>
> Hi Scott,
>
> > On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote:
> > > On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> > > > +#if defined(CONFIG_X86) && defined(CONFIG_AS_AVX512)
> > > > +
> > > > +#include
> > > > +
> > > > +static inline void copy512(void *src, void __iomem *dst)
> > > > +{
> > > > + kernel_fpu_begin();
> > > > +
> > > > + asm volatile("vmovdqu64 (%0), %%zmm0;"
> > > > +"vmovntdq %%zmm0, (%1);"
> > > > +:
> > > > +: "r"(src), "r"(dst));
> > > > +
> > > > + kernel_fpu_end();
> > > > +}
> > >
> > > Shouldn't there be some sort of check that AVX512 is actually
> > > supported
> > > on the running system?
> > >
> > > Also, src should be const, and the asm statement should have a memory
> > > clobber.
> > >
> > > > +#else
> > > > +static inline void copy512(void *src, void __iomem *dst)
> > > > +{
> > > > + WARN_ON_ONCE(1);
> > > > +}
> > > > +#endif
> > >
> > > Likewise, this will be called if a revision 2 device is used on non-
> > > x86
> > > (or on x86 with an old binutils). The driver should fall back to 32-
> > > bit
> > > in such cases.
> >
> > Sorry, I missed the comment about revision 2 only being on integrated
> > devices -- but will that always be the case? Seems worthwhile to check
> > for
> > AVX512 support anyway. And there's still the possibility of being built
> > with an old binutils such that CONFIG_AS_AVX512 is not set, or running
> > on a
> > kernel where avx512 was disabled via a boot option.
>
> The code checks for CONFIG_AS_AVX512 above.
That just indicates that binutils supports it. Plus, the code does not
check for CONFIG_AS_AVX512 when deciding whether to set pr_datawidth to 64
(and thus call copy512), so you'll get a WARN_ON rather than falling back to
32-bit.
> What boot option are you referring to?
clearcpuid=304
-Scott
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Mon, Mar 25, 2019 at 5:58 PM Scott Wood wrote:
Hi Scott,
>
> On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote:
> > On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> > > In early partial reconfiguration private feature, it only
> > > supports 32bit data width when writing data to hardware for
> > > PR. 512bit data width PR support is an important optimization
> > > for some specific solutions (e.g. XEON with FPGA integrated),
> > > it allows driver to use AVX512 instruction to improve the
> > > performance of partial reconfiguration. e.g. programming one
> > > 100MB bitstream image via this 512bit data width PR hardware
> > > only takes ~300ms, but 32bit revision requires ~3s per test
> > > result.
> > >
> > > Please note now this optimization is only done on revision 2
> > > of this PR private feature which is only used in integrated
> > > solution that AVX512 is always supported.
> > >
> > > Signed-off-by: Ananda Ravuri
> > > Signed-off-by: Xu Yilun
> > > Signed-off-by: Wu Hao
> > > ---
> > > drivers/fpga/dfl-fme-main.c | 3 ++
> > > drivers/fpga/dfl-fme-mgr.c | 75 +-
> > > --
> > > -
> > > drivers/fpga/dfl-fme-pr.c | 45 ---
> > > drivers/fpga/dfl-fme.h | 2 ++
> > > drivers/fpga/dfl.h | 5 +++
> > > 5 files changed, 99 insertions(+), 31 deletions(-)
> > >
> > > diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
> > > index 086ad24..076d74f 100644
> > > --- a/drivers/fpga/dfl-fme-main.c
> > > +++ b/drivers/fpga/dfl-fme-main.c
> > > @@ -21,6 +21,8 @@
> > > #include "dfl.h"
> > > #include "dfl-fme.h"
> > >
> > > +#define DRV_VERSION"0.8"
> >
> > What is this going to be used for? Under what circumstances will the
> > driver version be bumped? What does it have to do with 512-bit writes?
> >
> > > +#if defined(CONFIG_X86) && defined(CONFIG_AS_AVX512)
> > > +
> > > +#include
> > > +
> > > +static inline void copy512(void *src, void __iomem *dst)
> > > +{
> > > + kernel_fpu_begin();
> > > +
> > > + asm volatile("vmovdqu64 (%0), %%zmm0;"
> > > +"vmovntdq %%zmm0, (%1);"
> > > +:
> > > +: "r"(src), "r"(dst));
> > > +
> > > + kernel_fpu_end();
> > > +}
> >
> > Shouldn't there be some sort of check that AVX512 is actually supported
> > on the running system?
> >
> > Also, src should be const, and the asm statement should have a memory
> > clobber.
> >
> > > +#else
> > > +static inline void copy512(void *src, void __iomem *dst)
> > > +{
> > > + WARN_ON_ONCE(1);
> > > +}
> > > +#endif
> >
> > Likewise, this will be called if a revision 2 device is used on non-x86
> > (or on x86 with an old binutils). The driver should fall back to 32-bit
> > in such cases.
>
> Sorry, I missed the comment about revision 2 only being on integrated
> devices -- but will that always be the case? Seems worthwhile to check for
> AVX512 support anyway. And there's still the possibility of being built
> with an old binutils such that CONFIG_AS_AVX512 is not set, or running on a
> kernel where avx512 was disabled via a boot option.
The code checks for CONFIG_AS_AVX512 above.
What boot option are you referring to?
Alan
>
> What about future revisions >= 2? Currently the driver will treat them as
> if they were revision < 2. Is that intended?
>
> -Scott
>
>
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Mon, 2019-03-25 at 17:53 -0500, Scott Wood wrote:
> On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> > In early partial reconfiguration private feature, it only
> > supports 32bit data width when writing data to hardware for
> > PR. 512bit data width PR support is an important optimization
> > for some specific solutions (e.g. XEON with FPGA integrated),
> > it allows driver to use AVX512 instruction to improve the
> > performance of partial reconfiguration. e.g. programming one
> > 100MB bitstream image via this 512bit data width PR hardware
> > only takes ~300ms, but 32bit revision requires ~3s per test
> > result.
> >
> > Please note now this optimization is only done on revision 2
> > of this PR private feature which is only used in integrated
> > solution that AVX512 is always supported.
> >
> > Signed-off-by: Ananda Ravuri
> > Signed-off-by: Xu Yilun
> > Signed-off-by: Wu Hao
> > ---
> > drivers/fpga/dfl-fme-main.c | 3 ++
> > drivers/fpga/dfl-fme-mgr.c | 75 +-
> > --
> > -
> > drivers/fpga/dfl-fme-pr.c | 45 ---
> > drivers/fpga/dfl-fme.h | 2 ++
> > drivers/fpga/dfl.h | 5 +++
> > 5 files changed, 99 insertions(+), 31 deletions(-)
> >
> > diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
> > index 086ad24..076d74f 100644
> > --- a/drivers/fpga/dfl-fme-main.c
> > +++ b/drivers/fpga/dfl-fme-main.c
> > @@ -21,6 +21,8 @@
> > #include "dfl.h"
> > #include "dfl-fme.h"
> >
> > +#define DRV_VERSION"0.8"
>
> What is this going to be used for? Under what circumstances will the
> driver version be bumped? What does it have to do with 512-bit writes?
>
> > +#if defined(CONFIG_X86) && defined(CONFIG_AS_AVX512)
> > +
> > +#include
> > +
> > +static inline void copy512(void *src, void __iomem *dst)
> > +{
> > + kernel_fpu_begin();
> > +
> > + asm volatile("vmovdqu64 (%0), %%zmm0;"
> > +"vmovntdq %%zmm0, (%1);"
> > +:
> > +: "r"(src), "r"(dst));
> > +
> > + kernel_fpu_end();
> > +}
>
> Shouldn't there be some sort of check that AVX512 is actually supported
> on the running system?
>
> Also, src should be const, and the asm statement should have a memory
> clobber.
>
> > +#else
> > +static inline void copy512(void *src, void __iomem *dst)
> > +{
> > + WARN_ON_ONCE(1);
> > +}
> > +#endif
>
> Likewise, this will be called if a revision 2 device is used on non-x86
> (or on x86 with an old binutils). The driver should fall back to 32-bit
> in such cases.
Sorry, I missed the comment about revision 2 only being on integrated
devices -- but will that always be the case? Seems worthwhile to check for
AVX512 support anyway. And there's still the possibility of being built
with an old binutils such that CONFIG_AS_AVX512 is not set, or running on a
kernel where avx512 was disabled via a boot option.
What about future revisions >= 2? Currently the driver will treat them as
if they were revision < 2. Is that intended?
-Scott
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Mon, 2019-03-25 at 11:07 +0800, Wu Hao wrote:
> In early partial reconfiguration private feature, it only
> supports 32bit data width when writing data to hardware for
> PR. 512bit data width PR support is an important optimization
> for some specific solutions (e.g. XEON with FPGA integrated),
> it allows driver to use AVX512 instruction to improve the
> performance of partial reconfiguration. e.g. programming one
> 100MB bitstream image via this 512bit data width PR hardware
> only takes ~300ms, but 32bit revision requires ~3s per test
> result.
>
> Please note now this optimization is only done on revision 2
> of this PR private feature which is only used in integrated
> solution that AVX512 is always supported.
>
> Signed-off-by: Ananda Ravuri
> Signed-off-by: Xu Yilun
> Signed-off-by: Wu Hao
> ---
> drivers/fpga/dfl-fme-main.c | 3 ++
> drivers/fpga/dfl-fme-mgr.c | 75 +---
> -
> drivers/fpga/dfl-fme-pr.c | 45 ---
> drivers/fpga/dfl-fme.h | 2 ++
> drivers/fpga/dfl.h | 5 +++
> 5 files changed, 99 insertions(+), 31 deletions(-)
>
> diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
> index 086ad24..076d74f 100644
> --- a/drivers/fpga/dfl-fme-main.c
> +++ b/drivers/fpga/dfl-fme-main.c
> @@ -21,6 +21,8 @@
> #include "dfl.h"
> #include "dfl-fme.h"
>
> +#define DRV_VERSION "0.8"
What is this going to be used for? Under what circumstances will the
driver version be bumped? What does it have to do with 512-bit writes?
> +#if defined(CONFIG_X86) && defined(CONFIG_AS_AVX512)
> +
> +#include
> +
> +static inline void copy512(void *src, void __iomem *dst)
> +{
> + kernel_fpu_begin();
> +
> + asm volatile("vmovdqu64 (%0), %%zmm0;"
> + "vmovntdq %%zmm0, (%1);"
> + :
> + : "r"(src), "r"(dst));
> +
> + kernel_fpu_end();
> +}
Shouldn't there be some sort of check that AVX512 is actually supported
on the running system?
Also, src should be const, and the asm statement should have a memory
clobber.
> +#else
> +static inline void copy512(void *src, void __iomem *dst)
> +{
> + WARN_ON_ONCE(1);
> +}
> +#endif
Likewise, this will be called if a revision 2 device is used on non-x86
(or on x86 with an old binutils). The driver should fall back to 32-bit
in such cases.
> @@ -200,21 +228,32 @@ static int fme_mgr_write(struct fpga_manager *mgr,
> pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT,
> pr_status);
> }
>
> - if (count < 4) {
> + if (count < priv->pr_datawidth) {
> dev_err(dev, "Invalid PR bitstream size\n");
> return -EINVAL;
Shouldn't this have become a WARN_ON in patch 2 given that the kernel
already pads the buffer?
> }
>
> - pr_data = 0;
> - pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> - *(((u32 *)buf) + i));
> - writeq(pr_data, fme_pr + FME_PR_DATA);
> - count -= 4;
> + switch (priv->pr_datawidth) {
> + case 4:
> + pr_data = 0;
> + pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> + *((u32 *)buf));
I know it's not new, but why not just "pr_data = FIELD..."? Const should
also be preserved in the cast, and you can drop one set of parentheses.
> + writeq(pr_data, fme_pr + FME_PR_DATA);
> + break;
> + case 64:
> + copy512((void *)buf, fme_pr + FME_PR_512_DATA);
> + break;
Unnecessary cast.
> + default:
> + ret = -EFAULT;
> + goto done;
How is it EFAULT? Any other value for pr_datawidth should be WARN_ON
since it's set by kernel code.
> @@ -159,13 +161,10 @@ static int fme_pr(struct platform_device *pdev,
> unsigned long arg)
> fpga_bridges_put(>bridge_list);
>
> put_device(>dev);
> -unlock_exit:
> - mutex_unlock(>lock);
> free_exit:
> vfree(buf);
> - if (copy_to_user((void __user *)arg, _pr, minsz))
> - return -EFAULT;
> -
Why is the copy_to_user being removed?
-Scott
Re: [PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
On Sun, Mar 24, 2019 at 10:23 PM Wu Hao wrote: Hi Hao, This looks fine. > > In early partial reconfiguration private feature, it only > supports 32bit data width when writing data to hardware for > PR. 512bit data width PR support is an important optimization > for some specific solutions (e.g. XEON with FPGA integrated), > it allows driver to use AVX512 instruction to improve the > performance of partial reconfiguration. e.g. programming one > 100MB bitstream image via this 512bit data width PR hardware > only takes ~300ms, but 32bit revision requires ~3s per test > result. > > Please note now this optimization is only done on revision 2 > of this PR private feature which is only used in integrated > solution that AVX512 is always supported. > > Signed-off-by: Ananda Ravuri > Signed-off-by: Xu Yilun > Signed-off-by: Wu Hao Acked-by: Alan Tull
[PATCH 03/17] fpga: dfl: fme: support 512bit data width PR
In early partial reconfiguration private feature, it only
supports 32bit data width when writing data to hardware for
PR. 512bit data width PR support is an important optimization
for some specific solutions (e.g. XEON with FPGA integrated),
it allows driver to use AVX512 instruction to improve the
performance of partial reconfiguration. e.g. programming one
100MB bitstream image via this 512bit data width PR hardware
only takes ~300ms, but 32bit revision requires ~3s per test
result.
Please note now this optimization is only done on revision 2
of this PR private feature which is only used in integrated
solution that AVX512 is always supported.
Signed-off-by: Ananda Ravuri
Signed-off-by: Xu Yilun
Signed-off-by: Wu Hao
---
drivers/fpga/dfl-fme-main.c | 3 ++
drivers/fpga/dfl-fme-mgr.c | 75 +
drivers/fpga/dfl-fme-pr.c | 45 ---
drivers/fpga/dfl-fme.h | 2 ++
drivers/fpga/dfl.h | 5 +++
5 files changed, 99 insertions(+), 31 deletions(-)
diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index 086ad24..076d74f 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c
@@ -21,6 +21,8 @@
#include "dfl.h"
#include "dfl-fme.h"
+#define DRV_VERSION"0.8"
+
static ssize_t ports_num_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
@@ -277,3 +279,4 @@ MODULE_DESCRIPTION("FPGA Management Engine driver");
MODULE_AUTHOR("Intel Corporation");
MODULE_LICENSE("GPL v2");
MODULE_ALIAS("platform:dfl-fme");
+MODULE_VERSION(DRV_VERSION);
diff --git a/drivers/fpga/dfl-fme-mgr.c b/drivers/fpga/dfl-fme-mgr.c
index b3f7eee..027d457 100644
--- a/drivers/fpga/dfl-fme-mgr.c
+++ b/drivers/fpga/dfl-fme-mgr.c
@@ -22,14 +22,18 @@
#include
#include
+#include "dfl.h"
#include "dfl-fme-pr.h"
+#define DRV_VERSION"0.8"
+
/* FME Partial Reconfiguration Sub Feature Register Set */
#define FME_PR_DFH 0x0
#define FME_PR_CTRL0x8
#define FME_PR_STS 0x10
#define FME_PR_DATA0x18
#define FME_PR_ERR 0x20
+#define FME_PR_512_DATA0x40 /* Data Register for 512bit
datawidth PR */
#define FME_PR_INTFC_ID_L 0xA8
#define FME_PR_INTFC_ID_H 0xB0
@@ -67,8 +71,31 @@
#define PR_WAIT_TIMEOUT 800
#define PR_HOST_STATUS_IDLE0
+#if defined(CONFIG_X86) && defined(CONFIG_AS_AVX512)
+
+#include
+
+static inline void copy512(void *src, void __iomem *dst)
+{
+ kernel_fpu_begin();
+
+ asm volatile("vmovdqu64 (%0), %%zmm0;"
+"vmovntdq %%zmm0, (%1);"
+:
+: "r"(src), "r"(dst));
+
+ kernel_fpu_end();
+}
+#else
+static inline void copy512(void *src, void __iomem *dst)
+{
+ WARN_ON_ONCE(1);
+}
+#endif
+
struct fme_mgr_priv {
void __iomem *ioaddr;
+ unsigned int pr_datawidth;
u64 pr_error;
};
@@ -169,7 +196,7 @@ static int fme_mgr_write(struct fpga_manager *mgr,
struct fme_mgr_priv *priv = mgr->priv;
void __iomem *fme_pr = priv->ioaddr;
u64 pr_ctrl, pr_status, pr_data;
- int delay = 0, pr_credit, i = 0;
+ int ret = 0, delay = 0, pr_credit;
dev_dbg(dev, "start request\n");
@@ -181,9 +208,9 @@ static int fme_mgr_write(struct fpga_manager *mgr,
/*
* driver can push data to PR hardware using PR_DATA register once HW
-* has enough pr_credit (> 1), pr_credit reduces one for every 32bit
-* pr data write to PR_DATA register. If pr_credit <= 1, driver needs
-* to wait for enough pr_credit from hardware by polling.
+* has enough pr_credit (> 1), pr_credit reduces one for every pr data
+* width write to PR_DATA register. If pr_credit <= 1, driver needs to
+* wait for enough pr_credit from hardware by polling.
*/
pr_status = readq(fme_pr + FME_PR_STS);
pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
@@ -192,7 +219,8 @@ static int fme_mgr_write(struct fpga_manager *mgr,
while (pr_credit <= 1) {
if (delay++ > PR_WAIT_TIMEOUT) {
dev_err(dev, "PR_CREDIT timeout\n");
- return -ETIMEDOUT;
+ ret = -ETIMEDOUT;
+ goto done;
}
udelay(1);
@@ -200,21 +228,32 @@ static int fme_mgr_write(struct fpga_manager *mgr,
pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
}
- if (count < 4) {
+ if (count < priv->pr_datawidth) {
dev_err(dev, "Invalid PR bitstream size\n");
return -EINVAL;
}
- pr_data = 0;
- pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
-
