Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Keith Busch]

On Tue, Aug 08, 2017 at 12:33:40PM +0530, Sreekanth Reddy wrote:
> On Tue, Aug 8, 2017 at 9:34 AM, Keith Busch  wrote:
> >
> > It looks like they can make existing nvme tooling work with little
> > effort if they have the driver implement NVME_IOCTL_ADMIN_COMMAND, and
> 
> Precisely, I was thinking on the same line and you clarified. I will
> spend sometime on this and get back to you.

Sounds good. Just note that while the majority of NVMe management should
be reachable with just that IOCTL, some tooling features may not work
correctly since they look for /dev/nvmeX, and I assume this driver will
create /dev/sdX instead.

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Keith Busch]

On Tue, Aug 08, 2017 at 12:33:40PM +0530, Sreekanth Reddy wrote:
> On Tue, Aug 8, 2017 at 9:34 AM, Keith Busch  wrote:
> >
> > It looks like they can make existing nvme tooling work with little
> > effort if they have the driver implement NVME_IOCTL_ADMIN_COMMAND, and
> 
> Precisely, I was thinking on the same line and you clarified. I will
> spend sometime on this and get back to you.

Sounds good. Just note that while the majority of NVMe management should
be reachable with just that IOCTL, some tooling features may not work
correctly since they look for /dev/nvmeX, and I assume this driver will
create /dev/sdX instead.

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Sreekanth Reddy]

On Tue, Aug 8, 2017 at 9:34 AM, Keith Busch  wrote:
> On Mon, Aug 07, 2017 at 08:45:25AM -0700, James Bottomley wrote:
>> On Mon, 2017-08-07 at 20:01 +0530, Kashyap Desai wrote:
>> >
>> > We have to attempt this use case and see how it behaves. I have not
>> > tried this, so not sure if things are really bad or just some tuning
>> > may be helpful. I will revert back to you on this.
>> >
>> > I understood request as -  We need some udev rules to be working well
>> > for *same* NVME drives if it is behind  or native .
>> > Example - If user has OS installed on NVME drive which is behind
>> >  driver as SCSI disk should be able to boot if he/she hooked
>> > same NVME drive which is detected by native  driver (and vice
>> > versa.)
>>
>> It's not just the udev rules, it's the tools as well; possibly things
>> like that nvme-cli toolkit Intel is doing.
>
> It looks like they can make existing nvme tooling work with little
> effort if they have the driver implement NVME_IOCTL_ADMIN_COMMAND, and

Precisely, I was thinking on the same line and you clarified. I will
spend sometime on this and get back to you.

> then have their driver build the MPI NVMe Encapsulated Request from that.

Hi Everyone,

Thanks for the discussion and feedback.  We have noted this (i.e. will
Encapsulate NVME_IOCTL_ADMIN_CMD received from nvme cli in to MPI NVMe
Encapsulated Request message and will issue this request to firmware)
as our next to-do item and I will post possible options (this may need
some discussion with our FW group, so need time to get back with all
the details).

We will be posting next version of patch set i.e. v3, which will a
accommodate below changes suggested by Hannes and Martin over v2 patch
set.

1. In the MPI header files, we have reformatted headers to have type
and variable on one line as suggested by Hannes,
2. As suggested by Martin, started using blk_queue_virt_boundary() API
for NVMe drives and simplified the PRP formation.
3. Removed 'TODO' comments

Thanks,
Sreekanth

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Sreekanth Reddy]

On Tue, Aug 8, 2017 at 9:34 AM, Keith Busch  wrote:
> On Mon, Aug 07, 2017 at 08:45:25AM -0700, James Bottomley wrote:
>> On Mon, 2017-08-07 at 20:01 +0530, Kashyap Desai wrote:
>> >
>> > We have to attempt this use case and see how it behaves. I have not
>> > tried this, so not sure if things are really bad or just some tuning
>> > may be helpful. I will revert back to you on this.
>> >
>> > I understood request as -  We need some udev rules to be working well
>> > for *same* NVME drives if it is behind  or native .
>> > Example - If user has OS installed on NVME drive which is behind
>> >  driver as SCSI disk should be able to boot if he/she hooked
>> > same NVME drive which is detected by native  driver (and vice
>> > versa.)
>>
>> It's not just the udev rules, it's the tools as well; possibly things
>> like that nvme-cli toolkit Intel is doing.
>
> It looks like they can make existing nvme tooling work with little
> effort if they have the driver implement NVME_IOCTL_ADMIN_COMMAND, and

Precisely, I was thinking on the same line and you clarified. I will
spend sometime on this and get back to you.

> then have their driver build the MPI NVMe Encapsulated Request from that.

Hi Everyone,

Thanks for the discussion and feedback.  We have noted this (i.e. will
Encapsulate NVME_IOCTL_ADMIN_CMD received from nvme cli in to MPI NVMe
Encapsulated Request message and will issue this request to firmware)
as our next to-do item and I will post possible options (this may need
some discussion with our FW group, so need time to get back with all
the details).

We will be posting next version of patch set i.e. v3, which will a
accommodate below changes suggested by Hannes and Martin over v2 patch
set.

1. In the MPI header files, we have reformatted headers to have type
and variable on one line as suggested by Hannes,
2. As suggested by Martin, started using blk_queue_virt_boundary() API
for NVMe drives and simplified the PRP formation.
3. Removed 'TODO' comments

Thanks,
Sreekanth

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Keith Busch]

On Mon, Aug 07, 2017 at 08:45:25AM -0700, James Bottomley wrote:
> On Mon, 2017-08-07 at 20:01 +0530, Kashyap Desai wrote:
> > 
> > We have to attempt this use case and see how it behaves. I have not
> > tried this, so not sure if things are really bad or just some tuning
> > may be helpful. I will revert back to you on this.
> > 
> > I understood request as -  We need some udev rules to be working well
> > for *same* NVME drives if it is behind  or native .
> > Example - If user has OS installed on NVME drive which is behind
> >  driver as SCSI disk should be able to boot if he/she hooked
> > same NVME drive which is detected by native  driver (and vice
> > versa.)
> 
> It's not just the udev rules, it's the tools as well; possibly things
> like that nvme-cli toolkit Intel is doing.

It looks like they can make existing nvme tooling work with little
effort if they have the driver implement NVME_IOCTL_ADMIN_COMMAND, and
then have their driver build the MPI NVMe Encapsulated Request from that.

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Keith Busch]

On Mon, Aug 07, 2017 at 08:45:25AM -0700, James Bottomley wrote:
> On Mon, 2017-08-07 at 20:01 +0530, Kashyap Desai wrote:
> > 
> > We have to attempt this use case and see how it behaves. I have not
> > tried this, so not sure if things are really bad or just some tuning
> > may be helpful. I will revert back to you on this.
> > 
> > I understood request as -  We need some udev rules to be working well
> > for *same* NVME drives if it is behind  or native .
> > Example - If user has OS installed on NVME drive which is behind
> >  driver as SCSI disk should be able to boot if he/she hooked
> > same NVME drive which is detected by native  driver (and vice
> > versa.)
> 
> It's not just the udev rules, it's the tools as well; possibly things
> like that nvme-cli toolkit Intel is doing.

It looks like they can make existing nvme tooling work with little
effort if they have the driver implement NVME_IOCTL_ADMIN_COMMAND, and
then have their driver build the MPI NVMe Encapsulated Request from that.

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[James Bottomley]

On Mon, 2017-08-07 at 20:01 +0530, Kashyap Desai wrote:
> > 
> > -Original Message-
> > From: James Bottomley [mailto:james.bottom...@hansenpartnership.com
> > ]
> > Sent: Monday, August 07, 2017 7:48 PM
> > To: Kashyap Desai; Christoph Hellwig; Hannes Reinecke
> > Cc: Suganath Prabu Subramani; martin.peter...@oracle.com; linux-
> > s...@vger.kernel.org; Sathya Prakash Veerichetty; linux-
> > ker...@vger.kernel.org; Chaitra Basappa; Sreekanth Reddy; linux-
> > n...@lists.infradead.org
> > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > 
> > On Mon, 2017-08-07 at 19:26 +0530, Kashyap Desai wrote:
> > > 
> > > > 
> > > > 
> > > > -Original Message-
> > > > From: James Bottomley [mailto:James.Bottomley@HansenPartnership
> > > > .com
> > > > ]
> > > > Sent: Saturday, August 05, 2017 8:12 PM
> > > > To: Christoph Hellwig; Hannes Reinecke
> > > > Cc: Suganath Prabu S; martin.peter...@oracle.com; linux-
> > > > s...@vger.kernel.org; sathya.prak...@broadcom.com;
> > > > kashyap.de...@broadcom.com; linux-kernel@vger.kernel.org;
> > > > chaitra.basa...@broadcom.com; sreekanth.re...@broadcom.com;
> > > > linux-
> > > > n...@lists.infradead.org
> > > > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > > > 
> > > > On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > > > > 
> > > > > 
> > > > > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke
> > > > > wrote:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > I'm not happy with this approach.
> > > > > > NVMe devices should _not_ appear as SCSI devices; this will
> > > > > > just
> > > > > > confuse matters _and_ will be incompatible with 'normal'
> > > > > > NVMe
> > > > > > devices.
> > > > > > 
> > > > > > Rather I would like to see the driver to hook into the
> > > > > > existing
> > > > > > NVMe framework (which essentially means to treat the
> > > > > > mpt3sas as
> > > > > > a weird NVMe-over-Fabrics HBA), and expose the NVMe devices
> > > > > > like
> > > > > > any other NVMe HBA.
> > > > > 
> > > > > That doesn't make any sense.  The devices behind the mpt
> > > > > adapter
> > > > > don't look like NVMe devices at all for the hosts - there are
> > > > > no
> > > > > NVMe commands or queues involved at all, they hide behind the
> > > > > same
> > > > > somewhat leaky scsi abstraction as other devices behind the
> > > > > mpt
> > > > > controller.
> > > > 
> > > > You might think about what we did for SAS: split the generic
> > > > handler
> > > > into two pieces, libsas for driving the devices, which mpt
> > > > didn't
> > > > need because of the fat firmware and the SAS transport class so
> > > > mpt
> > > > could at least show the same sysfs files as everything else for
> > > > SAS
> > > > devices.
> > > 
> > >  Ventura generation of controllers are adding connectivity of
> > > NVME
> > >  drives seamlessly and protocol handling is in Firmware.
> > >  Same as SCSI to ATA translation done in firmware, Ventura
> > > controller
> > >  is doing SCSI to NVME translation and for end user protocol
> > > handling
> > >  is abstracted.
> > > 
> > >  This product handles new Transport protocol (NVME) same as ATA
> > > and
> > >  transport is abstracted for end user.
> > > 
> > > NVME pass-through related driver code, it is just a big tunnel
> > > for
> > > user space application. It is just a basic framework like SATA
> > > PASS-
> > > Through in existing mpt3sas driver.
> > 
> > I know how it works ... and I'm on record as not liking your SATL
> > approach
> > because we keep tripping across bugs in the SATL that we have to
> > fix in
> > the
> > driver.
> 
> We discussed about NVME device support behind  to Hannes and
> he suggested to describe to product behavior to wider audience to be
> aware. Just wanted to share the notes.

Great, thanks! it's certainly better than r

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[James Bottomley]

On Mon, 2017-08-07 at 20:01 +0530, Kashyap Desai wrote:
> > 
> > -Original Message-
> > From: James Bottomley [mailto:james.bottom...@hansenpartnership.com
> > ]
> > Sent: Monday, August 07, 2017 7:48 PM
> > To: Kashyap Desai; Christoph Hellwig; Hannes Reinecke
> > Cc: Suganath Prabu Subramani; martin.peter...@oracle.com; linux-
> > s...@vger.kernel.org; Sathya Prakash Veerichetty; linux-
> > ker...@vger.kernel.org; Chaitra Basappa; Sreekanth Reddy; linux-
> > n...@lists.infradead.org
> > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > 
> > On Mon, 2017-08-07 at 19:26 +0530, Kashyap Desai wrote:
> > > 
> > > > 
> > > > 
> > > > -Original Message-
> > > > From: James Bottomley [mailto:James.Bottomley@HansenPartnership
> > > > .com
> > > > ]
> > > > Sent: Saturday, August 05, 2017 8:12 PM
> > > > To: Christoph Hellwig; Hannes Reinecke
> > > > Cc: Suganath Prabu S; martin.peter...@oracle.com; linux-
> > > > s...@vger.kernel.org; sathya.prak...@broadcom.com;
> > > > kashyap.de...@broadcom.com; linux-kernel@vger.kernel.org;
> > > > chaitra.basa...@broadcom.com; sreekanth.re...@broadcom.com;
> > > > linux-
> > > > n...@lists.infradead.org
> > > > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > > > 
> > > > On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > > > > 
> > > > > 
> > > > > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke
> > > > > wrote:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > I'm not happy with this approach.
> > > > > > NVMe devices should _not_ appear as SCSI devices; this will
> > > > > > just
> > > > > > confuse matters _and_ will be incompatible with 'normal'
> > > > > > NVMe
> > > > > > devices.
> > > > > > 
> > > > > > Rather I would like to see the driver to hook into the
> > > > > > existing
> > > > > > NVMe framework (which essentially means to treat the
> > > > > > mpt3sas as
> > > > > > a weird NVMe-over-Fabrics HBA), and expose the NVMe devices
> > > > > > like
> > > > > > any other NVMe HBA.
> > > > > 
> > > > > That doesn't make any sense.  The devices behind the mpt
> > > > > adapter
> > > > > don't look like NVMe devices at all for the hosts - there are
> > > > > no
> > > > > NVMe commands or queues involved at all, they hide behind the
> > > > > same
> > > > > somewhat leaky scsi abstraction as other devices behind the
> > > > > mpt
> > > > > controller.
> > > > 
> > > > You might think about what we did for SAS: split the generic
> > > > handler
> > > > into two pieces, libsas for driving the devices, which mpt
> > > > didn't
> > > > need because of the fat firmware and the SAS transport class so
> > > > mpt
> > > > could at least show the same sysfs files as everything else for
> > > > SAS
> > > > devices.
> > > 
> > >  Ventura generation of controllers are adding connectivity of
> > > NVME
> > >  drives seamlessly and protocol handling is in Firmware.
> > >  Same as SCSI to ATA translation done in firmware, Ventura
> > > controller
> > >  is doing SCSI to NVME translation and for end user protocol
> > > handling
> > >  is abstracted.
> > > 
> > >  This product handles new Transport protocol (NVME) same as ATA
> > > and
> > >  transport is abstracted for end user.
> > > 
> > > NVME pass-through related driver code, it is just a big tunnel
> > > for
> > > user space application. It is just a basic framework like SATA
> > > PASS-
> > > Through in existing mpt3sas driver.
> > 
> > I know how it works ... and I'm on record as not liking your SATL
> > approach
> > because we keep tripping across bugs in the SATL that we have to
> > fix in
> > the
> > driver.
> 
> We discussed about NVME device support behind  to Hannes and
> he suggested to describe to product behavior to wider audience to be
> aware. Just wanted to share the notes.

Great, thanks! it's certainly better than r

RE: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Kashyap Desai]

> -Original Message-
> From: James Bottomley [mailto:james.bottom...@hansenpartnership.com]
> Sent: Monday, August 07, 2017 7:48 PM
> To: Kashyap Desai; Christoph Hellwig; Hannes Reinecke
> Cc: Suganath Prabu Subramani; martin.peter...@oracle.com; linux-
> s...@vger.kernel.org; Sathya Prakash Veerichetty; linux-
> ker...@vger.kernel.org; Chaitra Basappa; Sreekanth Reddy; linux-
> n...@lists.infradead.org
> Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
>
> On Mon, 2017-08-07 at 19:26 +0530, Kashyap Desai wrote:
> > >
> > > -Original Message-
> > > From: James Bottomley [mailto:james.bottom...@hansenpartnership.com
> > > ]
> > > Sent: Saturday, August 05, 2017 8:12 PM
> > > To: Christoph Hellwig; Hannes Reinecke
> > > Cc: Suganath Prabu S; martin.peter...@oracle.com; linux-
> > > s...@vger.kernel.org; sathya.prak...@broadcom.com;
> > > kashyap.de...@broadcom.com; linux-kernel@vger.kernel.org;
> > > chaitra.basa...@broadcom.com; sreekanth.re...@broadcom.com; linux-
> > > n...@lists.infradead.org
> > > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > >
> > > On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > > >
> > > > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> > > > >
> > > > >
> > > > > I'm not happy with this approach.
> > > > > NVMe devices should _not_ appear as SCSI devices; this will just
> > > > > confuse matters _and_ will be incompatible with 'normal' NVMe
> > > > > devices.
> > > > >
> > > > > Rather I would like to see the driver to hook into the existing
> > > > > NVMe framework (which essentially means to treat the mpt3sas as
> > > > > a weird NVMe-over-Fabrics HBA), and expose the NVMe devices like
> > > > > any other NVMe HBA.
> > > >
> > > > That doesn't make any sense.  The devices behind the mpt adapter
> > > > don't look like NVMe devices at all for the hosts - there are no
> > > > NVMe commands or queues involved at all, they hide behind the same
> > > > somewhat leaky scsi abstraction as other devices behind the mpt
> > > > controller.
> > >
> > > You might think about what we did for SAS: split the generic handler
> > > into two pieces, libsas for driving the devices, which mpt didn't
> > > need because of the fat firmware and the SAS transport class so mpt
> > > could at least show the same sysfs files as everything else for SAS
> > > devices.
> >
> >  Ventura generation of controllers are adding connectivity of NVME
> >  drives seamlessly and protocol handling is in Firmware.
> >  Same as SCSI to ATA translation done in firmware, Ventura controller
> >  is doing SCSI to NVME translation and for end user protocol handling
> >  is abstracted.
> >
> >  This product handles new Transport protocol (NVME) same as ATA and
> >  transport is abstracted for end user.
> >
> > NVME pass-through related driver code, it is just a big tunnel for
> > user space application. It is just a basic framework like SATA PASS-
> > Through in existing mpt3sas driver.
>
> I know how it works ... and I'm on record as not liking your SATL approach
> because we keep tripping across bugs in the SATL that we have to fix in
> the
> driver.

We discussed about NVME device support behind  to Hannes and
he suggested to describe to product behavior to wider audience to be aware.
Just wanted to share the notes.

>
> However, at least for bot SAS and SATA they appear to the system as SCSI
> devices regardless of HBA, so we've largely smoothed over any problems if
> you
> transfer from mp3sas to another SAS/SATA controller.
>
> I believe your current proposal is to have NVMe devices appear as SCSI,
> which
> isn't how the native NVMe driver handles them at all.  This is going to
> have to
> be special cased in any tool designed to handle nvme devices and it's
> going to
> cause big problems if someone changes controller (or moves the
> device).  What's the proposal for making this as painless as possible?

We have to attempt this use case and see how it behaves. I have not tried
this, so not sure if things are really bad or just some tuning may be
helpful. I will revert back to you on this.

I understood request as -  We need some udev rules to be working well for
*same* NVME drives if it is behind  or native .
Example - If user has OS installed on NVME drive which is behind 
driver as SCSI disk should be able to boot if he/she hooked same NVME drive
which is detected by native  driver (and vice versa.)

>
> James

RE: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Kashyap Desai]

> -Original Message-
> From: James Bottomley [mailto:james.bottom...@hansenpartnership.com]
> Sent: Monday, August 07, 2017 7:48 PM
> To: Kashyap Desai; Christoph Hellwig; Hannes Reinecke
> Cc: Suganath Prabu Subramani; martin.peter...@oracle.com; linux-
> s...@vger.kernel.org; Sathya Prakash Veerichetty; linux-
> ker...@vger.kernel.org; Chaitra Basappa; Sreekanth Reddy; linux-
> n...@lists.infradead.org
> Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
>
> On Mon, 2017-08-07 at 19:26 +0530, Kashyap Desai wrote:
> > >
> > > -Original Message-
> > > From: James Bottomley [mailto:james.bottom...@hansenpartnership.com
> > > ]
> > > Sent: Saturday, August 05, 2017 8:12 PM
> > > To: Christoph Hellwig; Hannes Reinecke
> > > Cc: Suganath Prabu S; martin.peter...@oracle.com; linux-
> > > s...@vger.kernel.org; sathya.prak...@broadcom.com;
> > > kashyap.de...@broadcom.com; linux-kernel@vger.kernel.org;
> > > chaitra.basa...@broadcom.com; sreekanth.re...@broadcom.com; linux-
> > > n...@lists.infradead.org
> > > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > >
> > > On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > > >
> > > > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> > > > >
> > > > >
> > > > > I'm not happy with this approach.
> > > > > NVMe devices should _not_ appear as SCSI devices; this will just
> > > > > confuse matters _and_ will be incompatible with 'normal' NVMe
> > > > > devices.
> > > > >
> > > > > Rather I would like to see the driver to hook into the existing
> > > > > NVMe framework (which essentially means to treat the mpt3sas as
> > > > > a weird NVMe-over-Fabrics HBA), and expose the NVMe devices like
> > > > > any other NVMe HBA.
> > > >
> > > > That doesn't make any sense.  The devices behind the mpt adapter
> > > > don't look like NVMe devices at all for the hosts - there are no
> > > > NVMe commands or queues involved at all, they hide behind the same
> > > > somewhat leaky scsi abstraction as other devices behind the mpt
> > > > controller.
> > >
> > > You might think about what we did for SAS: split the generic handler
> > > into two pieces, libsas for driving the devices, which mpt didn't
> > > need because of the fat firmware and the SAS transport class so mpt
> > > could at least show the same sysfs files as everything else for SAS
> > > devices.
> >
> >  Ventura generation of controllers are adding connectivity of NVME
> >  drives seamlessly and protocol handling is in Firmware.
> >  Same as SCSI to ATA translation done in firmware, Ventura controller
> >  is doing SCSI to NVME translation and for end user protocol handling
> >  is abstracted.
> >
> >  This product handles new Transport protocol (NVME) same as ATA and
> >  transport is abstracted for end user.
> >
> > NVME pass-through related driver code, it is just a big tunnel for
> > user space application. It is just a basic framework like SATA PASS-
> > Through in existing mpt3sas driver.
>
> I know how it works ... and I'm on record as not liking your SATL approach
> because we keep tripping across bugs in the SATL that we have to fix in
> the
> driver.

We discussed about NVME device support behind  to Hannes and
he suggested to describe to product behavior to wider audience to be aware.
Just wanted to share the notes.

>
> However, at least for bot SAS and SATA they appear to the system as SCSI
> devices regardless of HBA, so we've largely smoothed over any problems if
> you
> transfer from mp3sas to another SAS/SATA controller.
>
> I believe your current proposal is to have NVMe devices appear as SCSI,
> which
> isn't how the native NVMe driver handles them at all.  This is going to
> have to
> be special cased in any tool designed to handle nvme devices and it's
> going to
> cause big problems if someone changes controller (or moves the
> device).  What's the proposal for making this as painless as possible?

We have to attempt this use case and see how it behaves. I have not tried
this, so not sure if things are really bad or just some tuning may be
helpful. I will revert back to you on this.

I understood request as -  We need some udev rules to be working well for
*same* NVME drives if it is behind  or native .
Example - If user has OS installed on NVME drive which is behind 
driver as SCSI disk should be able to boot if he/she hooked same NVME drive
which is detected by native  driver (and vice versa.)

>
> James

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[James Bottomley]

On Mon, 2017-08-07 at 19:26 +0530, Kashyap Desai wrote:
> > 
> > -Original Message-
> > From: James Bottomley [mailto:james.bottom...@hansenpartnership.com
> > ]
> > Sent: Saturday, August 05, 2017 8:12 PM
> > To: Christoph Hellwig; Hannes Reinecke
> > Cc: Suganath Prabu S; martin.peter...@oracle.com; linux-
> > s...@vger.kernel.org; sathya.prak...@broadcom.com;
> > kashyap.de...@broadcom.com; linux-kernel@vger.kernel.org;
> > chaitra.basa...@broadcom.com; sreekanth.re...@broadcom.com; linux-
> > n...@lists.infradead.org
> > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > 
> > On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > > 
> > > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> > > > 
> > > > 
> > > > I'm not happy with this approach.
> > > > NVMe devices should _not_ appear as SCSI devices; this will
> > > > just
> > > > confuse matters _and_ will be incompatible with 'normal' NVMe
> > > > devices.
> > > > 
> > > > Rather I would like to see the driver to hook into the existing
> > > > NVMe
> > > > framework (which essentially means to treat the mpt3sas as a
> > > > weird
> > > > NVMe-over-Fabrics HBA), and expose the NVMe devices like any
> > > > other
> > > > NVMe HBA.
> > > 
> > > That doesn't make any sense.  The devices behind the mpt adapter
> > > don't
> > > look like NVMe devices at all for the hosts - there are no NVMe
> > > commands or queues involved at all, they hide behind the same
> > > somewhat
> > > leaky scsi abstraction as other devices behind the mpt
> > > controller.
> > 
> > You might think about what we did for SAS: split the generic
> > handler into
> > two
> > pieces, libsas for driving the devices, which mpt didn't need
> > because of
> > the fat
> > firmware and the SAS transport class so mpt could at least show the
> > same
> > sysfs
> > files as everything else for SAS devices.
> 
>  Ventura generation of controllers are adding connectivity of NVME
>  drives seamlessly and protocol handling is in Firmware.
>  Same as SCSI to ATA translation done in firmware, Ventura controller
>  is doing SCSI to NVME translation and for end user protocol handling
>  is abstracted.
> 
>  This product handles new Transport protocol (NVME) same as ATA and
>  transport is abstracted for end user.
> 
> NVME pass-through related driver code, it is just a big tunnel for
> user space application. It is just a basic framework like SATA PASS-
> Through in existing mpt3sas driver.

I know how it works ... and I'm on record as not liking your SATL
approach because we keep tripping across bugs in the SATL that we have
to fix in the driver.

However, at least for bot SAS and SATA they appear to the system as
SCSI devices regardless of HBA, so we've largely smoothed over any
problems if you transfer from mp3sas to another SAS/SATA controller.

I believe your current proposal is to have NVMe devices appear as SCSI,
which isn't how the native NVMe driver handles them at all.  This is
going to have to be special cased in any tool designed to handle nvme
devices and it's going to cause big problems if someone changes
controller (or moves the device).  What's the proposal for making this
as painless as possible?

James

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[James Bottomley]

On Mon, 2017-08-07 at 19:26 +0530, Kashyap Desai wrote:
> > 
> > -Original Message-
> > From: James Bottomley [mailto:james.bottom...@hansenpartnership.com
> > ]
> > Sent: Saturday, August 05, 2017 8:12 PM
> > To: Christoph Hellwig; Hannes Reinecke
> > Cc: Suganath Prabu S; martin.peter...@oracle.com; linux-
> > s...@vger.kernel.org; sathya.prak...@broadcom.com;
> > kashyap.de...@broadcom.com; linux-kernel@vger.kernel.org;
> > chaitra.basa...@broadcom.com; sreekanth.re...@broadcom.com; linux-
> > n...@lists.infradead.org
> > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > 
> > On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > > 
> > > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> > > > 
> > > > 
> > > > I'm not happy with this approach.
> > > > NVMe devices should _not_ appear as SCSI devices; this will
> > > > just
> > > > confuse matters _and_ will be incompatible with 'normal' NVMe
> > > > devices.
> > > > 
> > > > Rather I would like to see the driver to hook into the existing
> > > > NVMe
> > > > framework (which essentially means to treat the mpt3sas as a
> > > > weird
> > > > NVMe-over-Fabrics HBA), and expose the NVMe devices like any
> > > > other
> > > > NVMe HBA.
> > > 
> > > That doesn't make any sense.  The devices behind the mpt adapter
> > > don't
> > > look like NVMe devices at all for the hosts - there are no NVMe
> > > commands or queues involved at all, they hide behind the same
> > > somewhat
> > > leaky scsi abstraction as other devices behind the mpt
> > > controller.
> > 
> > You might think about what we did for SAS: split the generic
> > handler into
> > two
> > pieces, libsas for driving the devices, which mpt didn't need
> > because of
> > the fat
> > firmware and the SAS transport class so mpt could at least show the
> > same
> > sysfs
> > files as everything else for SAS devices.
> 
>  Ventura generation of controllers are adding connectivity of NVME
>  drives seamlessly and protocol handling is in Firmware.
>  Same as SCSI to ATA translation done in firmware, Ventura controller
>  is doing SCSI to NVME translation and for end user protocol handling
>  is abstracted.
> 
>  This product handles new Transport protocol (NVME) same as ATA and
>  transport is abstracted for end user.
> 
> NVME pass-through related driver code, it is just a big tunnel for
> user space application. It is just a basic framework like SATA PASS-
> Through in existing mpt3sas driver.

I know how it works ... and I'm on record as not liking your SATL
approach because we keep tripping across bugs in the SATL that we have
to fix in the driver.

However, at least for bot SAS and SATA they appear to the system as
SCSI devices regardless of HBA, so we've largely smoothed over any
problems if you transfer from mp3sas to another SAS/SATA controller.

I believe your current proposal is to have NVMe devices appear as SCSI,
which isn't how the native NVMe driver handles them at all.  This is
going to have to be special cased in any tool designed to handle nvme
devices and it's going to cause big problems if someone changes
controller (or moves the device).  What's the proposal for making this
as painless as possible?

James

RE: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Kashyap Desai]

> -Original Message-
> From: James Bottomley [mailto:james.bottom...@hansenpartnership.com]
> Sent: Saturday, August 05, 2017 8:12 PM
> To: Christoph Hellwig; Hannes Reinecke
> Cc: Suganath Prabu S; martin.peter...@oracle.com; linux-
> s...@vger.kernel.org; sathya.prak...@broadcom.com;
> kashyap.de...@broadcom.com; linux-kernel@vger.kernel.org;
> chaitra.basa...@broadcom.com; sreekanth.re...@broadcom.com; linux-
> n...@lists.infradead.org
> Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
>
> On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> > >
> > > I'm not happy with this approach.
> > > NVMe devices should _not_ appear as SCSI devices; this will just
> > > confuse matters _and_ will be incompatible with 'normal' NVMe
> > > devices.
> > >
> > > Rather I would like to see the driver to hook into the existing NVMe
> > > framework (which essentially means to treat the mpt3sas as a weird
> > > NVMe-over-Fabrics HBA), and expose the NVMe devices like any other
> > > NVMe HBA.
> >
> > That doesn't make any sense.  The devices behind the mpt adapter don't
> > look like NVMe devices at all for the hosts - there are no NVMe
> > commands or queues involved at all, they hide behind the same somewhat
> > leaky scsi abstraction as other devices behind the mpt controller.
>
> You might think about what we did for SAS: split the generic handler into
> two
> pieces, libsas for driving the devices, which mpt didn't need because of
> the fat
> firmware and the SAS transport class so mpt could at least show the same
> sysfs
> files as everything else for SAS devices.

 Ventura generation of controllers are adding connectivity of NVME
 drives seamlessly and protocol handling is in Firmware.
 Same as SCSI to ATA translation done in firmware, Ventura controller
 is doing SCSI to NVME translation and for end user protocol handling
 is abstracted.

 This product handles new Transport protocol (NVME) same as ATA and
 transport is abstracted for end user.

NVME pass-through related driver code, it is just a big tunnel for user
space application.
It is just a basic framework like SATA PASS-Through in existing mpt3sas
driver.

>
> Fortunately for NVMe it's very simple at the moment its just a couple of
> host
> files and wwid on the devices.
>
> James
>
>
> > The only additional leak is that the controller now supports NVMe-
> > like PRPs in additions to its existing two SGL formats.
> >

RE: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Kashyap Desai]

> -Original Message-
> From: James Bottomley [mailto:james.bottom...@hansenpartnership.com]
> Sent: Saturday, August 05, 2017 8:12 PM
> To: Christoph Hellwig; Hannes Reinecke
> Cc: Suganath Prabu S; martin.peter...@oracle.com; linux-
> s...@vger.kernel.org; sathya.prak...@broadcom.com;
> kashyap.de...@broadcom.com; linux-kernel@vger.kernel.org;
> chaitra.basa...@broadcom.com; sreekanth.re...@broadcom.com; linux-
> n...@lists.infradead.org
> Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
>
> On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> > >
> > > I'm not happy with this approach.
> > > NVMe devices should _not_ appear as SCSI devices; this will just
> > > confuse matters _and_ will be incompatible with 'normal' NVMe
> > > devices.
> > >
> > > Rather I would like to see the driver to hook into the existing NVMe
> > > framework (which essentially means to treat the mpt3sas as a weird
> > > NVMe-over-Fabrics HBA), and expose the NVMe devices like any other
> > > NVMe HBA.
> >
> > That doesn't make any sense.  The devices behind the mpt adapter don't
> > look like NVMe devices at all for the hosts - there are no NVMe
> > commands or queues involved at all, they hide behind the same somewhat
> > leaky scsi abstraction as other devices behind the mpt controller.
>
> You might think about what we did for SAS: split the generic handler into
> two
> pieces, libsas for driving the devices, which mpt didn't need because of
> the fat
> firmware and the SAS transport class so mpt could at least show the same
> sysfs
> files as everything else for SAS devices.

 Ventura generation of controllers are adding connectivity of NVME
 drives seamlessly and protocol handling is in Firmware.
 Same as SCSI to ATA translation done in firmware, Ventura controller
 is doing SCSI to NVME translation and for end user protocol handling
 is abstracted.

 This product handles new Transport protocol (NVME) same as ATA and
 transport is abstracted for end user.

NVME pass-through related driver code, it is just a big tunnel for user
space application.
It is just a basic framework like SATA PASS-Through in existing mpt3sas
driver.

>
> Fortunately for NVMe it's very simple at the moment its just a couple of
> host
> files and wwid on the devices.
>
> James
>
>
> > The only additional leak is that the controller now supports NVMe-
> > like PRPs in additions to its existing two SGL formats.
> >

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[James Bottomley]

On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> > 
> > I'm not happy with this approach.
> > NVMe devices should _not_ appear as SCSI devices; this will just
> > confuse matters _and_ will be incompatible with 'normal' NVMe
> > devices.
> > 
> > Rather I would like to see the driver to hook into the existing
> > NVMe framework (which essentially means to treat the mpt3sas as a
> > weird NVMe-over-Fabrics HBA), and expose the NVMe devices like any
> > other NVMe HBA.
> 
> That doesn't make any sense.  The devices behind the mpt adapter
> don't look like NVMe devices at all for the hosts - there are no NVMe
> commands or queues involved at all, they hide behind the same
> somewhat leaky scsi abstraction as other devices behind the mpt
> controller.

You might think about what we did for SAS: split the generic handler
into two pieces, libsas for driving the devices, which mpt didn't need
because of the fat firmware and the SAS transport class so mpt could at
least show the same sysfs files as everything else for SAS devices.

Fortunately for NVMe it's very simple at the moment its just a couple
of host files and wwid on the devices.

James


> The only additional leak is that the controller now supports NVMe-
> like PRPs in additions to its existing two SGL formats.
> 

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[James Bottomley]

On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> > 
> > I'm not happy with this approach.
> > NVMe devices should _not_ appear as SCSI devices; this will just
> > confuse matters _and_ will be incompatible with 'normal' NVMe
> > devices.
> > 
> > Rather I would like to see the driver to hook into the existing
> > NVMe framework (which essentially means to treat the mpt3sas as a
> > weird NVMe-over-Fabrics HBA), and expose the NVMe devices like any
> > other NVMe HBA.
> 
> That doesn't make any sense.  The devices behind the mpt adapter
> don't look like NVMe devices at all for the hosts - there are no NVMe
> commands or queues involved at all, they hide behind the same
> somewhat leaky scsi abstraction as other devices behind the mpt
> controller.

You might think about what we did for SAS: split the generic handler
into two pieces, libsas for driving the devices, which mpt didn't need
because of the fat firmware and the SAS transport class so mpt could at
least show the same sysfs files as everything else for SAS devices.

Fortunately for NVMe it's very simple at the moment its just a couple
of host files and wwid on the devices.

James


> The only additional leak is that the controller now supports NVMe-
> like PRPs in additions to its existing two SGL formats.
> 

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Christoph Hellwig]

On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> I'm not happy with this approach.
> NVMe devices should _not_ appear as SCSI devices; this will just confuse
> matters _and_ will be incompatible with 'normal' NVMe devices.
> 
> Rather I would like to see the driver to hook into the existing NVMe
> framework (which essentially means to treat the mpt3sas as a weird
> NVMe-over-Fabrics HBA), and expose the NVMe devices like any other NVMe HBA.

That doesn't make any sense.  The devices behind the mpt adapter don't
look like NVMe devices at all for the hosts - there are no NVMe commands
or queues involved at all, they hide behind the same somewhat leaky
scsi abstraction as other devices behind the mpt controller.

The only additional leak is that the controller now supports NVMe-like
PRPs in additions to its existing two SGL formats.

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Christoph Hellwig]

On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke wrote:
> I'm not happy with this approach.
> NVMe devices should _not_ appear as SCSI devices; this will just confuse
> matters _and_ will be incompatible with 'normal' NVMe devices.
> 
> Rather I would like to see the driver to hook into the existing NVMe
> framework (which essentially means to treat the mpt3sas as a weird
> NVMe-over-Fabrics HBA), and expose the NVMe devices like any other NVMe HBA.

That doesn't make any sense.  The devices behind the mpt adapter don't
look like NVMe devices at all for the hosts - there are no NVMe commands
or queues involved at all, they hide behind the same somewhat leaky
scsi abstraction as other devices behind the mpt controller.

The only additional leak is that the controller now supports NVMe-like
PRPs in additions to its existing two SGL formats.

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Hannes Reinecke]

On 08/02/2017 10:14 AM, Hannes Reinecke wrote:
> On 07/14/2017 03:22 PM, Suganath Prabu S wrote:
>> Ventura Series controller are Tri-mode. The controller and
>> firmware are capable of supporting NVMe devices and
>> PCIe switches to be connected with the controller. This
>> patch set adds driver level support for NVMe devices and
>> PCIe switches.
>>
>> Suganath Prabu S (13):
>>   mpt3sas: Update MPI Header
>>   mpt3sas: Add nvme device support in slave alloc, target alloc and
>> probe
>>   mpt3sas: SGL to PRP Translation for I/Os to NVMe  devices
>>   mpt3sas: Added support for nvme encapsulated request message.
>>   mpt3sas: API 's to support NVMe drive addition to SML
>>   mpt3sas: API's to remove nvme drive from sml
>>   mpt3sas: Handle NVMe PCIe device related events generated
>>from firmware.
>>   mpt3sas: Set NVMe device queue depth as 128
>>   mpt3sas: scan and add nvme device after controller reset
>>   mpt3as: Add-Task-management-debug-info-for-NVMe-drives.
>>   mpt3sas: NVMe drive support for BTDHMAPPING ioctl command and log
>> info
>>   mpt3sas: Fix nvme drives checking for tlr.
>>   mpt3sas: Update mpt3sas driver version.
>>
>>  drivers/scsi/mpt3sas/mpi/mpi2.h  |   43 +-
>>  drivers/scsi/mpt3sas/mpi/mpi2_cnfg.h |  647 ++-
>>  drivers/scsi/mpt3sas/mpi/mpi2_init.h |   11 +-
>>  drivers/scsi/mpt3sas/mpi/mpi2_ioc.h  |  331 ++-
>>  drivers/scsi/mpt3sas/mpi/mpi2_pci.h  |  142 +++
>>  drivers/scsi/mpt3sas/mpi/mpi2_tool.h |   14 +-
>>  drivers/scsi/mpt3sas/mpt3sas_base.c  |  710 +++-
>>  drivers/scsi/mpt3sas/mpt3sas_base.h  |  171 +++-
>>  drivers/scsi/mpt3sas/mpt3sas_config.c|  100 ++
>>  drivers/scsi/mpt3sas/mpt3sas_ctl.c   |  158 ++-
>>  drivers/scsi/mpt3sas/mpt3sas_scsih.c | 1874 
>> --
>>  drivers/scsi/mpt3sas/mpt3sas_warpdrive.c |2 +-
>>  12 files changed, 4063 insertions(+), 140 deletions(-)
>>  create mode 100644 drivers/scsi/mpt3sas/mpi/mpi2_pci.h
>>
> I'm not happy with this approach.
> NVMe devices should _not_ appear as SCSI devices; this will just confuse
> matters _and_ will be incompatible with 'normal' NVMe devices.
> 
> Rather I would like to see the driver to hook into the existing NVMe
> framework (which essentially means to treat the mpt3sas as a weird
> NVMe-over-Fabrics HBA), and expose the NVMe devices like any other NVMe HBA.
> 
> I'm sorry that you'll have to redo your patchset (again), but this is
> the only way I see how this patchset can be brought forward.
> 
> I'd be happy to discuss implementation details with you.
> 
After discussion with Broadcom it turns out that the NVMe passthrough
functionality of the current firmware is rather limited and wouldn't be
suitable for full NVMe device support.
And one of the goals is to have intermixed NVMe and SCSI support,
possibly with RAID thrown in for good measure.
So my idea of exposing the NVMe devices directly won't work, and I'll
have to retract my above comment.

But I'll continue to bug Broadcom; maybe we'll be getting NVMe device
support eventually :-)

Cheers,

Hannes
-- 
Dr. Hannes ReineckeTeamlead Storage & Networking
h...@suse.de   +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Hannes Reinecke]

On 08/02/2017 10:14 AM, Hannes Reinecke wrote:
> On 07/14/2017 03:22 PM, Suganath Prabu S wrote:
>> Ventura Series controller are Tri-mode. The controller and
>> firmware are capable of supporting NVMe devices and
>> PCIe switches to be connected with the controller. This
>> patch set adds driver level support for NVMe devices and
>> PCIe switches.
>>
>> Suganath Prabu S (13):
>>   mpt3sas: Update MPI Header
>>   mpt3sas: Add nvme device support in slave alloc, target alloc and
>> probe
>>   mpt3sas: SGL to PRP Translation for I/Os to NVMe  devices
>>   mpt3sas: Added support for nvme encapsulated request message.
>>   mpt3sas: API 's to support NVMe drive addition to SML
>>   mpt3sas: API's to remove nvme drive from sml
>>   mpt3sas: Handle NVMe PCIe device related events generated
>>from firmware.
>>   mpt3sas: Set NVMe device queue depth as 128
>>   mpt3sas: scan and add nvme device after controller reset
>>   mpt3as: Add-Task-management-debug-info-for-NVMe-drives.
>>   mpt3sas: NVMe drive support for BTDHMAPPING ioctl command and log
>> info
>>   mpt3sas: Fix nvme drives checking for tlr.
>>   mpt3sas: Update mpt3sas driver version.
>>
>>  drivers/scsi/mpt3sas/mpi/mpi2.h  |   43 +-
>>  drivers/scsi/mpt3sas/mpi/mpi2_cnfg.h |  647 ++-
>>  drivers/scsi/mpt3sas/mpi/mpi2_init.h |   11 +-
>>  drivers/scsi/mpt3sas/mpi/mpi2_ioc.h  |  331 ++-
>>  drivers/scsi/mpt3sas/mpi/mpi2_pci.h  |  142 +++
>>  drivers/scsi/mpt3sas/mpi/mpi2_tool.h |   14 +-
>>  drivers/scsi/mpt3sas/mpt3sas_base.c  |  710 +++-
>>  drivers/scsi/mpt3sas/mpt3sas_base.h  |  171 +++-
>>  drivers/scsi/mpt3sas/mpt3sas_config.c|  100 ++
>>  drivers/scsi/mpt3sas/mpt3sas_ctl.c   |  158 ++-
>>  drivers/scsi/mpt3sas/mpt3sas_scsih.c | 1874 
>> --
>>  drivers/scsi/mpt3sas/mpt3sas_warpdrive.c |2 +-
>>  12 files changed, 4063 insertions(+), 140 deletions(-)
>>  create mode 100644 drivers/scsi/mpt3sas/mpi/mpi2_pci.h
>>
> I'm not happy with this approach.
> NVMe devices should _not_ appear as SCSI devices; this will just confuse
> matters _and_ will be incompatible with 'normal' NVMe devices.
> 
> Rather I would like to see the driver to hook into the existing NVMe
> framework (which essentially means to treat the mpt3sas as a weird
> NVMe-over-Fabrics HBA), and expose the NVMe devices like any other NVMe HBA.
> 
> I'm sorry that you'll have to redo your patchset (again), but this is
> the only way I see how this patchset can be brought forward.
> 
> I'd be happy to discuss implementation details with you.
> 
After discussion with Broadcom it turns out that the NVMe passthrough
functionality of the current firmware is rather limited and wouldn't be
suitable for full NVMe device support.
And one of the goals is to have intermixed NVMe and SCSI support,
possibly with RAID thrown in for good measure.
So my idea of exposing the NVMe devices directly won't work, and I'll
have to retract my above comment.

But I'll continue to bug Broadcom; maybe we'll be getting NVMe device
support eventually :-)

Cheers,

Hannes
-- 
Dr. Hannes ReineckeTeamlead Storage & Networking
h...@suse.de   +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Hannes Reinecke]

On 07/14/2017 03:22 PM, Suganath Prabu S wrote:
> Ventura Series controller are Tri-mode. The controller and
> firmware are capable of supporting NVMe devices and
> PCIe switches to be connected with the controller. This
> patch set adds driver level support for NVMe devices and
> PCIe switches.
> 
> Suganath Prabu S (13):
>   mpt3sas: Update MPI Header
>   mpt3sas: Add nvme device support in slave alloc, target alloc and
> probe
>   mpt3sas: SGL to PRP Translation for I/Os to NVMe  devices
>   mpt3sas: Added support for nvme encapsulated request message.
>   mpt3sas: API 's to support NVMe drive addition to SML
>   mpt3sas: API's to remove nvme drive from sml
>   mpt3sas: Handle NVMe PCIe device related events generated
>from firmware.
>   mpt3sas: Set NVMe device queue depth as 128
>   mpt3sas: scan and add nvme device after controller reset
>   mpt3as: Add-Task-management-debug-info-for-NVMe-drives.
>   mpt3sas: NVMe drive support for BTDHMAPPING ioctl command and log
> info
>   mpt3sas: Fix nvme drives checking for tlr.
>   mpt3sas: Update mpt3sas driver version.
> 
>  drivers/scsi/mpt3sas/mpi/mpi2.h  |   43 +-
>  drivers/scsi/mpt3sas/mpi/mpi2_cnfg.h |  647 ++-
>  drivers/scsi/mpt3sas/mpi/mpi2_init.h |   11 +-
>  drivers/scsi/mpt3sas/mpi/mpi2_ioc.h  |  331 ++-
>  drivers/scsi/mpt3sas/mpi/mpi2_pci.h  |  142 +++
>  drivers/scsi/mpt3sas/mpi/mpi2_tool.h |   14 +-
>  drivers/scsi/mpt3sas/mpt3sas_base.c  |  710 +++-
>  drivers/scsi/mpt3sas/mpt3sas_base.h  |  171 +++-
>  drivers/scsi/mpt3sas/mpt3sas_config.c|  100 ++
>  drivers/scsi/mpt3sas/mpt3sas_ctl.c   |  158 ++-
>  drivers/scsi/mpt3sas/mpt3sas_scsih.c | 1874 
> --
>  drivers/scsi/mpt3sas/mpt3sas_warpdrive.c |2 +-
>  12 files changed, 4063 insertions(+), 140 deletions(-)
>  create mode 100644 drivers/scsi/mpt3sas/mpi/mpi2_pci.h
> 
I'm not happy with this approach.
NVMe devices should _not_ appear as SCSI devices; this will just confuse
matters _and_ will be incompatible with 'normal' NVMe devices.

Rather I would like to see the driver to hook into the existing NVMe
framework (which essentially means to treat the mpt3sas as a weird
NVMe-over-Fabrics HBA), and expose the NVMe devices like any other NVMe HBA.

I'm sorry that you'll have to redo your patchset (again), but this is
the only way I see how this patchset can be brought forward.

I'd be happy to discuss implementation details with you.

Cheers,

Hannes
-- 
Dr. Hannes ReineckeTeamlead Storage & Networking
h...@suse.de   +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Hannes Reinecke]

On 07/14/2017 03:22 PM, Suganath Prabu S wrote:
> Ventura Series controller are Tri-mode. The controller and
> firmware are capable of supporting NVMe devices and
> PCIe switches to be connected with the controller. This
> patch set adds driver level support for NVMe devices and
> PCIe switches.
> 
> Suganath Prabu S (13):
>   mpt3sas: Update MPI Header
>   mpt3sas: Add nvme device support in slave alloc, target alloc and
> probe
>   mpt3sas: SGL to PRP Translation for I/Os to NVMe  devices
>   mpt3sas: Added support for nvme encapsulated request message.
>   mpt3sas: API 's to support NVMe drive addition to SML
>   mpt3sas: API's to remove nvme drive from sml
>   mpt3sas: Handle NVMe PCIe device related events generated
>from firmware.
>   mpt3sas: Set NVMe device queue depth as 128
>   mpt3sas: scan and add nvme device after controller reset
>   mpt3as: Add-Task-management-debug-info-for-NVMe-drives.
>   mpt3sas: NVMe drive support for BTDHMAPPING ioctl command and log
> info
>   mpt3sas: Fix nvme drives checking for tlr.
>   mpt3sas: Update mpt3sas driver version.
> 
>  drivers/scsi/mpt3sas/mpi/mpi2.h  |   43 +-
>  drivers/scsi/mpt3sas/mpi/mpi2_cnfg.h |  647 ++-
>  drivers/scsi/mpt3sas/mpi/mpi2_init.h |   11 +-
>  drivers/scsi/mpt3sas/mpi/mpi2_ioc.h  |  331 ++-
>  drivers/scsi/mpt3sas/mpi/mpi2_pci.h  |  142 +++
>  drivers/scsi/mpt3sas/mpi/mpi2_tool.h |   14 +-
>  drivers/scsi/mpt3sas/mpt3sas_base.c  |  710 +++-
>  drivers/scsi/mpt3sas/mpt3sas_base.h  |  171 +++-
>  drivers/scsi/mpt3sas/mpt3sas_config.c|  100 ++
>  drivers/scsi/mpt3sas/mpt3sas_ctl.c   |  158 ++-
>  drivers/scsi/mpt3sas/mpt3sas_scsih.c | 1874 
> --
>  drivers/scsi/mpt3sas/mpt3sas_warpdrive.c |2 +-
>  12 files changed, 4063 insertions(+), 140 deletions(-)
>  create mode 100644 drivers/scsi/mpt3sas/mpi/mpi2_pci.h
> 
I'm not happy with this approach.
NVMe devices should _not_ appear as SCSI devices; this will just confuse
matters _and_ will be incompatible with 'normal' NVMe devices.

Rather I would like to see the driver to hook into the existing NVMe
framework (which essentially means to treat the mpt3sas as a weird
NVMe-over-Fabrics HBA), and expose the NVMe devices like any other NVMe HBA.

I'm sorry that you'll have to redo your patchset (again), but this is
the only way I see how this patchset can be brought forward.

I'd be happy to discuss implementation details with you.

Cheers,

Hannes
-- 
Dr. Hannes ReineckeTeamlead Storage & Networking
h...@suse.de   +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Suganath Prabu Subramani]

Is there any update on these patches ?

Thanks,
Suganath Prabu S

On Fri, Jul 14, 2017 at 6:52 PM, Suganath Prabu S
 wrote:
> Ventura Series controller are Tri-mode. The controller and
> firmware are capable of supporting NVMe devices and
> PCIe switches to be connected with the controller. This
> patch set adds driver level support for NVMe devices and
> PCIe switches.
>
> Suganath Prabu S (13):
>   mpt3sas: Update MPI Header
>   mpt3sas: Add nvme device support in slave alloc, target alloc and
> probe
>   mpt3sas: SGL to PRP Translation for I/Os to NVMe  devices
>   mpt3sas: Added support for nvme encapsulated request message.
>   mpt3sas: API 's to support NVMe drive addition to SML
>   mpt3sas: API's to remove nvme drive from sml
>   mpt3sas: Handle NVMe PCIe device related events generated
>from firmware.
>   mpt3sas: Set NVMe device queue depth as 128
>   mpt3sas: scan and add nvme device after controller reset
>   mpt3as: Add-Task-management-debug-info-for-NVMe-drives.
>   mpt3sas: NVMe drive support for BTDHMAPPING ioctl command and log
> info
>   mpt3sas: Fix nvme drives checking for tlr.
>   mpt3sas: Update mpt3sas driver version.
>
>  drivers/scsi/mpt3sas/mpi/mpi2.h  |   43 +-
>  drivers/scsi/mpt3sas/mpi/mpi2_cnfg.h |  647 ++-
>  drivers/scsi/mpt3sas/mpi/mpi2_init.h |   11 +-
>  drivers/scsi/mpt3sas/mpi/mpi2_ioc.h  |  331 ++-
>  drivers/scsi/mpt3sas/mpi/mpi2_pci.h  |  142 +++
>  drivers/scsi/mpt3sas/mpi/mpi2_tool.h |   14 +-
>  drivers/scsi/mpt3sas/mpt3sas_base.c  |  710 +++-
>  drivers/scsi/mpt3sas/mpt3sas_base.h  |  171 +++-
>  drivers/scsi/mpt3sas/mpt3sas_config.c|  100 ++
>  drivers/scsi/mpt3sas/mpt3sas_ctl.c   |  158 ++-
>  drivers/scsi/mpt3sas/mpt3sas_scsih.c | 1874 
> --
>  drivers/scsi/mpt3sas/mpt3sas_warpdrive.c |2 +-
>  12 files changed, 4063 insertions(+), 140 deletions(-)
>  create mode 100644 drivers/scsi/mpt3sas/mpi/mpi2_pci.h
>
> Thanks,
> Suganath Prabu S

Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[Suganath Prabu Subramani]

Is there any update on these patches ?

Thanks,
Suganath Prabu S

On Fri, Jul 14, 2017 at 6:52 PM, Suganath Prabu S
 wrote:
> Ventura Series controller are Tri-mode. The controller and
> firmware are capable of supporting NVMe devices and
> PCIe switches to be connected with the controller. This
> patch set adds driver level support for NVMe devices and
> PCIe switches.
>
> Suganath Prabu S (13):
>   mpt3sas: Update MPI Header
>   mpt3sas: Add nvme device support in slave alloc, target alloc and
> probe
>   mpt3sas: SGL to PRP Translation for I/Os to NVMe  devices
>   mpt3sas: Added support for nvme encapsulated request message.
>   mpt3sas: API 's to support NVMe drive addition to SML
>   mpt3sas: API's to remove nvme drive from sml
>   mpt3sas: Handle NVMe PCIe device related events generated
>from firmware.
>   mpt3sas: Set NVMe device queue depth as 128
>   mpt3sas: scan and add nvme device after controller reset
>   mpt3as: Add-Task-management-debug-info-for-NVMe-drives.
>   mpt3sas: NVMe drive support for BTDHMAPPING ioctl command and log
> info
>   mpt3sas: Fix nvme drives checking for tlr.
>   mpt3sas: Update mpt3sas driver version.
>
>  drivers/scsi/mpt3sas/mpi/mpi2.h  |   43 +-
>  drivers/scsi/mpt3sas/mpi/mpi2_cnfg.h |  647 ++-
>  drivers/scsi/mpt3sas/mpi/mpi2_init.h |   11 +-
>  drivers/scsi/mpt3sas/mpi/mpi2_ioc.h  |  331 ++-
>  drivers/scsi/mpt3sas/mpi/mpi2_pci.h  |  142 +++
>  drivers/scsi/mpt3sas/mpi/mpi2_tool.h |   14 +-
>  drivers/scsi/mpt3sas/mpt3sas_base.c  |  710 +++-
>  drivers/scsi/mpt3sas/mpt3sas_base.h  |  171 +++-
>  drivers/scsi/mpt3sas/mpt3sas_config.c|  100 ++
>  drivers/scsi/mpt3sas/mpt3sas_ctl.c   |  158 ++-
>  drivers/scsi/mpt3sas/mpt3sas_scsih.c | 1874 
> --
>  drivers/scsi/mpt3sas/mpt3sas_warpdrive.c |2 +-
>  12 files changed, 4063 insertions(+), 140 deletions(-)
>  create mode 100644 drivers/scsi/mpt3sas/mpi/mpi2_pci.h
>
> Thanks,
> Suganath Prabu S