[
https://issues.apache.org/jira/browse/CRAIL-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785443#comment-16785443
]
Jonas Pfefferle commented on CRAIL-93:
--------------------------------------
Sorry maybe my answer was a bit confusing.
1) We always have one datanode connect to only one subsystem and that will not
change in the future. To clarify the terminology: a datanode is not the NVMf
target but only connects to a target and sets up the available space with the
namenode. Furthermore, the datanode does not serve any IO requests but all IO
requests are done directly from the client to the NVMf target. Now, a NVMf
subsystem is basically identified by ip, port and subsystem NQN. Since we only
store the ip and port information for each block on the namenode and the
subsystem NQN is statically defined in the configuration file we need to use
different ports if we want to connect to the same ip resp. target host. This is
needed if you have multiple NVMe drives on a target host since you typically
will have one subsystem per drive (except e.g. when using software RAID in
SPDK). Last time I set this up with SPDK this was only possible if those
subsystems also had different NQNs. This is why we attached the port number to
the NQN.
2) I don't know exactly what you mean with "it demands that it's host NQN be
used". If I remember correctly: when a NVMf client (initiator) connects to a
target it provides the subsystem NQN and its (own) host NQN. Every initiator
typically has its own host NQN. However, you can use the same if you want to
group them. Again according to specification this host NQN can be a randomly
generated UUID in the format described above, which will generate a new NQN
every time you start a Crail NVMf client. Now I do understand that this does
not allow you to put the host NQN in the allowed host list in SPDK since you do
not know the host NQN. So as proposed I will change this to be configurable.
3) All metadata is stored on the Crail namenode except the file and directory
names which are stored in directory files which consist out of blocks which are
stored on the datanodes (You might want to take a look at some of the talks or
papers that explain this in detail). Crail per default is not persistent and
you will lose all metadata information stored on the namenode even if you use
the NVMf datanode. However you can enable namenode logging cf.
[https://incubator-crail.readthedocs.io/en/latest/config.html#logging] note
that this requires the block allocation algorithm to be "roundrobin" (default).
Logging logs all operations to a log and allows to replay the log when starting
the namenode again after being shut down.
Regards,
Jonas
> Using Crail with NVMf, the Default NQN also attaches the port number to the
> name.
> ---------------------------------------------------------------------------------
>
> Key: CRAIL-93
> URL: https://issues.apache.org/jira/browse/CRAIL-93
> Project: Apache Crail
> Issue Type: Bug
> Affects Versions: 1.2
> Environment: I'm running crail nodes with docker containers, on a
> Ubuntu 18.04 base.
> Reporter: David Crespi
> Assignee: Jonas Pfefferle
> Priority: Critical
>
> This is the version I'm actually using: v1.1-2-gf0afadc
> I'm set up to use spdk on the backend to crail. When attempting to attach,
> it appears that crail needs two subsystems to achieve the connection. Instead
> of allowing the default name, I have the system variable set: -e
> NVMF_NQN="nqn.2017-06.io.crail:cnode"
> 1) subsystem NQN: nqn.2017-06.io.crail:cnode
> 2) subsystem NQN: nqn.2017-06.io.crail:cnode4420
> 19/03/01 17:46:45 INFO crail: CrailHadoopFileSystem fs initialization done..
> 19/03/01 17:46:45 INFO crail: Connecting to NVMf target at Transport address
> = /192.168.2.104:4420, subsystem NQN = nqn.2017-06.io.crail:cnode4420
> It appears that the initial connect/discovery of the subsystem uses #1, but
> using the
> crail commands (crail fs -mkdir /test) uses #2.
> Both have to have a valid namespace attached as well.
>
> It also appears that when using my own subsystem NQN (NVMF_NQN) name, crail
> wants to generate its own Host NQN. A new one every time. First, how do you
> learn of that
> NQN, and 2nd, it would be great to disable it if spdk has "allow any hosts"
> set. It refuses
> to connect to spdk.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
