I’m a bit confused now. If it only ever handles 1 namespace, then multiple
datanodes are useless.
Or are you saying only 1 namespace for type-1? If you’re not saying only one
for type-1, how do you
mix types if limited to 1 namespace?
Regards,
David
________________________________
From: Jonas Pfefferle <[email protected]>
Sent: Wednesday, June 26, 2019 12:25:38 AM
To: [email protected]; David Crespi
Subject: Re: Question on crail type-1 with terasort shuffle data
I forgot to add: Namespace has to be 1 for the moment. This is a limitation
in how the namenode handles storage tier metadata (essentially it only
stores ip/port). You can set it on the server side however it will not be
respected by the client, i.e. the client will always use namespace id 1.
Regards,
Jonas
On Wed, 26 Jun 2019 09:17:40 +0200
"Jonas Pfefferle" <[email protected]> wrote:
> Hi David,
>
>
> This is currently a limitation of the NVMf storage tier. You need to
>use the same NQN on all datanodes. The port can be different however.
>I will clarify this in the documentation.
>
> Regards,
> Jonas
>
> On Tue, 25 Jun 2019 19:43:04 +0000
> David Crespi <[email protected]> wrote:
>> It occurred to me that perhaps since I’m using the NVMf, that the
>>namenode can only handle one
>>
>> Subsystem. And for the type-1, these resources would need to be
>>added under the same subsystem,
>>
>> but as different namespace’s. Is this correct? I did try this and
>>it appears to be working. If this IS the
>>
>> correct way of utilizing these, how do I know which namespace(s) are
>>being accessed? I don’t see
>>
>> anything in the logs that identify which is being addressed.
>>
>>
>> Regards,
>>
>>
>> David
>>
>>
>> ________________________________
>>From: David Crespi <[email protected]>
>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>> To: [email protected]
>> Subject: Question on crail type-1 with terasort shuffle data
>>
>> I’m starting to run terasort now with the shuffle data going to
>>crail type-1.
>> I’ve got the type-2 set to nqn:
>>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>> which appears to be working okay for the result data. When I
>>introduce type-1 into my config,
>> it looks like the namenode gets confused and picks the nqn of the
>>type-2 crailstore, instead of
>> the one it is assigned (bolded in red below).
>>
>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>then 5 type-1 added to the
>> spark worker nodes (192.168.3.101-105). All are running the same
>>image as the namenode,
>> which is the spark master.
>>
>> Is there some additional setting I need to include for this?
>> These are the variables that get set for each container (added via
>>environment)
>> # NVMf storage
>> crail.storage.nvmf.ip $NVMF_IP
>> crail.storage.nvmf.port $NVMF_PORT
>> crail.storage.nvmf.nqn $NVMF_NQN
>> crail.storage.nvmf.hostnqn $NVMF_HOSTNQN
>> crail.storage.nvmf.namespace $NVMF_NAMESPACE
>>
>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>> 19/06/25 10:46:43 INFO crail: crail.user crail
>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>> 19/06/25 10:46:43 INFO crail: crail.debug true
>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>org.apache.crail.memory.MappedBufferCache
>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>crail://192.168.1.164:9060
>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>roundrobin
>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>org.apache.crail.namenode.rpc.tcp.TcpNameNode
>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>org.apache.crail.storage.nvmf.NvmfStorageTier
>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>bufferCount 1024
>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>1073741824
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>queueDepth 32, messageSize 512, nodealy true
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>/192.168.1.164:9060
>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>streamId 1, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>address = /192.168.3.100:4420, subsystem NQN =
>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>> 19/06/25 10:46:43 INFO disni: creating RdmaProvider of type 'nat'
>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>size 28, native size 16
>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>native size 32
>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>72, native size 128
>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>native size 48
>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>native size 16
>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>40, native size 40
>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>native size 48
>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>139964194703408
>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>64
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>send_wr size 32, recv_wr_size 32
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type +
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>128
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>send_wr size 64, recv_wr_size 64
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type +
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>/192.168.3.100:4420, fsId 0, cache size 1
>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>success
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>streamId 1, capacity 262656
>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>DIRECTORY, storageAffinity 0, locationAffinity 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>streamId 2, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>fsId 0
>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>4, token 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>streamId 2, capacity 524800
>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>type DIRECTORY, storageAffinity 0, locationAffinity 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>fd 4, streamId 3, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>address = /192.168.3.104:4420, subsystem NQN =
>>nqn.2018-12.com.StorEdgeSystems:cntlr13 (<-- should be
>>nqn.2018-12.com.StorEdgeSystems:worker-4)
>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>139964195131168
>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>/192.168.3.104:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>64
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>send_wr size 32, recv_wr_size 32
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type +
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
>>/192.168.3.104:4420
>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>The host is not allowed to establish an association to any controller
>>in the NVM subsystem or the host is not allowed to establish an
>>association to the specified controller., CID: 0, Do_not_retry:
>>false, More: false, SQHD: 0}
>> at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>> at
>>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>
>>
>> Regards,
>>
>> David
>>
>
