OK, I take it back. Having trouble with crail fs. =====
[root@abbey-road ~]# $CRAIL_HOME/bin/crail fs -mkdir /foobar 20/02/10 10:07:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ... 20/02/10 10:07:16 INFO crail: connected to namenode(s) abbey-road/ 10.114.222.23:9060 20/02/10 10:07:16 INFO crail: CrailHadoopFileSystem fs initialization done.. mkdir: java.util.concurrent.ExecutionException: java.io.IOException: Map failed 20/02/10 10:07:16 INFO crail: Closing CrailHadoopFileSystem 20/02/10 10:07:16 INFO crail: Closing CrailFS singleton 20/02/10 10:07:16 INFO crail: mapped client cache closed [root@abbey-road ~]# $CRAIL_HOME/bin/crail fs -rmdir /foobar 20/02/10 10:08:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ...20/02/10 10:08:55 INFO crail: CrailHadoopFileSystem fs initialization done.. rmdir: /foobar 20/02/10 10:08:55 INFO crail: Closing CrailHadoopFileSystem 20/02/10 10:08:55 INFO crail: Closing CrailFS singleton 20/02/10 10:08:55 INFO crail: mapped client cache closed [root@abbey-road ~]# $CRAIL_HOME/bin/crail fs -mkdir /foobar 20/02/10 10:09:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ... 20/02/10 10:09:43 INFO crail: CrailHadoopFileSystem fs initialization done.. mkdir: `/foobar': File exists 20/02/10 10:09:43 INFO crail: Closing CrailHadoopFileSystem 20/02/10 10:09:43 INFO crail: Closing CrailFS singleton 20/02/10 10:09:43 INFO crail: mapped client cache closed ===== Seems that mkdir works, but rmdir does not? Also, IOException was reported on mkdir, which seems worrisome. Lou. On Mon, Feb 10, 2020 at 8:16 AM Lou DeGenaro <[email protected]> wrote: > David, > > Thanks. I'm able to use *crail fs *successfully to create/list/delete. > Will try to increase available storage space next... > > Lou. > > On Fri, Feb 7, 2020 at 6:37 PM David Crespi < > [email protected]> wrote: > >> Oh, and it may have already written something on the datastore, so you’d >> need >> to check it and remove it with the fs command anyway. >> >> crail fs -ls -R / >> >> would show you what you have there. >> >> Regards, >> David >> >> (C) 714-476-2692 >> >> From: Lou DeGenaro<mailto:[email protected]> >> Sent: Friday, February 7, 2020 12:15 PM >> To: [email protected]<mailto:[email protected]> >> Subject: iobench >> >> Still a noob. Got namenode and datanode running as docker images on my >> VM. I'm looking for the simplest example of writing something then >> reading >> something. >> >> [root@abbey-road conf]# $CRAIL_HOME/bin/crail iobench -t write -f >> /filename >> -s 1024 -k 1 >> 20/02/07 14:10:22 INFO crail: creating singleton crail file system >> 20/02/07 14:10:23 INFO crail: crail.version 3101 >> 20/02/07 14:10:23 INFO crail: crail.directorydepth 16 >> 20/02/07 14:10:23 INFO crail: crail.tokenexpiration 10 >> 20/02/07 14:10:23 INFO crail: crail.blocksize 1048576 >> 20/02/07 14:10:23 INFO crail: crail.cachelimit 1073741824 >> 20/02/07 14:10:23 INFO crail: crail.cachepath /dev/hugepages/cache >> 20/02/07 14:10:23 INFO crail: crail.user crail >> 20/02/07 14:10:23 INFO crail: crail.shadowreplication 1 >> 20/02/07 14:10:23 INFO crail: crail.debug false >> 20/02/07 14:10:23 INFO crail: crail.statistics true >> 20/02/07 14:10:23 INFO crail: crail.rpctimeout 1000 >> 20/02/07 14:10:23 INFO crail: crail.datatimeout 1000 >> 20/02/07 14:10:23 INFO crail: crail.buffersize 1048576 >> 20/02/07 14:10:23 INFO crail: crail.slicesize 524288 >> 20/02/07 14:10:23 INFO crail: crail.singleton true >> 20/02/07 14:10:23 INFO crail: crail.regionsize 1073741824 >> 20/02/07 14:10:23 INFO crail: crail.directoryrecord 512 >> 20/02/07 14:10:23 INFO crail: crail.directoryrandomize true >> 20/02/07 14:10:23 INFO crail: crail.cacheimpl >> org.apache.crail.memory.MappedBufferCache >> 20/02/07 14:10:23 INFO crail: crail.locationmap >> 20/02/07 14:10:23 INFO crail: crail.namenode.address >> crail://abbey-road:9060 >> 20/02/07 14:10:23 INFO crail: crail.namenode.blockselection roundrobin >> 20/02/07 14:10:23 INFO crail: crail.namenode.fileblocks 16 >> 20/02/07 14:10:23 INFO crail: crail.namenode.rpctype >> org.apache.crail.namenode.rpc.tcp.TcpNameNode >> 20/02/07 14:10:23 INFO crail: crail.namenode.log >> 20/02/07 14:10:23 INFO crail: crail.storage.types >> org.apache.crail.storage.tcp.TcpStorageTier >> 20/02/07 14:10:23 INFO crail: crail.storage.classes 1 >> 20/02/07 14:10:23 INFO crail: crail.storage.rootclass 0 >> 20/02/07 14:10:23 INFO crail: crail.storage.keepalive 2 >> 20/02/07 14:10:23 INFO crail: buffer cache, allocationCount 1, bufferCount >> 1024 >> 20/02/07 14:10:23 INFO narpc: new NaRPC client group v1.5.0, queueDepth >> 16, >> messageSize 2097152, nodealy false >> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.interface eth0 >> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.port 50020 >> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.storagelimit 1073741824 >> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.allocationsize 1073741824 >> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.datapath >> /dev/hugepages/data >> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.queuedepth 16 >> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.cores 1 >> 20/02/07 14:10:23 INFO narpc: new NaRPC client group v1.5.0, queueDepth >> 32, >> messageSize 512, nodealy true >> 20/02/07 14:10:23 INFO crail: crail.namenode.tcp.queueDepth 32 >> 20/02/07 14:10:23 INFO crail: crail.namenode.tcp.messageSize 512 >> 20/02/07 14:10:23 INFO crail: crail.namenode.tcp.cores 1 >> 20/02/07 14:10:23 INFO crail: connected to namenode(s) abbey-road/ >> 10.114.222.23:9060 >> write, filename /filename, size 1024, loop 1, storageClass 0, >> locationClass >> 0, buffered true >> Exception in thread "main" java.io.IOException: Map failed >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938) >> at >> >> org.apache.crail.memory.MappedBufferCache.allocateRegion(MappedBufferCache.java:94) >> at org.apache.crail.memory.BufferCache.allocateBuffer(BufferCache.java:95) >> at >> org.apache.crail.core.CoreDataStore.allocateBuffer(CoreDataStore.java:482) >> at org.apache.crail.tools.CrailBenchmark.write(CrailBenchmark.java:87) >> at org.apache.crail.tools.CrailBenchmark.main(CrailBenchmark.java:1070) >> Caused by: java.lang.OutOfMemoryError: Map failed >> at sun.nio.ch.FileChannelImpl.map0(Native Method) >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935) >> ... 5 more >> >>
