That looks like it is indeed working. If the datanode is providing that info,
you’re running.
That’s set up then to use the VM’s memory as temporary storage, but it’s very
small, so
don’t overrun it… it dies a horrible death 😊
I guess IObench would be okay, never tried it. You don’t use crail, however,
like regular storage. It’s main use is for temporary data, and it’s ephemeral,
since the namenode is
always in memory… i.e. no persistence. It’s designed to be used with modern
applications that tend to retry when something goes wrong… so the data, in
these cases, doesn’t have
to be persistent.
Regards,
David
From: Lou DeGenaro<mailto:[email protected]>
Sent: Friday, February 7, 2020 7:13 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: docker
I have a VM and 2 terminals.
terminal1:
[root@abbey-road ~]# docker run -it --network host -e
NAMENODE_HOST=localhost -e INTERFACE=eth5 apache/incubator-crail:1.2
namenode
20/02/07 14:27:04 INFO crail: initalizing namenode
20/02/07 14:27:04 INFO crail: crail.version 3101
...
terminal2:
[root@abbey-road ~]# docker run -it --network host -e
NAMENODE_HOST=abbey-road apache/incubator-crail:1.2 datanode
20/02/07 14:48:44 INFO crail: crail.version 3101
...
20/02/07 14:48:44 INFO crail: connected to namenode(s) abbey-road/
10.114.222.23:9060
20/02/07 14:48:45 INFO crail: datanode statistics, freeBlocks 1024
20/02/07 14:48:45 INFO crail: datanode statistics, freeBlocks 1024
...
So I propose a terminal 3 that has an app such as iobench. I'm missing
something?
Lou.
On Fri, Feb 7, 2020 at 10:06 AM David Crespi <
[email protected]> wrote:
> Not quite sure I understand your question. You can go into either node
> and use
> the crail command line to see if it’s actually working. They have
> borrowed the
> HDFS command line for this.
>
> Example: /crail/bin/crail fs -ls -R /
>
> If you just do “/crail/bin/crail fs” you will get help.
>
> This will show you what’s in crailstore. Not sure how much you know about
> crail,
> so it would make sense for you to read their documentation closely. There
> is quite
> a bit, however, of inaccurate info there, as they haven’t been keeping up
> with keeping
> up. Crail is temporary storage and can be used with memory or a storage
> device
> (disk, ssd etc).
>
> If you’re looking to drive crail with something, you can attach it to
> spark, for example.
>
> Regards,
> David
>
> From: Lou DeGenaro<mailto:[email protected]>
> Sent: Friday, February 7, 2020 6:58 AM
> To: [email protected]<mailto:[email protected]>
> Subject: Re: docker
>
> Thanks, that helps!
>
> I've got docker namenode and datanode running on same VM and they seem to
> be working. There does not seem to be a docker image for an app, though.
> I guess I'll try to download and run the iobench app...and maybe try to
> turn it into a docker image?
>
> Lou.
>
> On Fri, Feb 7, 2020 at 9:20 AM David Crespi <
> [email protected]> wrote:
>
> > Excellent and sorry, should have been more specific. The docker hub
> > website for apache-/incubator-crail.
> > https://hub.docker.com/r/apache/incubator-crail/tags
> >
> >
> > Regards,
> > David
> >
> >
> > From: Lou DeGenaro<mailto:[email protected]>
> > Sent: Friday, February 7, 2020 6:18 AM
> > To: [email protected]<mailto:[email protected]>
> > Subject: Re: docker
> >
> > Not seeing where the tags tab is, but your suggestion worked!
> >
> > [root@abbey-road ~]# docker pull apache/incubator-crail:1.2
> > 1.2: Pulling from apache/incubator-crail
> > 2746a4a261c9: Pull complete
> > 4c1d20cdee96: Pull complete
> > 0d3160e1d0de: Pull complete
> > c8e37668deea: Pull complete
> > 197ed90b6875: Pull complete
> > 20fa8d874ecf: Pull complete
> > 526ce669d291: Pull complete
> > 97008d23d6f3: Pull complete
> > 82ca1025c990: Pull complete
> > 3d908580a7c7: Pull complete
> > 1b675fc8e8fe: Pull complete
> > bb57ee7ee007: Pull complete
> > Digest:
> > sha256:e235a6d8bbb01f1f680822c51a9f9f598620c5b801355d52d3f6b1cb6f4551d1
> > Status: Downloaded newer image for apache/incubator-crail:1.2
> >
> > Lou.
> >
> > On Fri, Feb 7, 2020 at 9:11 AM David Crespi <
> > [email protected]> wrote:
> >
> > > Lou,
> > > If you look under the tags tab of the website, you’d see that they are
> > > tagging the versions.
> > > What you want is: docker pull apache/incubator-crail:1.2
> > >
> > > That will pull down the latest version of crail.
> > >
> > > Regards,
> > > David
> > >
> > > From: Lou DeGenaro<mailto:[email protected]>
> > > Sent: Friday, February 7, 2020 6:03 AM
> > > To: [email protected]<mailto:[email protected]>
> > > Subject: Re: docker
> > >
> > > David,
> > >
> > > Thanks, but still no joy. Something I need to configure to make docker
> > > happy?
> > >
> > > =====
> > >
> > > [root@abbey-road ~]# docker pull apache/incubator-crail
> > > Using default tag: latest
> > > Error response from daemon: manifest for apache/incubator-crail:latest
> > not
> > > found
> > >
> > > =====
> > >
> > > I can run tomcat:
> > >
> > > [root@abbey-road ~]# docker run -it --rm tomcat:9.0
> > > Unable to find image 'tomcat:9.0' locally
> > > 9.0: Pulling from library/tomcat
> > > dc65f448a2e2: Pull complete
> > > 346ffb2b67d7: Pull complete
> > > dea4ecac934f: Pull complete
> > > 8ac92ddf84b3: Pull complete
> > > d8ef64070a18: Pull complete
> > > 2a5539a27f97: Pull complete
> > > 9732d0b27efa: Pull complete
> > > 9203e26c4e49: Pull complete
> > > 18c676ee5ea4: Pull complete
> > > 2fbe51cba072: Pull complete
> > > Digest:
> > > sha256:cba009c0ef8cec83df8178cf4f4668170bdb02440e4c7c576fc397e477c715d3
> > > Status: Downloaded newer image for tomcat:9.0
> > > Using CATALINA_BASE: /usr/local/tomcat
> > > Using CATALINA_HOME: /usr/local/tomcat
> > > Using CATALINA_TMPDIR: /usr/local/tomcat/temp
> > > Using JRE_HOME: /usr/local/openjdk-11
> > > Using CLASSPATH:
> > >
> /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
> > > NOTE: Picked up JDK_JAVA_OPTIONS:
> > > --add-opens=java.base/java.lang=ALL-UNNAMED
> > > --add-opens=java.base/java.io=ALL-UNNAMED
> > > --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
> > > 07-Feb-2020 13:56:25.718 INFO [main]
> > > org.apache.catalina.startup.VersionLoggerListener.log Server version
> > name:
> > > Apache Tomcat/9.0.30
> > > 07-Feb-2020 13:56:25.725 INFO [main]
> > > org.apache.catalina.startup.VersionLoggerListener.log Server built:
> > > Dec 7 2019 16:42:04 UTC
> > > 07-Feb-2020 13:56:25.725 INFO [main]
> > > org.apache.catalina.startup.VersionLoggerListener.log Server version
> > > number: 9.0.30.0
> > > 07-Feb-2020 13:56:25.726 INFO [main]
> > > org.apache.catalina.startup.VersionLoggerListener.log OS Name:
> > > Linux
> > >
> > > On Thu, Feb 6, 2020 at 4:10 PM David Crespi <
> > > [email protected]> wrote:
> > >
> > > > Lou,
> > > > The repo is incorrect. You should be pulling from
> > > “apache/incubator-crail”
> > > > Here’s their docker image:
> > > https://hub.docker.com/r/apache/incubator-crail
> > > >
> > > > Regards,
> > > > David
> > > >
> > > >
> > > > From: Lou DeGenaro<mailto:[email protected]>
> > > > Sent: Thursday, February 6, 2020 11:20 AM
> > > > To: [email protected]<mailto:[email protected]>
> > > > Subject: docker
> > > >
> > > > I'm a noob to crail and not much better with docker, however I am
> able
> > > to:
> > > >
> > > > [root@abbey-road ~]# docker run hello-world
> > > > Unable to find image 'hello-world:latest' locally
> > > > latest: Pulling from library/hello-world
> > > > 1b930d010525: Pull complete
> > > > Digest:
> > > >
> sha256:9572f7cdcee8591948c2963463447a53466950b3fc15a247fcad1917ca215a2f
> > > > Status: Downloaded newer image for hello-world:latest
> > > >
> > > > Hello from Docker!
> > > >
> > > > =====
> > > >
> > > > But run into trouble with crail:
> > > >
> > > > [root@abbey-road ~]# docker run apache/crail
> > > > Unable to find image 'apache/crail:latest' locally
> > > > docker: Error response from daemon: pull access denied for
> > apache/crail,
> > > > repository does not exist or may require 'docker login'.
> > > > See 'docker run --help'.
> > > >
> > > > Advise/pointers appreciated.
> > > >
> > > > Lou.
> > > >
> > > >
> > >
> > >
> >
> >
>
>
