Re: Deploy multi-node Hadoop with Docker

[Ayush Saxena]

Hi Nikos,
I think you are talking about the documentation in the overview
section of the docker image: https://hub.docker.com/r/apache/hadoop

I just wrote that 2-3 Months back particularly for dev purposes not
for any prod use case, you should change those values accordingly. The
docker-compose file I copied from
https://github.com/apache/hadoop/blob/docker-hadoop-3/docker-compose.yaml

-Ayush

On Fri, 22 Sept 2023 at 22:28, Nikos Spanos  wrote:
>
> Hi,
>
>
>
> I am creating a multi-node Hadoop cluster for a personal project, and I would 
> like to use the official docker image (apache/hadoop).
>
>
>
> However, looking at the official docker image documentation and the 
> docker-compose file I have seen the following environment variable:
>
>
>
> environment:
>
>   ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
>
>
>
> I would like to know if it is safe to create the namenode directory in the 
> /tmp folder since this kind of folder is neither secure nor data persistent. 
> Thus, I would like to understand which path is the best practice for this. 
> Moreover, which are other environment variables I could use of.
>
>
>
> Thanks a lot, in advance.
>
>
>
> Kind regards,
>
>
>
> Nikos Spanos
>
>
>
> M.Sc Business Analytics & Big Data| Athens University of Economics & Business
>
> Phone Number: +306982310494
>
> Linkedin profile
>
>

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Re: Deploy multi-node Hadoop with Docker

[Wei-Chiu Chuang]

The Hadoop's docker image is not for production use. That's why

But we should update that if people are thinking to use it for production.
Not familiar with docker compose but contributions welcomed:
https://github.com/apache/hadoop/blob/docker-hadoop-3/docker-compose.yaml

On Fri, Sep 22, 2023 at 5:44 AM Nikos Spanos 
wrote:

> Hi,
>
>
>
> I am creating a multi-node Hadoop cluster for a personal project, and I
> would like to use the official docker image (apache/hadoop
> ).
>
>
>
> However, looking at the official docker image documentation and the
> docker-compose file I have seen the following environment variable:
>
>
>
> environment:
>
>   ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
>
>
>
> I would like to know if it is safe to create the namenode directory in the
> /tmp folder since this kind of folder is neither secure nor data
> persistent. Thus, I would like to understand which path is the best
> practice for this. Moreover, which are other environment variables I could
> use of.
>
>
>
> Thanks a lot, in advance.
>
>
>
> Kind regards,
>
>
>
> *Nikos Spanos*
>
>
>
> M.Sc Business Analytics & Big Data| Athens University of Economics &
> Business
>
> Phone Number: +306982310494
>
> Linkedin profile  
>
>
>

Deploy multi-node Hadoop with Docker

[Nikos Spanos]

Hi,

I am creating a multi-node Hadoop cluster for a personal project, and I would 
like to use the official docker image 
(apache/hadoop).

However, looking at the official docker image documentation and the 
docker-compose file I have seen the following environment variable:

environment:
  ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"

I would like to know if it is safe to create the namenode directory in the /tmp 
folder since this kind of folder is neither secure nor data persistent. Thus, I 
would like to understand which path is the best practice for this. Moreover, 
which are other environment variables I could use of.

Thanks a lot, in advance.

Kind regards,

Nikos Spanos

M.Sc Business Analytics & Big Data| Athens University of Economics & Business
Phone Number: +306982310494
Linkedin profile