[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112098#comment-16112098
]
Jaeboo Jeong commented on YARN-6494:
If we use white list for all kinds of mounting, the cluster administrator has
to be concerned with additional settings for the docker environment. And there
is no way to know what configuration is missed.
I'm not sure it is fine, because after we finished the cluster setup, we expect
all the containers in the cluster to run in the same environment.
However, I think it is safe for the cluster administrator to be more careful
because administrator does not know which applications are running. I think it
would be better to document this later.
So I agree to close this issue.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111200#comment-16111200
]
Eric Badger commented on YARN-6494:
---
[~Jaeboo], do you agree with using YARN-6919 and adding the short-circuit
socket that way or do you still have reservations about that? If you agree, we
can close this out as a dup of that JIRA
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109066#comment-16109066
]
Eric Badger commented on YARN-6494:
---
bq. It seems there are several use cases where a default bind list would be
useful, so I'd say let's pursue that route.
Filed YARN-6919
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109065#comment-16109065
]
Shane Kumpf commented on YARN-6494:
---
[~ebadger] thanks for the feedback and I agree with your points. With
administrator involvement, we reduce the chance of surprise, which I believe is
important.
It seems there are several use cases where a default bind list would be useful,
so I'd say let's pursue that route.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108978#comment-16108978
]
Eric Badger commented on YARN-6494:
---
bq. At a minimum, we'd need the ability to turn this off for containers that
don't need the HDFS socket, but I feel that it would be better to have a more
holistic approach, which is what I hope YARN-5534 can become. The reason I
believe we shouldn't hard code mounts is that not every container will require
that mount.
I generally agree with this approach, but adding the volume whitelist doesn't
fix the potential security issue. It only sort of mitigates it. If the
administrator allows for the socket to be in the whitelist, then any container
can ask for it. So yes, existing containers and/or containers that the attacker
does not control upon startup will not have this bind-mounted in. However, if
the attacker is the one submitting the job, they'll just ask for the socket to
be bind-mounted and will be granted that request. Basically what I'm trying to
get at is that if the administrator allows short-circuit reads, they are taking
the potential security risk. At that point, I'm not sure if it matters whether
all containers have the socket or just the ones that asked for it, especially
when the attacker can explicitly ask for it.
I think both points can be resolved by letting the administrator decide their
destiny here. We can use YARN-5534 to create a whitelist of volumes that the
jobs can specify. Then, as we touched upon in [this
comment|https://issues.apache.org/jira/browse/YARN-5534?focusedCommentId=16093026=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16093026]
in YARN-5534, we can create a default bind list, which is empty by default. If
the administrator wants the short-circuit socket for all containers, they can
add it to the default list. If they only want it for certain containers, they
can add it to the whitelist and let users ask for it.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107631#comment-16107631
]
Shane Kumpf commented on YARN-6494:
---
{quote}
However I thought this case is different from ordinary file mounts. I think
that the path for the short circuit is the configuration of the cluster, and
the cluster user doesn't need to consider about that path. I just think it's
just part of the cluster setup.
{quote}
Thanks for the explanation [~Jaeboo]. I do understand your point, however I'm
still not sure I agree with the idea of hard coded/automatic bind mounts in
code. IMO we need to be very, very careful about what we mount from the host
and an opt-in model puts the responsibility on administrators to at least
consider the impact of adding the mount to the whitelist. At a minimum, we'd
need the ability to turn this off for containers that don't need the HDFS
socket, but I feel that it would be better to have a more holistic approach,
which is what I hope YARN-5534 can become. The reason I believe we shouldn't
hard code mounts is that not every container will require that mount. Let's say
I'm running web servers in containers on YARN that don't use or need HDFS,
having this socket mounted exposes HDFS when it is not needed. It's also
unclear to me if a malicious user could leverage this socket to exploit the
system. I hope that helps explain my hesitation here.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107424#comment-16107424
]
Jaeboo Jeong commented on YARN-6494:
Thank you for the feedbacks.
[~dan...@cloudera.com] Current patch was referenced another test cases in
TestDockerContainerRuntime.
DockerLinuxContainerRuntime write the docker command to a temporary file, so I
think it's not bad to compare command strings with the contents of the
temporary file. Of course, it is true that it is difficult to memorize the
order of commands and make sure that they are correct.
[~shaneku...@gmail.com] I agree with the white list for safe volume mounts.
However I thought this case is different from ordinary file mounts. I think
that the path for the short circuit is the configuration of the cluster, and
the cluster user doesn't need to consider about that path. I just think it's
just part of the cluster setup.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103113#comment-16103113
]
Shane Kumpf commented on YARN-6494:
---
Thanks for the patch [~Jaeboo]! I do have a concern as this automatically bind
mounts the dn socket in every container if short circuit reads are enabled. I
still feel this should be handled via YARN-5534 (or the "auto bind mount list"
approach, if that is pursued). Additionally, in a traditional HDFS cluster, we
only have one DataNode process per server, is it safe to have multiple DNs on a
single server interacting with this socket?
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102276#comment-16102276
]
Daniel Templeton commented on YARN-6494:
Thank you for fixing this issue! Looks like patch #2 failed to apply. Aside
from that it looks good.
I have a couple of comments, though. It would be nice to add messages to the
asserts in the test. Also is it safe to assume the order of the volume mounts
in the test? (I don't recall and didn't look it up.) When Jenkins does run, I
expect there will also be some line length issues.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102258#comment-16102258
]
Hadoop QA commented on YARN-6494:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} YARN-6494 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6494 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864527/YARN-6494.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16562/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102248#comment-16102248
]
Eric Badger commented on YARN-6494:
---
Moved to subtask of YARN-3611
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102242#comment-16102242
]
Eric Badger commented on YARN-6494:
---
This looks like a good addition to me. [~shaneku...@gmail.com], [~templedf],
could you take a look?
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978913#comment-15978913
]
Jaeboo Jeong commented on YARN-6494:
On branch-3.0.0-alpha2, it compiled well.
But on trunk branch, there is a compile error because of HDFS-11596.
I made a new patch.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977704#comment-15977704
]
Eric Badger commented on YARN-6494:
---
I'm getting a compilation error. Looks like you're missing a dependency for
hadoop-hdfs-client in the pom.
{noformat}
+
+ org.apache.hadoop
+ hadoop-hdfs-client
+
{noformat}
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
