[jira] [Commented] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15807883#comment-15807883
 ] 

Remi Catherinot commented on YARN-5714:
---

Thanks for your feedbacks.

No problem for me if the patch is not used in preference to a LinkedHashMap 
solution (even after 6 rounds :) ).

If the control over the env order is to be left to the client (and so, to any 
external lib that relies on yarn) and enforce the use of a LinkedHashMap in any 
Yarn entry point I think such kind of patch level is beyond my current 
knowledge of yarn's internal code and protobuf.

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
>  Labels: oct16-medium
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch, YARN-5714.005.patch, 
> YARN-5714.006.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remi Catherinot updated YARN-5714:
--
Attachment: YARN-5714.006.patch

fixed checkstyle issue (one unused import in the test class)

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
>  Labels: oct16-medium
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch, YARN-5714.005.patch, 
> YARN-5714.006.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remi Catherinot updated YARN-5714:
--
Attachment: YARN-5714.005.patch

Changed deps ordering algorithm + fix %% case for windows. Adjusted tests for 
both changes

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
>  Labels: oct16-medium
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch, YARN-5714.005.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753981#comment-15753981
 ] 

Remi Catherinot commented on YARN-5714:
---

Hi, thanks for the review too :)
# are you sure about the double % ? i've tested it on windows 7, 8, 8.1 and 10 
and double-ing the % does not escape it.
{noformat}
set A=toto
set B=%A%
set C=%%A%%
echo %A% %B% %C%
{noformat}
produce _toto %toto% %toto%_, so i always got _A_'s value, so both _B_ and _C_ 
depend on _A_.
# I still prefer to keep it the way it is because it is the last line of the 
while that does this test. Personnally when i read code that double check 
things I tend to think the code is still under dev and not yet production-ready 
and then i start to tripple check everything i read. So for me, such double 
check would lead in code maintenance/review time increase. if this is really a 
blocking issue and does not permit the patch to be accepted, then i'll change 
it.
# I've reworked the code to match you algorithm so the comment (and its tipo) 
no longer exist
# I've tested it with deps in reverse order and having each env entry depending 
on each of its successors and with 100 entries. On my computer it returns in 0 
or 16ms (no lesser time precision). My algorithm start to be slower in a 
perceptible way with the same test case when we have more than 200 entries. it 
hits the second for computation when i have more than ~700 entries. So I've 
implemented the new algorithm (more or less, i use recursion to handle the 
stack stuff) but i need to rework the tests too because both algorithm do not 
exactly order the same way, even if both of them do the job. I'm currently on 
it (just to let you know).

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
>  Labels: oct16-medium
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744799#comment-15744799
 ] 

Remi Catherinot commented on YARN-5714:
---

Hi,

thanks for the review and here are my answers :

1 - echo %%A%% in windows does output A's value and not %%A%%, so %%A%% depends 
on A so it is normal that the patch returns A as being a dependency. i think 
you tried it with a none existing variable. If B does not exist, echo %%B%% 
output is %%B%% but since the patch cannot really assume what will be the 
target environment and the known variables in it I assume that if someone 
through the hadoop configuration use %A% syntaxe it means he really expect A to 
be known at runtime on the container side. So i won't fix because this is not a 
bug.

2 - the last "i++" is the last line of a while loop with its test condition 
being "i won't fix

3 - i don't see any tipo, can you be more explicit + add the line below because 
the sentence is truncated in your code snippet (the next line starts with 
"removed", because the full comment is "... to be removed " but since i 
tried my best to follow code rules i had to wrapp the comment to avoid a long 
line. waiting for more feedback

4 - containers are launched and will runs for several seconds, minute even 
hours. the loop is used only once at the container start. I mean, your 
algorithm may be faster/slower than mine, may consumme more/less memory but 
this woul be less than 1 millisecond difference and less than a few bytes. Is 
it really an issue ? i don't think so. may be better than rigth now but it does 
the job with acceptable performance (CPU & memory) => won't fix

5 - i've considered using regular expression, but i can't see a way of handling 
with simple quote/double quote/slashes and other special bash syntaxes with 
regex. regular expression may be used in the windows case but again, i'm no 
windows expert so i prefer to keep the patch and its parsing the way it is now 
for a first release and leave it to people with better skill/knowledge the 
tweaking in maybe a 2nd release. Plus by experience, i've seen regex that 
looked ok, runned well, but on specific strings going to burn CPU as hell for 
several minutes and no match while it is usually returning in a few ms on most 
of the input strings. my algorithm just goes forward (i++ only, never i--) and 
has no back reference, so i'm assured each env entry string will be scanned 
char by char only once and each char will be read only once in a timely 
fashion. My conculsion would be : regex can't do the job or it would be even 
more complexe than the actual patch code and may fall into really heavy 
performance issues. max time used to parse + don't see how to do it in a single 
regex => won't fix

to conclude : i stay with patch 4

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
>  Labels: oct16-medium
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To 

[jira] [Commented] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589585#comment-15589585
 ] 

Remi Catherinot commented on YARN-5714:
---

I think the patch is ready for a review.

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563058#comment-15563058
 ] 

Remi Catherinot edited comment on YARN-5714 at 10/10/16 7:32 PM:
-

HOMELOC depends on both HADOOP_MAPRED_HOME and HADOOP_COMMON_HOME. This is more 
or less the test done line 1560 in TestContainerLaunch when i'm testing that 
$\{A:-$B\} extract A and B as its dependencies. Other tests take care of 
testing $var, $\{var\} and $\{#var\} syntaxes, so I limited the test of a var 
in a var to just the $B and not the $\{B\} one.

I don't parse using simplier indexOf( "$" ) because it would miss escapes by 
backslashes and single quote protections.

It is also a best effort dependency parsing, which means it it fails to see the 
dependencies, we end up with a "random" ordered map which is what we have 
without the patch.

The parser in fact does not see that B is used inside A. It just sees A and B 
which is enough for dependency checking. Parsers to evaluate $\{A:-$B\} would 
require to see that B is inside A are harder to code and would certainly need a 
grammar and generated code, which would be overkill for such a need.


was (Author: rcatherinot):
HOMELOC depends on both HADOOP_MAPRED_HOME and HADOOP_COMMON_HOME. This is more 
or less the test done line 1560 in TestContainerLaunch when i'm testing that 
${A:-$B} extract A and B as dependencies. Other tests take care of testing $var 
${var} and ${#var} syntaxes, so i limited the test of a var in a var to just $B 
and not ${B}.

The parser in fact does not see that B is used inside A, it just sees A and B 
which is enought for dependency checking. The parser is designed to evalutate 
which would require to see that B is inside A.

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563058#comment-15563058
 ] 

Remi Catherinot commented on YARN-5714:
---

HOMELOC depends on both HADOOP_MAPRED_HOME and HADOOP_COMMON_HOME. This is more 
or less the test done line 1560 in TestContainerLaunch when i'm testing that 
${A:-$B} extract A and B as dependencies. Other tests take care of testing $var 
${var} and ${#var} syntaxes, so i limited the test of a var in a var to just $B 
and not ${B}.

The parser in fact does not see that B is used inside A, it just sees A and B 
which is enought for dependency checking. The parser is designed to evalutate 
which would require to see that B is inside A.

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remi Catherinot updated YARN-5714:
--
Attachment: YARN-5714.004.patch

missed one last whitespace at EOL

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch, YARN-5714.004.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remi Catherinot updated YARN-5714:
--
Attachment: YARN-5714.003.patch

fixed (again :) ) checkstyle issues

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, 
> YARN-5714.003.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remi Catherinot updated YARN-5714:
--
Attachment: YARN-5714.002.patch

fixed code style issues (tabs, eol whitespace, javadoc, method too long, etc.)

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remi Catherinot updated YARN-5714:
--
Attachment: YARN-5714.001.patch

includes patch + test
one thing is changed and not tested : 
ContainerLaunch.ShellScriptBuilder.create() which now use a versiobn accepting 
a Shell.OSType enum value because i needed it for tests purposes (added the 
@VisibleForTesting annotation).

> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Assignee: Remi Catherinot
>Priority: Trivial
> Attachments: YARN-5714.001.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remi Catherinot updated YARN-5714:
--
Description: 
when dumping the launch container script, environment variables are dumped 
based on the order internally used by the map implementation (hash based). It 
does not take into consideration that some env varibales may refer each other, 
and so that some env variables must be declared before those referencing them.

In my case, i ended up having LD_LIBRARY_PATH which was depending on 
HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a wrong 
value and so native libraries weren't loaded. jobs were running but not at 
their best efficiency. This is just a use case falling into that bug, but i'm 
sure others may happen as well.

I already have a patch running in my production environment, i just estimate to 
5 days for packaging the patch in the right fashion for JIRA + try my best to 
add tests.

Note : the patch is not OS aware with a default empty implementation. I will 
only implement the unix version on a 1st release. I'm not used to windows env 
variables syntax so it will take me more time/research for it.

  was:
when dumping the launch container script, environment variables are dumped 
based on the order internally used by the map implementation (hash based). It 
does not take into consideration that some env varibales may refer each other, 
and so that some env variables must be declared before those referencing them.

In my case, i ended having LD_LIBRARY_PATH which was depending on 
HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a wrong 
value and so native libraries weren't loaded. jobs were running but not at 
their best. This is just a use case falling into that bug, but i'm sure others 
may happen as well.

I already have a patch running in my production environment, i just estimate to 
5 days for packing the path in the right fashion for JIRA + try my best to add 
tests.

Note : the patch is not OS aware with a default empty implementation. I will 
only implement the unix version on a 1st release. I'm not used to windows env 
variables syntax so it will take me more time/research for it.


> ContainerExecutor does not order environment map
> 
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
>Reporter: Remi Catherinot
>Priority: Trivial
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped 
> based on the order internally used by the map implementation (hash based). It 
> does not take into consideration that some env varibales may refer each 
> other, and so that some env variables must be declared before those 
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on 
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a 
> wrong value and so native libraries weren't loaded. jobs were running but not 
> at their best efficiency. This is just a use case falling into that bug, but 
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate 
> to 5 days for packaging the patch in the right fashion for JIRA + try my best 
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will 
> only implement the unix version on a 1st release. I'm not used to windows env 
> variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5714) ContainerExecutor does not order environment map

[Remi Catherinot (JIRA)]

Remi Catherinot created YARN-5714:
-

 Summary: ContainerExecutor does not order environment map
 Key: YARN-5714
 URL: https://issues.apache.org/jira/browse/YARN-5714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 3.0.0-alpha1, 2.6.4, 2.7.3, 2.5.2, 2.4.1
 Environment: all (linux and windows alike)
Reporter: Remi Catherinot
Priority: Trivial


when dumping the launch container script, environment variables are dumped 
based on the order internally used by the map implementation (hash based). It 
does not take into consideration that some env varibales may refer each other, 
and so that some env variables must be declared before those referencing them.

In my case, i ended having LD_LIBRARY_PATH which was depending on 
HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a wrong 
value and so native libraries weren't loaded. jobs were running but not at 
their best. This is just a use case falling into that bug, but i'm sure others 
may happen as well.

I already have a patch running in my production environment, i just estimate to 
5 days for packing the path in the right fashion for JIRA + try my best to add 
tests.

Note : the patch is not OS aware with a default empty implementation. I will 
only implement the unix version on a 1st release. I'm not used to windows env 
variables syntax so it will take me more time/research for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org