[jira] [Updated] (SLIDER-1115) stdout/stderr files are in agent working directory rather than agent log directory
[ https://issues.apache.org/jira/browse/SLIDER-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1115: --- Attachment: slider-1115-001.patch Please find the fix for this issue attached. > stdout/stderr files are in agent working directory rather than agent log > directory > -- > > Key: SLIDER-1115 > URL: https://issues.apache.org/jira/browse/SLIDER-1115 > Project: Slider > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: thomas liu > Attachments: slider-1115-001.patch > > > In the DockerManager.py, YarnDockerManager.py, windows/system.py and > shell.py, the stdout and stderr files are created in the current working > directory, rather than in the log directory. The files should be in the log > directory. > In shell.py and windows/system.py, the files are created based on the pid > file name, but this doesn't work because when the pid file is specified it > has an absolute path. The file names could just be changed to > application.log and application.err. > Also in shell.py and windows/system.py, the stdout and stderr files are > opened with 'w' mode, but each container may run multiple Execute commands, > so the files will be overwritten with the latest output. The mode should be > changed to 'w+' so all the output is captured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-1135) Slider app XML configuration merge with existing configuration in package
[ https://issues.apache.org/jira/browse/SLIDER-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308474#comment-15308474 ] Billie Rinaldi commented on SLIDER-1135: [~shanyu], I am trying to understand your use case better. Are you are trying to override the Storm configuration defaults with a new storm-site.xml file included in your Storm tarball? Could you modify the Storm app package defaults in [configuration/storm-site.xml|https://github.com/apache/incubator-slider/blob/develop/app-packages/storm/configuration/storm-site.xml] instead? > Slider app XML configuration merge with existing configuration in package > - > > Key: SLIDER-1135 > URL: https://issues.apache.org/jira/browse/SLIDER-1135 > Project: Slider > Issue Type: Bug >Reporter: shanyu zhao > Fix For: Slider 0.90.2 > > Attachments: SLIDER-1135.patch > > > Today when you specify a configuration for a Slider app, e.g. in app_config > you have: "site.storm-site.drpc.port": "1234", and in metainfo.xml you > specify , this would overwrite the existing storm-site.xml in the > package. This forces you to set all storm configurations in app_config file. > What we can do is to merge the provided configuration with existing conf > files (storm-site.xml in this case). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-1134) App config key-value pairs specified on the command line seeps into resources definition
[ https://issues.apache.org/jira/browse/SLIDER-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308286#comment-15308286 ] Billie Rinaldi commented on SLIDER-1134: I think you're probably right. It looks like getCompOptionMap() is being merged into the appConf elsewhere in SliderClient, so this doesn't need to be done in buildConfTree. > App config key-value pairs specified on the command line seeps into resources > definition > > > Key: SLIDER-1134 > URL: https://issues.apache.org/jira/browse/SLIDER-1134 > Project: Slider > Issue Type: Bug > Components: client >Affects Versions: Slider 0.90.2 >Reporter: Gour Saha >Assignee: Gour Saha > > When app config key-value pairs specified on the command line using > _*--compopt*_, they seep into resources definition and causes app fail to > start. Only yarn.* keys are allowed in resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-1134) App config key-value pairs specified on the command line seeps into resources definition
[
https://issues.apache.org/jira/browse/SLIDER-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308218#comment-15308218
]
Gour Saha commented on SLIDER-1134:
---
I am trying to understand now why _*getCompOptionMap()*_ is always merged into
_*buildAppOptionsConfTree()*_ and _*buildResourceOptionsConfTree()*_. Seems
like that is the source of the problem. [~billie.rinaldi],
[~ste...@apache.org], any idea why?
{code}
public ConfTree buildAppOptionsConfTree() throws
BadCommandArgumentsException {
return buildConfTree(getOptionsMap());
}
public ConfTree buildResourceOptionsConfTree() throws
BadCommandArgumentsException {
return buildConfTree(getResourceOptionsMap());
}
protected ConfTree buildConfTree(Map optionsMap) throws
BadCommandArgumentsException {
ConfTree confTree = new ConfTree();
ConfTreeOperations ops = new ConfTreeOperations(confTree);
confTree.global.putAll(optionsMap);
Map> roleOptionMap = getCompOptionMap();
for (Map.Entry> entry :
roleOptionMap.entrySet()) {
String key = entry.getKey();
Map value = entry.getValue();
ops.getOrAddComponent(key).putAll(value);
}
return confTree;
}
{code}
> App config key-value pairs specified on the command line seeps into resources
> definition
>
>
> Key: SLIDER-1134
> URL: https://issues.apache.org/jira/browse/SLIDER-1134
> Project: Slider
> Issue Type: Bug
> Components: client
>Affects Versions: Slider 0.90.2
>Reporter: Gour Saha
>Assignee: Gour Saha
>
> When app config key-value pairs specified on the command line using
> _*--compopt*_, they seep into resources definition and causes app fail to
> start. Only yarn.* keys are allowed in resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLIDER-1134) App config key-value pairs specified on the command line seeps into resources definition
[ https://issues.apache.org/jira/browse/SLIDER-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308179#comment-15308179 ] Gour Saha commented on SLIDER-1134: --- Right [~billie.rinaldi], a couple of tests failed too. I am looking into this, will revert if required. > App config key-value pairs specified on the command line seeps into resources > definition > > > Key: SLIDER-1134 > URL: https://issues.apache.org/jira/browse/SLIDER-1134 > Project: Slider > Issue Type: Bug > Components: client >Affects Versions: Slider 0.90.2 >Reporter: Gour Saha >Assignee: Gour Saha > > When app config key-value pairs specified on the command line using > _*--compopt*_, they seep into resources definition and causes app fail to > start. Only yarn.* keys are allowed in resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-1134) App config key-value pairs specified on the command line seeps into resources definition
[ https://issues.apache.org/jira/browse/SLIDER-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308167#comment-15308167 ] Billie Rinaldi commented on SLIDER-1134: [~gsaha], I don't think the resources.merge(cmdLineResourceOptions) line should be removed. That looks like it's pulling in --resopt options. The other change looks good, though. > App config key-value pairs specified on the command line seeps into resources > definition > > > Key: SLIDER-1134 > URL: https://issues.apache.org/jira/browse/SLIDER-1134 > Project: Slider > Issue Type: Bug > Components: client >Affects Versions: Slider 0.90.2 >Reporter: Gour Saha >Assignee: Gour Saha > > When app config key-value pairs specified on the command line using > _*--compopt*_, they seep into resources definition and causes app fail to > start. Only yarn.* keys are allowed in resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Slider-develop - Build # 813 - Still Failing
The Apache Jenkins build system has built Slider-develop (build #813) Status: Still Failing Check console output at https://builds.apache.org/job/Slider-develop/813/ to view the results.
[jira] [Commented] (SLIDER-1107) Generate app configuration files in AM
[ https://issues.apache.org/jira/browse/SLIDER-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307990#comment-15307990 ] ASF subversion and git services commented on SLIDER-1107: - Commit 0732af6d6a2eb9172d0b989f3cbb5522f35add61 in incubator-slider's branch refs/heads/feature/SLIDER-1107_AM_config_generation from [~billie.rina...@gmail.com] [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=0732af6 ] SLIDER-1107 add client install command to hbase-nopkg readme > Generate app configuration files in AM > -- > > Key: SLIDER-1107 > URL: https://issues.apache.org/jira/browse/SLIDER-1107 > Project: Slider > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: Slider 0.91 > > > Currently, each container generates its own application configuration files. > Instead, we could do this in the AM and have YARN localize the configuration > files. Having some basic config generation in the AM may allow us to > simplify the config generation code in the app packages. Also, it would be > much better in the case of Docker containers, where we would prefer not to > have to execute our own code inside the container. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-1107) Generate app configuration files in AM
[ https://issues.apache.org/jira/browse/SLIDER-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307947#comment-15307947 ] ASF subversion and git services commented on SLIDER-1107: - Commit 106d96c711c22fa54c518d28e3dcf1b6dac1c40e in incubator-slider's branch refs/heads/feature/SLIDER-1107_AM_config_generation from [~billie.rina...@gmail.com] [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=106d96c ] SLIDER-1107 add IT for unique component names > Generate app configuration files in AM > -- > > Key: SLIDER-1107 > URL: https://issues.apache.org/jira/browse/SLIDER-1107 > Project: Slider > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: Slider 0.91 > > > Currently, each container generates its own application configuration files. > Instead, we could do this in the AM and have YARN localize the configuration > files. Having some basic config generation in the AM may allow us to > simplify the config generation code in the app packages. Also, it would be > much better in the case of Docker containers, where we would prefer not to > have to execute our own code inside the container. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-1107) Generate app configuration files in AM
[ https://issues.apache.org/jira/browse/SLIDER-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307946#comment-15307946 ] ASF subversion and git services commented on SLIDER-1107: - Commit e2ad57bb128490d4df380673a705806be8f90c76 in incubator-slider's branch refs/heads/feature/SLIDER-1107_AM_config_generation from [~billie.rina...@gmail.com] [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=e2ad57b ] SLIDER-1107 add IT for AM config generation > Generate app configuration files in AM > -- > > Key: SLIDER-1107 > URL: https://issues.apache.org/jira/browse/SLIDER-1107 > Project: Slider > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: Slider 0.91 > > > Currently, each container generates its own application configuration files. > Instead, we could do this in the AM and have YARN localize the configuration > files. Having some basic config generation in the AM may allow us to > simplify the config generation code in the app packages. Also, it would be > much better in the case of Docker containers, where we would prefer not to > have to execute our own code inside the container. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Slider-develop - Build # 812 - Still Failing
The Apache Jenkins build system has built Slider-develop (build #812) Status: Still Failing Check console output at https://builds.apache.org/job/Slider-develop/812/ to view the results.
[jira] [Commented] (SLIDER-1134) App config key-value pairs specified on the command line seeps into resources definition
[ https://issues.apache.org/jira/browse/SLIDER-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307843#comment-15307843 ] ASF subversion and git services commented on SLIDER-1134: - Commit e6999d0c1df987705855406791a46f80e1c3effc in incubator-slider's branch refs/heads/develop from [~gsaha] [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=e6999d0 ] SLIDER-1134 App config key-value pairs specified on the command line seeps into resources definition > App config key-value pairs specified on the command line seeps into resources > definition > > > Key: SLIDER-1134 > URL: https://issues.apache.org/jira/browse/SLIDER-1134 > Project: Slider > Issue Type: Bug > Components: client >Affects Versions: Slider 0.90.2 >Reporter: Gour Saha >Assignee: Gour Saha > > When app config key-value pairs specified on the command line using > _*--compopt*_, they seep into resources definition and causes app fail to > start. Only yarn.* keys are allowed in resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1134) App config key-value pairs specified on the command line seeps into resources definition
[ https://issues.apache.org/jira/browse/SLIDER-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated SLIDER-1134: -- Summary: App config key-value pairs specified on the command line seeps into resources definition (was: App config key-value pairs specified on the command line seeps into resources definition and causes app fail to start) > App config key-value pairs specified on the command line seeps into resources > definition > > > Key: SLIDER-1134 > URL: https://issues.apache.org/jira/browse/SLIDER-1134 > Project: Slider > Issue Type: Bug > Components: client >Affects Versions: Slider 0.90.2 >Reporter: Gour Saha >Assignee: Gour Saha > > When app config key-value pairs specified on the command line using > _*--compopt*_, they seep into resources definition and causes app fail to > start. Only yarn.* keys are allowed in resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
