[jira] [Updated] (SLIDER-1115) stdout/stderr files are in agent working directory rather than agent log directory
[ https://issues.apache.org/jira/browse/SLIDER-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1115: --- Attachment: slider-1115-001.patch Please find the fix for this issue attached. > stdout/stderr files are in agent working directory rather than agent log > directory > -- > > Key: SLIDER-1115 > URL: https://issues.apache.org/jira/browse/SLIDER-1115 > Project: Slider > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: thomas liu > Attachments: slider-1115-001.patch > > > In the DockerManager.py, YarnDockerManager.py, windows/system.py and > shell.py, the stdout and stderr files are created in the current working > directory, rather than in the log directory. The files should be in the log > directory. > In shell.py and windows/system.py, the files are created based on the pid > file name, but this doesn't work because when the pid file is specified it > has an absolute path. The file names could just be changed to > application.log and application.err. > Also in shell.py and windows/system.py, the stdout and stderr files are > opened with 'w' mode, but each container may run multiple Execute commands, > so the files will be overwritten with the latest output. The mode should be > changed to 'w+' so all the output is captured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1118) Making YarnDockerManager retrieve IP and hostname through python module instead of system command
[ https://issues.apache.org/jira/browse/SLIDER-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1118: --- Attachment: SLIDER-1118-002.patch Uploading a patch with logging > Making YarnDockerManager retrieve IP and hostname through python module > instead of system command > - > > Key: SLIDER-1118 > URL: https://issues.apache.org/jira/browse/SLIDER-1118 > Project: Slider > Issue Type: Improvement >Reporter: thomas liu >Assignee: thomas liu > Attachments: SLIDER-1118-001.patch, SLIDER-1118-002.patch > > > Nowadays, some OS in docker containers don't have the system command used in > YarnDockerManager. So we migrate to using Python modules to retrieve IP and > hostname of the container, instead of system command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1118) Making YarnDockerManager retrieve IP and hostname through python module instead of system command
[ https://issues.apache.org/jira/browse/SLIDER-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1118: --- Attachment: SLIDER-1118-001.patch > Making YarnDockerManager retrieve IP and hostname through python module > instead of system command > - > > Key: SLIDER-1118 > URL: https://issues.apache.org/jira/browse/SLIDER-1118 > Project: Slider > Issue Type: Improvement >Reporter: thomas liu >Assignee: thomas liu > Attachments: SLIDER-1118-001.patch > > > Nowadays, some OS in docker containers don't have the system command used in > YarnDockerManager. So we migrate to using Python modules to retrieve IP and > hostname of the container, instead of system command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-1118) Making YarnDockerManager retrieve IP and hostname through python module instead of system command
thomas liu created SLIDER-1118: -- Summary: Making YarnDockerManager retrieve IP and hostname through python module instead of system command Key: SLIDER-1118 URL: https://issues.apache.org/jira/browse/SLIDER-1118 Project: Slider Issue Type: Improvement Reporter: thomas liu Assignee: thomas liu Nowadays, some OS in docker containers don't have the system command used in YarnDockerManager. So we migrate to using Python modules to retrieve IP and hostname of the container, instead of system command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-1112) ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForNoComponents fails in Windows
[ https://issues.apache.org/jira/browse/SLIDER-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu resolved SLIDER-1112. Resolution: Fixed > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForNoComponents > fails in Windows > > > Key: SLIDER-1112 > URL: https://issues.apache.org/jira/browse/SLIDER-1112 > Project: Slider > Issue Type: Improvement > Components: test >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 0.91 > > Attachments: slider-1112-001.patch, slider-1112-002.patch > > > Create a separate createTemplatedSliderApplication utility function and let > testCreateApplicationWithOneAddonPackagesForNoComponents use that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1112) ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForNoComponents fails in Windows
[ https://issues.apache.org/jira/browse/SLIDER-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1112: --- Attachment: slider-1112-002.patch refactoring createTemplatedSliderApplication plan cancelled. decided to provide 'non failing' behavior now with additional optional parameter. uploaded this new patch with the change desired. will close the ticket tracking refactoring createTemplatedSliderApplication > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForNoComponents > fails in Windows > > > Key: SLIDER-1112 > URL: https://issues.apache.org/jira/browse/SLIDER-1112 > Project: Slider > Issue Type: Improvement > Components: test >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 0.91 > > Attachments: slider-1112-001.patch, slider-1112-002.patch > > > Create a separate createTemplatedSliderApplication utility function and let > testCreateApplicationWithOneAddonPackagesForNoComponents use that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1112) Create a separate createTemplatedSliderApplication utility function and let testCreateApplicationWithOneAddonPackagesForNoComponents use that
[ https://issues.apache.org/jira/browse/SLIDER-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1112: --- Attachment: slider-1112-001.patch > Create a separate createTemplatedSliderApplication utility function and let > testCreateApplicationWithOneAddonPackagesForNoComponents use that > - > > Key: SLIDER-1112 > URL: https://issues.apache.org/jira/browse/SLIDER-1112 > Project: Slider > Issue Type: Improvement >Reporter: thomas liu >Assignee: thomas liu > Attachments: slider-1112-001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-1113) refactor overloading createTemplatedSliderApplication functions to reuse code
thomas liu created SLIDER-1113: -- Summary: refactor overloading createTemplatedSliderApplication functions to reuse code Key: SLIDER-1113 URL: https://issues.apache.org/jira/browse/SLIDER-1113 Project: Slider Issue Type: Improvement Reporter: thomas liu Assignee: thomas liu Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-1112) Create a separate createTemplatedSliderApplication utility function and let testCreateApplicationWithOneAddonPackagesForNoComponents use that
thomas liu created SLIDER-1112: -- Summary: Create a separate createTemplatedSliderApplication utility function and let testCreateApplicationWithOneAddonPackagesForNoComponents use that Key: SLIDER-1112 URL: https://issues.apache.org/jira/browse/SLIDER-1112 Project: Slider Issue Type: Improvement Reporter: thomas liu Assignee: thomas liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-1109) Create some documentation of the task here
[ https://issues.apache.org/jira/browse/SLIDER-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236615#comment-15236615 ] thomas liu commented on SLIDER-1109: [~elserj] Not problem! Please make comments if any! [~gsaha] Thank you for reviewing it! > Create some documentation of the task here > -- > > Key: SLIDER-1109 > URL: https://issues.apache.org/jira/browse/SLIDER-1109 > Project: Slider > Issue Type: Sub-task > Components: agent, agent-provider, core >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-906) Support for Docker based application packaging with first class YARN support - Phase 2
[ https://issues.apache.org/jira/browse/SLIDER-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236612#comment-15236612 ] thomas liu commented on SLIDER-906: --- [~elserj] That's a good idea! Please find the high level design doc uploaded. Thank you [~gsaha] for reviewing & uploading it! > Support for Docker based application packaging with first class YARN support > - Phase 2 > -- > > Key: SLIDER-906 > URL: https://issues.apache.org/jira/browse/SLIDER-906 > Project: Slider > Issue Type: New Feature > Components: agent, agent-provider, core >Affects Versions: Slider 0.80 >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 2.0.0 > > Attachments: SLIDER-906-002.patch, > SLIDER-Docker-Based-App-Packaging-With-First-Class-YARN-Support-v0.pdf > > > This is phase 2 of Docker support in Slider. SLIDER-780 was phase 1, which > was mostly tech-preview level of work. This bug will track work to provide > first class Docker support deeply integrated with YARN Docker support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-1109) Create some documentation of the task here
thomas liu created SLIDER-1109: -- Summary: Create some documentation of the task here Key: SLIDER-1109 URL: https://issues.apache.org/jira/browse/SLIDER-1109 Project: Slider Issue Type: Sub-task Reporter: thomas liu Assignee: thomas liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-999) stdout and stderr spit out by application process on the agent side should be saved into a log file
[ https://issues.apache.org/jira/browse/SLIDER-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu resolved SLIDER-999. --- Resolution: Fixed > stdout and stderr spit out by application process on the agent side should be > saved into a log file > --- > > Key: SLIDER-999 > URL: https://issues.apache.org/jira/browse/SLIDER-999 > Project: Slider > Issue Type: Bug > Components: agent >Affects Versions: Slider 0.81 >Reporter: Gour Saha >Assignee: thomas liu > Fix For: Slider 0.91 > > Attachments: slider-999-004.patch > > > Currently the stdout and stderr spit out by application processes is lost. > Slider agent should capture them and dump them into a log. This is very > helpful in debugging application start issues. There are a good bunch of > applications which dump logs to stdout and stderr. The tarball packages might > be easier to change. Those in docker image form would be difficult to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1098) AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows
[ https://issues.apache.org/jira/browse/SLIDER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1098: --- Attachment: slider-1098-002.patch removing comments > AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows > > > Key: SLIDER-1098 > URL: https://issues.apache.org/jira/browse/SLIDER-1098 > Project: Slider > Issue Type: Bug >Reporter: thomas liu >Assignee: thomas liu > Attachments: slider-1098-001.patch, slider-1098-002.patch > > > AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-1101) Five tests fail during 'verifyFileExist'
[ https://issues.apache.org/jira/browse/SLIDER-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu resolved SLIDER-1101. Resolution: Fixed > Five tests fail during 'verifyFileExist' > > > Key: SLIDER-1101 > URL: https://issues.apache.org/jira/browse/SLIDER-1101 > Project: Slider > Issue Type: Bug >Reporter: thomas liu >Assignee: thomas liu > Attachments: slider-1101-001.patch, slider-1101-002.patch > > > ComponentConfigsInAppConfigShowUpOnAgentIT.testComponentConfigsInAppConfigCanShowUpOnAgentSide; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForAllComponents; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithMultipeAddonPackages; > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForOneComponent; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForMultipleComponents > are failing on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1101) Five tests fail during 'verifyFileExist'
[ https://issues.apache.org/jira/browse/SLIDER-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1101: --- Attachment: slider-1101-001.patch > Five tests fail during 'verifyFileExist' > > > Key: SLIDER-1101 > URL: https://issues.apache.org/jira/browse/SLIDER-1101 > Project: Slider > Issue Type: Bug >Reporter: thomas liu >Assignee: thomas liu > Attachments: slider-1101-001.patch > > > ComponentConfigsInAppConfigShowUpOnAgentIT.testComponentConfigsInAppConfigCanShowUpOnAgentSide; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForAllComponents; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithMultipeAddonPackages; > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForOneComponent; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForMultipleComponents > are failing on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-1098) AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows
[ https://issues.apache.org/jira/browse/SLIDER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu resolved SLIDER-1098. Resolution: Fixed > AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows > > > Key: SLIDER-1098 > URL: https://issues.apache.org/jira/browse/SLIDER-1098 > Project: Slider > Issue Type: Bug >Reporter: thomas liu >Assignee: thomas liu > Attachments: slider-1098-001.patch, slider-1098-002.patch > > > AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1101) Five tests fail during 'verifyFileExist'
[ https://issues.apache.org/jira/browse/SLIDER-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1101: --- Attachment: slider-1101-001.patch > Five tests fail during 'verifyFileExist' > > > Key: SLIDER-1101 > URL: https://issues.apache.org/jira/browse/SLIDER-1101 > Project: Slider > Issue Type: Bug >Reporter: thomas liu >Assignee: thomas liu > Attachments: slider-1101-001.patch > > > ComponentConfigsInAppConfigShowUpOnAgentIT.testComponentConfigsInAppConfigCanShowUpOnAgentSide; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForAllComponents; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithMultipeAddonPackages; > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForOneComponent; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForMultipleComponents > are failing on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-999) stdout and stderr spit out by application process on the agent side should be saved into a log file
[ https://issues.apache.org/jira/browse/SLIDER-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-999: -- Attachment: (was: BUG-48463-APACHE-004.patch) > stdout and stderr spit out by application process on the agent side should be > saved into a log file > --- > > Key: SLIDER-999 > URL: https://issues.apache.org/jira/browse/SLIDER-999 > Project: Slider > Issue Type: Bug > Components: agent >Affects Versions: Slider 0.81 >Reporter: Gour Saha >Assignee: thomas liu > Fix For: Slider 0.91 > > > Currently the stdout and stderr spit out by application processes is lost. > Slider agent should capture them and dump them into a log. This is very > helpful in debugging application start issues. There are a good bunch of > applications which dump logs to stdout and stderr. The tarball packages might > be easier to change. Those in docker image form would be difficult to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1101) Five tests fail during 'verifyFileExist'
[ https://issues.apache.org/jira/browse/SLIDER-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1101: --- Attachment: (was: slider-1101-001.patch) > Five tests fail during 'verifyFileExist' > > > Key: SLIDER-1101 > URL: https://issues.apache.org/jira/browse/SLIDER-1101 > Project: Slider > Issue Type: Bug >Reporter: thomas liu >Assignee: thomas liu > > ComponentConfigsInAppConfigShowUpOnAgentIT.testComponentConfigsInAppConfigCanShowUpOnAgentSide; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForAllComponents; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithMultipeAddonPackages; > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForOneComponent; > > ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForMultipleComponents > are failing on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-1101) Five tests fail during 'verifyFileExist'
thomas liu created SLIDER-1101: -- Summary: Five tests fail during 'verifyFileExist' Key: SLIDER-1101 URL: https://issues.apache.org/jira/browse/SLIDER-1101 Project: Slider Issue Type: Bug Reporter: thomas liu Assignee: thomas liu ComponentConfigsInAppConfigShowUpOnAgentIT.testComponentConfigsInAppConfigCanShowUpOnAgentSide; ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForAllComponents; ApplicationWithAddonPackagesIT.testCreateApplicationWithMultipeAddonPackages; ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForOneComponent; ApplicationWithAddonPackagesIT.testCreateApplicationWithOneAddonPackagesForMultipleComponents are failing on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-999) stdout and stderr spit out by application process on the agent side should be saved into a log file
[ https://issues.apache.org/jira/browse/SLIDER-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-999: -- Attachment: slider-999-004.patch > stdout and stderr spit out by application process on the agent side should be > saved into a log file > --- > > Key: SLIDER-999 > URL: https://issues.apache.org/jira/browse/SLIDER-999 > Project: Slider > Issue Type: Bug > Components: agent >Affects Versions: Slider 0.81 >Reporter: Gour Saha >Assignee: thomas liu > Fix For: Slider 0.91 > > Attachments: slider-999-004.patch > > > Currently the stdout and stderr spit out by application processes is lost. > Slider agent should capture them and dump them into a log. This is very > helpful in debugging application start issues. There are a good bunch of > applications which dump logs to stdout and stderr. The tarball packages might > be easier to change. Those in docker image form would be difficult to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-999) stdout and stderr spit out by application process on the agent side should be saved into a log file
[ https://issues.apache.org/jira/browse/SLIDER-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-999: -- Attachment: BUG-48463-APACHE-004.patch > stdout and stderr spit out by application process on the agent side should be > saved into a log file > --- > > Key: SLIDER-999 > URL: https://issues.apache.org/jira/browse/SLIDER-999 > Project: Slider > Issue Type: Bug > Components: agent >Affects Versions: Slider 0.81 >Reporter: Gour Saha >Assignee: thomas liu > Fix For: Slider 0.91 > > Attachments: BUG-48463-APACHE-004.patch > > > Currently the stdout and stderr spit out by application processes is lost. > Slider agent should capture them and dump them into a log. This is very > helpful in debugging application start issues. There are a good bunch of > applications which dump logs to stdout and stderr. The tarball packages might > be easier to change. Those in docker image form would be difficult to change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-1098) AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows
[ https://issues.apache.org/jira/browse/SLIDER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-1098: --- Attachment: slider-1098-001.patch > AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows > > > Key: SLIDER-1098 > URL: https://issues.apache.org/jira/browse/SLIDER-1098 > Project: Slider > Issue Type: Bug >Reporter: thomas liu >Assignee: thomas liu > Attachments: slider-1098-001.patch > > > AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-1098) AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows
thomas liu created SLIDER-1098: -- Summary: AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows Key: SLIDER-1098 URL: https://issues.apache.org/jira/browse/SLIDER-1098 Project: Slider Issue Type: Bug Reporter: thomas liu Assignee: thomas liu AMClientCertStoreRetrievalIT test fails with invalid HDFS URI on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-950) Implement GetStatus API call for multiple applications
[
https://issues.apache.org/jira/browse/SLIDER-950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
thomas liu updated SLIDER-950:
--
Attachment: SLIDER-950-001.patch
I have implemented an end to end get multi status API. Please take a look to
see how you think of it.
I tried to run it on the 5node cluster
the output of 'ApplicationReport' is:
[applicationId { id: 56 cluster_timestamp: 1445117923783 } user: "root" queue:
"default" name: "oct2705" host: "cn007.l42scl.hortonworks.com" rpc_port: 1024
yarn_application_state: RUNNING trackingUrl:
"http://cn007.l42scl.hortonworks.com:8088/proxy/application_1445117923783_0056/;
diagnostics: "" startTime: 1446003405432 finishTime: 0
final_application_status: APP_UNDEFINED app_resource_Usage {
num_used_containers: 1 num_reserved_containers: 0 used_resources { memory: 1024
virtual_cores: 1 } reserved_resources { memory: 0 virtual_cores: 0 }
needed_resources { memory: 1024 virtual_cores: 1 } memory_seconds: 13097783673
vcore_seconds: 4567487 } originalTrackingUrl:
"http://cn007.l42scl.hortonworks.com:1025; currentApplicationAttemptId {
application_id { id: 56 cluster_timestamp: 1445117923783 } attemptId: 2 }
progress: 0.5 applicationType: "org-apache-slider" applicationTags:
"description: null" applicationTags: "version: null" applicationTags: "name:
myycloud" log_aggregation_status: LOG_NOT_START]
Do you think it has covered all we need?
Thanks
> Implement GetStatus API call for multiple applications
> --
>
> Key: SLIDER-950
> URL: https://issues.apache.org/jira/browse/SLIDER-950
> Project: Slider
> Issue Type: New Feature
> Components: client
>Affects Versions: Slider 0.80
>Reporter: thomas liu
>Assignee: thomas liu
> Fix For: Slider 0.90
>
> Attachments: SLIDER-950-001.patch, SLIDER-950-DISCUSSION-1.patch
>
>
> We found it useful to implement an API to allow getting status for multiple
> applications in a cluster. We restrict the applications to a list of names we
> pass into the API
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SLIDER-950) Implement GetStatus API call for multiple applications
[ https://issues.apache.org/jira/browse/SLIDER-950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-950: -- Attachment: SLIDER-950-DISCUSSION-1.patch I read the code in SliderYarnClientImpl.java and make an experiment here to create a new function getReportForMultiInstanceByName, in a similar way to listDeployedInstances. Please find the code change in the attachment Please note listDeployedInstances return a list of ApplicationReport, which is different than what the current GetStatus returns (ClusterDescription) Does ApplicationReport suffice for our need in GetStatus? > Implement GetStatus API call for multiple applications > -- > > Key: SLIDER-950 > URL: https://issues.apache.org/jira/browse/SLIDER-950 > Project: Slider > Issue Type: New Feature > Components: client >Affects Versions: Slider 0.80 >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 0.90 > > Attachments: SLIDER-950-DISCUSSION-1.patch > > > We found it useful to implement an API to allow getting status for multiple > applications in a cluster. We restrict the applications to a list of names we > pass into the API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-950) Implement GetStatus API call for multiple applications
[ https://issues.apache.org/jira/browse/SLIDER-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993306#comment-14993306 ] thomas liu commented on SLIDER-950: --- Ah, Sounds good. I didn't look into this class SliderYarnClientImpl as it is not in the current code path of 'getStatus'. I'm trying to make use of it now > Implement GetStatus API call for multiple applications > -- > > Key: SLIDER-950 > URL: https://issues.apache.org/jira/browse/SLIDER-950 > Project: Slider > Issue Type: New Feature > Components: client >Affects Versions: Slider 0.80 >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 0.90 > > > We found it useful to implement an API to allow getting status for multiple > applications in a cluster. We restrict the applications to a list of names we > pass into the API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-950) Implement GetStatus API call for multiple applications
[ https://issues.apache.org/jira/browse/SLIDER-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986791#comment-14986791 ] thomas liu commented on SLIDER-950: --- One of the ways to implement this getStatus API is to: * implement a new action, say getMultiAppStatus, with its argument class and all necessary changes on the client side; * in SliderClient, the new action would invoke a new method called 'actionMultiAppStatus' in exec() * actionMultiAppStatus would do similar things as their single app origin. the difference is it now invokes YARN API multiple times: once per app > Implement GetStatus API call for multiple applications > -- > > Key: SLIDER-950 > URL: https://issues.apache.org/jira/browse/SLIDER-950 > Project: Slider > Issue Type: New Feature > Components: client >Affects Versions: Slider 0.80 >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 0.90 > > > We found it useful to implement an API to allow getting status for multiple > applications in a cluster. We restrict the applications to a list of names we > pass into the API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-951) Implement GetRegistry API for multiple applications
[ https://issues.apache.org/jira/browse/SLIDER-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986790#comment-14986790 ] thomas liu commented on SLIDER-951: --- One of the ways to implement this getRegistry API is to: * implement a new action, say getMultiAppRegistry, with its argument class and all necessary changes on the client side; * in SliderClient, the new action would invoke a new method called 'actionMultiAppRegistry' in exec() * actionMultiAppRegistry would still invoke 5 new downstream methods according to different options in the action, similar to what actionRegistry is invoking: actionMultiAppRegistryList, actionMultiAppRegistryListConfigsYarn, actionMultiAppRegistryListExports, actionMultiAppRegistryGetConfig, actionMultiAppRegistryGetExport * the 5 new methods do similar things as their single app origins. the difference is they now invoke YARN API multiple times: once per app > Implement GetRegistry API for multiple applications > --- > > Key: SLIDER-951 > URL: https://issues.apache.org/jira/browse/SLIDER-951 > Project: Slider > Issue Type: New Feature > Components: client >Affects Versions: Slider 0.80 >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 0.90 > > > We find it useful to return registry information for multiple applications, > restricted by the list of names passed into the API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-957) Make all Slider's resource keys replace 'yarn' suffix with 'slider' suffix
thomas liu created SLIDER-957: - Summary: Make all Slider's resource keys replace 'yarn' suffix with 'slider' suffix Key: SLIDER-957 URL: https://issues.apache.org/jira/browse/SLIDER-957 Project: Slider Issue Type: Task Reporter: thomas liu Priority: Minor Currently all resource keys in slider's config file 'resources.json' are using 'yarn' as suffix to indicate that those keys will be passed to YARN as corresponding yarn resource keys. As suggested by Gour, maybe we can investigate whether it would be appropriate to use 'slider' as suffix and then replace all 'yarn' suffix with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-951) Implement GetRegistry API for multiple applications
thomas liu created SLIDER-951: - Summary: Implement GetRegistry API for multiple applications Key: SLIDER-951 URL: https://issues.apache.org/jira/browse/SLIDER-951 Project: Slider Issue Type: New Feature Reporter: thomas liu We find it useful to return registry information for multiple applications, restricted by the list of names passed into the API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-929) Fix test_signal_handler failure
[
https://issues.apache.org/jira/browse/SLIDER-929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
thomas liu updated SLIDER-929:
--
Attachment: SLIDER-929-001.patch
Fix for the test case is ready
> Fix test_signal_handler failure
> ---
>
> Key: SLIDER-929
> URL: https://issues.apache.org/jira/browse/SLIDER-929
> Project: Slider
> Issue Type: Bug
> Components: agent, test
>Affects Versions: Slider 0.80
>Reporter: Gour Saha
>Assignee: thomas liu
> Fix For: Slider 0.81
>
> Attachments: SLIDER-929-001.patch
>
>
> test_signal_handler started failing after commit for SLIDER-916 went in. Need
> to fix this.
> {code}
> ==
> ERROR: test_signal_handler (TestMain.TestMain)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/jenkins-slave/workspace/Slider-develop/slider-agent/src/test/python/mock/mock.py",
> line 1199, in patched
> return func(*args, **keywargs)
> File
> "/home/jenkins/jenkins-slave/workspace/Slider-develop/slider-agent/src/test/python/agent/TestMain.py",
> line 60, in test_signal_handler
> main.signal_handler("signum", "frame")
> File
> "/home/jenkins/jenkins-slave/workspace/Slider-develop/slider-agent/src/main/python/agent/main.py",
> line 59, in signal_handler
> tmpdir = controller.actionQueue.dockerManager.stop_container()
> AttributeError: 'Controller' object has no attribute 'actionQueue'
> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLIDER-929) Fix test_signal_handler failure
[
https://issues.apache.org/jira/browse/SLIDER-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958184#comment-14958184
]
thomas liu commented on SLIDER-929:
---
OK. Cool
> Fix test_signal_handler failure
> ---
>
> Key: SLIDER-929
> URL: https://issues.apache.org/jira/browse/SLIDER-929
> Project: Slider
> Issue Type: Bug
> Components: agent, test
>Affects Versions: Slider 0.80
>Reporter: Gour Saha
>Assignee: thomas liu
> Fix For: Slider 0.81
>
> Attachments: SLIDER-929-001.patch
>
>
> test_signal_handler started failing after commit for SLIDER-916 went in. Need
> to fix this.
> {code}
> ==
> ERROR: test_signal_handler (TestMain.TestMain)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/jenkins-slave/workspace/Slider-develop/slider-agent/src/test/python/mock/mock.py",
> line 1199, in patched
> return func(*args, **keywargs)
> File
> "/home/jenkins/jenkins-slave/workspace/Slider-develop/slider-agent/src/test/python/agent/TestMain.py",
> line 60, in test_signal_handler
> main.signal_handler("signum", "frame")
> File
> "/home/jenkins/jenkins-slave/workspace/Slider-develop/slider-agent/src/main/python/agent/main.py",
> line 59, in signal_handler
> tmpdir = controller.actionQueue.dockerManager.stop_container()
> AttributeError: 'Controller' object has no attribute 'actionQueue'
> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLIDER-946) PyTest test_signal_handler failing
[
https://issues.apache.org/jira/browse/SLIDER-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955388#comment-14955388
]
thomas liu commented on SLIDER-946:
---
Agree with Steve.
This test case failure is not one of the partial merge issues we saw previously
To fix it:
The simplest fix would be checking if actionQueue is created in controller or
not, just for fixing the test case.
It also doesn't break the protocol: actionQueue is not created until
controller.run()
> PyTest test_signal_handler failing
> ---
>
> Key: SLIDER-946
> URL: https://issues.apache.org/jira/browse/SLIDER-946
> Project: Slider
> Issue Type: Bug
> Components: agent, test
>Affects Versions: Slider 0.90
>Reporter: Steve Loughran
>
> The test {{test_signal_handler}} is failing; message is {{ 'Controller'
> object has no attribute 'actionQueue'}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SLIDER-933) Component status in status command result always stuck in '3' even after start command is successful
thomas liu created SLIDER-933:
-
Summary: Component status in status command result always stuck in
'3' even after start command is successful
Key: SLIDER-933
URL: https://issues.apache.org/jira/browse/SLIDER-933
Project: Slider
Issue Type: Bug
Reporter: thomas liu
Component status in status command result always stuck in '3', which is
starting, even after start command is completed successfully
Suspect it should be '4'
public enum State {
INIT, // Not installed
INSTALLING, // Being installed
INSTALLED, // Installed (or stopped)
STARTING, // Starting
...
Need investigation for the root cause
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLIDER-919) Slider DockerManager should check whether image is available locally before pull from repository
[
https://issues.apache.org/jira/browse/SLIDER-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14638033#comment-14638033
]
thomas liu commented on SLIDER-919:
---
In our immediate release, we are not using DockerContainerExecutor (obsolete)
We will use docker runtime in LC.
There we do pulling
Slider DockerManager should check whether image is available locally before
pull from repository
Key: SLIDER-919
URL: https://issues.apache.org/jira/browse/SLIDER-919
Project: Slider
Issue Type: Improvement
Components: agent
Affects Versions: Slider 0.80
Reporter: Chen He
if command['roleCommand'] == 'INSTALL':
returncode, out, err = self.pull_image(command)
logger.info(docker pull result: + str(returncode) + ;)
if command['roleCommand'] == 'START':
returncode, out, err = self.start_container(command)
return {Constants.EXIT_CODE:returncode, 'stdout':out, 'stderr':err}
If user load image through tar file, it is available in local disk but not in
remote repository or private repository. Slider agent should do a check
before pulling
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLIDER-919) Slider DockerManager should check whether image is available locally before pull from repository
[
https://issues.apache.org/jira/browse/SLIDER-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14636106#comment-14636106
]
thomas liu commented on SLIDER-919:
---
If it is, then pull won't do much.
Besides, we are migrating this into YARN.
So no plan to fix
Slider DockerManager should check whether image is available locally before
pull from repository
Key: SLIDER-919
URL: https://issues.apache.org/jira/browse/SLIDER-919
Project: Slider
Issue Type: Improvement
Components: agent
Affects Versions: Slider 0.80
Reporter: Chen He
if command['roleCommand'] == 'INSTALL':
returncode, out, err = self.pull_image(command)
logger.info(docker pull result: + str(returncode) + ;)
if command['roleCommand'] == 'START':
returncode, out, err = self.start_container(command)
# need check
return {Constants.EXIT_CODE:returncode, 'stdout':out, 'stderr':err}
If user load image through tar file, it is available in local disk but not in
remote repository or private repository. Slider agent should do a check
before pulling
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SLIDER-916) Fix Docker related bugs
[ https://issues.apache.org/jira/browse/SLIDER-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-916: -- Attachment: SLIDER-916-001.patch Patch of the code change Will post a patch for doc only soon Fix Docker related bugs --- Key: SLIDER-916 URL: https://issues.apache.org/jira/browse/SLIDER-916 Project: Slider Issue Type: Bug Affects Versions: Slider 0.80 Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.81 Attachments: SLIDER-916-001.patch Following things needs to be taken care of on top of work done on SLIDER-780 - # add the missing line in AgentProviderService.java because of merge # make options optional in appConfig.json or metainfo.json # add the missing con as global var in Controller.py because of merge # container_id can use the whole container_id string provided by YARN # review the whole documentation of docker and fix issues -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-916) Fix Docker related bugs
[ https://issues.apache.org/jira/browse/SLIDER-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-916: -- Description: add the missing line in AgentProviderService.java because of merge make options optional in appConfig.json or metainfo.json add the missing con as global var in Controller.py because of merge container_id used in slideragent sometimes is rite, sometimes not was: add the missing line in AgentProviderService.java because of merge allow empty options String add the missing con as global var in Controller.py because of merge container_id used in slideragent sometimes is rite, sometimes not Fix Docker related bugs --- Key: SLIDER-916 URL: https://issues.apache.org/jira/browse/SLIDER-916 Project: Slider Issue Type: Bug Reporter: thomas liu add the missing line in AgentProviderService.java because of merge make options optional in appConfig.json or metainfo.json add the missing con as global var in Controller.py because of merge container_id used in slideragent sometimes is rite, sometimes not -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-916) Fix Docker related bugs
[ https://issues.apache.org/jira/browse/SLIDER-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-916: -- Description: add the missing line in AgentProviderService.java because of merge allow empty options String add the missing con as global var in Controller.py because of merge container_id used in slideragent sometimes is rite, sometimes not was: add the missing line because of merge allow empty options String add the missing con as global var in Controller.py because of merge container_id used in slideragent sometimes is rite, sometimes not Fix Docker related bugs --- Key: SLIDER-916 URL: https://issues.apache.org/jira/browse/SLIDER-916 Project: Slider Issue Type: Bug Reporter: thomas liu add the missing line in AgentProviderService.java because of merge allow empty options String add the missing con as global var in Controller.py because of merge container_id used in slideragent sometimes is rite, sometimes not -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-916) Fix Docker related bugs
[ https://issues.apache.org/jira/browse/SLIDER-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-916: -- Description: add the missing line in AgentProviderService.java because of merge make options optional in appConfig.json or metainfo.json add the missing con as global var in Controller.py because of merge container_id can use the whole container_id string provided by YARN review the whole documentation of docker and fix issues was: add the missing line in AgentProviderService.java because of merge make options optional in appConfig.json or metainfo.json add the missing con as global var in Controller.py because of merge container_id used in slideragent sometimes is rite, sometimes not Fix Docker related bugs --- Key: SLIDER-916 URL: https://issues.apache.org/jira/browse/SLIDER-916 Project: Slider Issue Type: Bug Reporter: thomas liu add the missing line in AgentProviderService.java because of merge make options optional in appConfig.json or metainfo.json add the missing con as global var in Controller.py because of merge container_id can use the whole container_id string provided by YARN review the whole documentation of docker and fix issues -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-916) Fix Docker related bugs
thomas liu created SLIDER-916: - Summary: Fix Docker related bugs Key: SLIDER-916 URL: https://issues.apache.org/jira/browse/SLIDER-916 Project: Slider Issue Type: Bug Reporter: thomas liu add the missing line because of merge allow empty options String add the missing con as global var in Controller.py because of merge -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-901) AgentClientInstallIT#testAgentClientInstall last validation statement is failing on Windows
[
https://issues.apache.org/jira/browse/SLIDER-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
thomas liu updated SLIDER-901:
--
Attachment: SLIDER-901-002.patch
AgentClientInstallIT#testAgentClientInstall last validation statement is
failing on Windows
---
Key: SLIDER-901
URL: https://issues.apache.org/jira/browse/SLIDER-901
Project: Slider
Issue Type: Bug
Affects Versions: Slider 0.80
Reporter: thomas liu
Assignee: thomas liu
Attachments: SLIDER-901-001.patch, SLIDER-901-002.patch
AgentClientInstallIT passes on Linux.
However, on Windows, the second validation statement in the end of the test
case fails:
{noformat}
assert list.contains(expectedFile2)
|||
|false
C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\command-logger-app\operations.log
[C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\operations.log]
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
​at
org.apache.slider.funtest.lifecycle.AgentClientInstallIT.testAgentClientInstall(AgentClientInstallIT.groovy:77)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SLIDER-901) AgentClientInstallIT last validation statement is failing on Windows
thomas liu created SLIDER-901: - Summary: AgentClientInstallIT last validation statement is failing on Windows Key: SLIDER-901 URL: https://issues.apache.org/jira/browse/SLIDER-901 Project: Slider Issue Type: Bug Reporter: thomas liu AgentClientInstallIT passes on Linux. However, on Windows, the second validation statement in the end of the test case fails: assert list.contains(expectedFile2) ||| |false C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\command-logger-app\operations.log [C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\operations.log] at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648) ​at org.apache.slider.funtest.lifecycle.AgentClientInstallIT.testAgentClientInstall(AgentClientInstallIT.groovy:77) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-901) AgentClientInstallIT last validation statement is failing on Windows
[
https://issues.apache.org/jira/browse/SLIDER-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
thomas liu updated SLIDER-901:
--
Description:
AgentClientInstallIT passes on Linux.
However, on Windows, the second validation statement in the end of the test
case fails:
{quote}
assert list.contains(expectedFile2)
|||
|false
C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\command-logger-app\operations.log
[C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\operations.log]
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
​at
org.apache.slider.funtest.lifecycle.AgentClientInstallIT.testAgentClientInstall(AgentClientInstallIT.groovy:77)
{quote}
was:
AgentClientInstallIT passes on Linux.
However, on Windows, the second validation statement in the end of the test
case fails:
assert list.contains(expectedFile2)
|||
|false
C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\command-logger-app\operations.log
[C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\operations.log]
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
​at
org.apache.slider.funtest.lifecycle.AgentClientInstallIT.testAgentClientInstall(AgentClientInstallIT.groovy:77)
AgentClientInstallIT last validation statement is failing on Windows
Key: SLIDER-901
URL: https://issues.apache.org/jira/browse/SLIDER-901
Project: Slider
Issue Type: Bug
Reporter: thomas liu
AgentClientInstallIT passes on Linux.
However, on Windows, the second validation statement in the end of the test
case fails:
{quote}
assert list.contains(expectedFile2)
|||
|false
C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\command-logger-app\operations.log
[C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\operations.log]
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
​at
org.apache.slider.funtest.lifecycle.AgentClientInstallIT.testAgentClientInstall(AgentClientInstallIT.groovy:77)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SLIDER-902) AMClientCertStoreRetrievalIT failing on Windows
thomas liu created SLIDER-902: - Summary: AMClientCertStoreRetrievalIT failing on Windows Key: SLIDER-902 URL: https://issues.apache.org/jira/browse/SLIDER-902 Project: Slider Issue Type: Bug Reporter: thomas liu The test case is passing on Linux It uses some Linux specific path so fails on Windows Fixing the Linux specific path still produce error below: 2015-06-09 19:41:55,343 [JUnit] ERROR framework.ShellBase (NativeMethodAccessorImpl.java:invoke0(?)) - stderr 2015-06-09 19:41:53,482 [main] INFO client.RMProxy - Connecting to ResourceManager at dal-slider1/10.79.146.141:80322015-06-09 19:41:53,673 [main] INFO client.SliderClient - No hostname specified via command line. Using dal-slider1 Exception: org.apache.slider.core.exceptions.SliderException: Error running command openssl pkcs12 -export -in C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.crt -inkey C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.key -certfile C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.crt -out C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\keystore-client-.p12 -password pass:welcome -passin pass:Kt8H0w2kJK6k7FyW52BAbJB82OcXowC7DW37TM5P0YVIilaciV -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-901) AgentClientInstallIT last validation statement is failing on Windows
[
https://issues.apache.org/jira/browse/SLIDER-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
thomas liu updated SLIDER-901:
--
Attachment: SLIDER-901-001.patch
Attaching first fix
AgentClientInstallIT last validation statement is failing on Windows
Key: SLIDER-901
URL: https://issues.apache.org/jira/browse/SLIDER-901
Project: Slider
Issue Type: Bug
Affects Versions: Slider 0.80
Reporter: thomas liu
Assignee: thomas liu
Attachments: SLIDER-901-001.patch
AgentClientInstallIT passes on Linux.
However, on Windows, the second validation statement in the end of the test
case fails:
{noformat}
assert list.contains(expectedFile2)
|||
|false
C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\command-logger-app\operations.log
[C:\Users\hadoopqa\AppData\Local\Temp\del8816690427180892258dir\operations.log]
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
​at
org.apache.slider.funtest.lifecycle.AgentClientInstallIT.testAgentClientInstall(AgentClientInstallIT.groovy:77)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (SLIDER-902) AMClientCertStoreRetrievalIT failing on Windows
[
https://issues.apache.org/jira/browse/SLIDER-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
thomas liu updated SLIDER-902:
--
Comment: was deleted
(was: Attaching the first fix)
AMClientCertStoreRetrievalIT failing on Windows
---
Key: SLIDER-902
URL: https://issues.apache.org/jira/browse/SLIDER-902
Project: Slider
Issue Type: Bug
Affects Versions: Slider 0.80
Reporter: thomas liu
Assignee: thomas liu
The test case is passing on Linux
It uses some Linux specific path so fails on Windows
Fixing the Linux specific path still produce error below:
{noformat}
2015-06-09 19:41:55,343 [JUnit] ERROR framework.ShellBase
(NativeMethodAccessorImpl.java:invoke0(?)) -
stderr
2015-06-09 19:41:53,482 [main] INFO client.RMProxy - Connecting to
ResourceManager at dal-slider1/10.79.146.141:8032
2015-06-09 19:41:53,673 [main] INFO client.SliderClient - No hostname
specified via command line. Using dal-slider1
Exception: org.apache.slider.core.exceptions.SliderException: Error running
command openssl pkcs12 -export -in
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.crt
-inkey
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.key
-certfile
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.crt -out
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\keystore-client-.p12
-password pass:welcome -passin
pass:Kt8H0w2kJK6k7FyW52BAbJB82OcXowC7DW37TM5P0YVIilaciV
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SLIDER-902) AMClientCertStoreRetrievalIT failing on Windows
[
https://issues.apache.org/jira/browse/SLIDER-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
thomas liu updated SLIDER-902:
--
Attachment: SLIDER-902-001.patch
Attaching the first fix
AMClientCertStoreRetrievalIT failing on Windows
---
Key: SLIDER-902
URL: https://issues.apache.org/jira/browse/SLIDER-902
Project: Slider
Issue Type: Bug
Affects Versions: Slider 0.80
Reporter: thomas liu
Assignee: thomas liu
Attachments: SLIDER-902-001.patch
The test case is passing on Linux
It uses some Linux specific path so fails on Windows
Fixing the Linux specific path still produce error below:
{noformat}
2015-06-09 19:41:55,343 [JUnit] ERROR framework.ShellBase
(NativeMethodAccessorImpl.java:invoke0(?)) -
stderr
2015-06-09 19:41:53,482 [main] INFO client.RMProxy - Connecting to
ResourceManager at dal-slider1/10.79.146.141:8032
2015-06-09 19:41:53,673 [main] INFO client.SliderClient - No hostname
specified via command line. Using dal-slider1
Exception: org.apache.slider.core.exceptions.SliderException: Error running
command openssl pkcs12 -export -in
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.crt
-inkey
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.key
-certfile
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.crt -out
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\keystore-client-.p12
-password pass:welcome -passin
pass:Kt8H0w2kJK6k7FyW52BAbJB82OcXowC7DW37TM5P0YVIilaciV
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SLIDER-902) AMClientCertStoreRetrievalIT failing on Windows
[
https://issues.apache.org/jira/browse/SLIDER-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
thomas liu updated SLIDER-902:
--
Attachment: slider-902-001.patch
Attaching the first fix
AMClientCertStoreRetrievalIT failing on Windows
---
Key: SLIDER-902
URL: https://issues.apache.org/jira/browse/SLIDER-902
Project: Slider
Issue Type: Bug
Affects Versions: Slider 0.80
Reporter: thomas liu
Assignee: thomas liu
Attachments: slider-902-001.patch
The test case is passing on Linux
It uses some Linux specific path so fails on Windows
Fixing the Linux specific path still produce error below:
{noformat}
2015-06-09 19:41:55,343 [JUnit] ERROR framework.ShellBase
(NativeMethodAccessorImpl.java:invoke0(?)) -
stderr
2015-06-09 19:41:53,482 [main] INFO client.RMProxy - Connecting to
ResourceManager at dal-slider1/10.79.146.141:8032
2015-06-09 19:41:53,673 [main] INFO client.SliderClient - No hostname
specified via command line. Using dal-slider1
Exception: org.apache.slider.core.exceptions.SliderException: Error running
command openssl pkcs12 -export -in
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.crt
-inkey
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.key
-certfile
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\client.crt -out
C:\Users\hadoop\AppData\Local\Temp\sec1433878889711\security\keystore-client-.p12
-password pass:welcome -passin
pass:Kt8H0w2kJK6k7FyW52BAbJB82OcXowC7DW37TM5P0YVIilaciV
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SLIDER-897) Slider fun test failing on Windows
[ https://issues.apache.org/jira/browse/SLIDER-897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-897: -- Attachment: JIRA-897-001.patch Slider fun test failing on Windows -- Key: SLIDER-897 URL: https://issues.apache.org/jira/browse/SLIDER-897 Project: Slider Issue Type: Bug Components: test Reporter: thomas liu Fix For: Slider 0.81 Attachments: JIRA-897-001.patch 5 slider STs are failing on Windows. - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithMultipeAddonPackages - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithOneAddonPackagesForMultipleComponents - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithOneAddonPackagesForAllComponents - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithOneAddonPackagesForNoComponents - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithOneAddonPackagesForOneComponent -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-897) Slider fun test failing on Windows
thomas liu created SLIDER-897: - Summary: Slider fun test failing on Windows Key: SLIDER-897 URL: https://issues.apache.org/jira/browse/SLIDER-897 Project: Slider Issue Type: Bug Components: test Reporter: thomas liu Fix For: Slider 0.81 5 slider STs are failing on Windows. - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithMultipeAddonPackages - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithOneAddonPackagesForMultipleComponents - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithOneAddonPackagesForAllComponents - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithOneAddonPackagesForNoComponents - org.apache.slider.funtest.coprocessors.ApplicationWithAddonPackagesIT-testCreateApplicationWithOneAddonPackagesForOneComponent -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-812) Making component configurations in appConfig available on the SliderAgent side
[ https://issues.apache.org/jira/browse/SLIDER-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-812: -- Attachment: JIRA-812-001.patch Making component configurations in appConfig available on the SliderAgent side -- Key: SLIDER-812 URL: https://issues.apache.org/jira/browse/SLIDER-812 Project: Slider Issue Type: New Feature Components: agent-provider Affects Versions: Slider 0.70 Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.81 Attachments: JIRA-812-001.patch, JIRA-812-001.patch, allConfig.patch, allConfig.patch Currently, there is a need on the Agent side to access component specific configuration values in appConfig. Here in AgentProviderService, we pass the component section in appConfig json blob inside the execution commands to Agents. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-895) Slider Unit Test Cases Failing on Windows
[ https://issues.apache.org/jira/browse/SLIDER-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-895: -- Attachment: JIRA-895-001.patch 2) should be due to misconfig of 'winutils.exe'. For 3), communicated with the author and the cause of the failure is believed to be misconfigured system time on the machine. Please re-run to verify. Thanks Slider Unit Test Cases Failing on Windows - Key: SLIDER-895 URL: https://issues.apache.org/jira/browse/SLIDER-895 Project: Slider Issue Type: Bug Reporter: thomas liu Fix For: Slider 0.81 Attachments: JIRA-895-001.patch There are three test cases failing on Win: 1)org.apache.slider.agent.standalone.TestStandaloneAMMonkeyRestart.testStandaloneAMMonkeyRestart 2) org.apache.slider.providers.agent.TestAgentClientProvider2.testRunCommand 3) org.apache.slider.server.services.security.TestCertificateManager.testContainerTrusttoreGeneration -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-895) Slider Unit Test Cases Failing on Windows
thomas liu created SLIDER-895: - Summary: Slider Unit Test Cases Failing on Windows Key: SLIDER-895 URL: https://issues.apache.org/jira/browse/SLIDER-895 Project: Slider Issue Type: Bug Reporter: thomas liu Fix For: Slider 0.81 There are three test cases failing on Win: 1)org.apache.slider.agent.standalone.TestStandaloneAMMonkeyRestart.testStandaloneAMMonkeyRestart 2) org.apache.slider.providers.agent.TestAgentClientProvider2.testRunCommand 3) org.apache.slider.server.services.security.TestCertificateManager.testContainerTrusttoreGeneration -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases for App packages
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Attachment: JIRA-847-004.patch Add unit and functional test cases for App packages --- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch, JIRA-847-002.patch, JIRA-847-003.patch, JIRA-847-004.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases for App packages
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Attachment: JIRA-847-005.patch Add unit and functional test cases for App packages --- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch, JIRA-847-002.patch, JIRA-847-003.patch, JIRA-847-004.patch, JIRA-847-005.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-847) Add unit and functional test cases for App packages
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542978#comment-14542978 ] thomas liu commented on SLIDER-847: --- Attached a new patch with those useless lines removed. Thanks Add unit and functional test cases for App packages --- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch, JIRA-847-002.patch, JIRA-847-003.patch, JIRA-847-004.patch, JIRA-847-005.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases for App packages
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Attachment: JIRA-847-004.patch Add unit and functional test cases for App packages --- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch, JIRA-847-002.patch, JIRA-847-003.patch, JIRA-847-004.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases for App packages
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Attachment: (was: JIRA-847-004.patch) Add unit and functional test cases for App packages --- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch, JIRA-847-002.patch, JIRA-847-003.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases for App packages
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Attachment: JIRA-847-003.patch Fixes for windows test cases Add unit and functional test cases for App packages --- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch, JIRA-847-002.patch, JIRA-847-003.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases for App packages
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Attachment: JIRA-847-002.patch Fixes for running on windows Add unit and functional test cases for App packages --- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch, JIRA-847-002.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-865) Create test cases for Slider-Docker support
[ https://issues.apache.org/jira/browse/SLIDER-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-865: -- Fix Version/s: (was: Slider 1.0.0) Slider 2.0.0 Create test cases for Slider-Docker support --- Key: SLIDER-865 URL: https://issues.apache.org/jira/browse/SLIDER-865 Project: Slider Issue Type: Sub-task Reporter: thomas liu Assignee: thomas liu Fix For: Slider 2.0.0 As Docker applications need to have Docker daemon running in the target host, it is more feasible to create functional test cases, where installation of Docker daemon in all hosts in the functional test cluster is required prior to running the test cases. Normal cases: 1) launch a single Docker container application, which uploads a file to HDFS, verify the existence of that file. need to create this Docker image. this is to test the application inside the container can access service outside the container 2) launch a single Docker container application, which is a web server, verify with http get. need to create this Docker image. this is to test the application inside the container is accessible from outside 3) launch a Docker application with multi Docker containers communicating with each other, for example, nodejs to redis, verify with http get. can reuse existing nodejs and redis Docker image. this is to test the components can access each other Exceptional cases: 1) invalid metainfo.json 2) non-existing Docker image 3) invalid 'docker run' command to start the containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-865) Create test cases for Slider-Docker support
[ https://issues.apache.org/jira/browse/SLIDER-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-865: -- Fix Version/s: (was: Slider 0.80) Slider 1.0.0 Create test cases for Slider-Docker support --- Key: SLIDER-865 URL: https://issues.apache.org/jira/browse/SLIDER-865 Project: Slider Issue Type: Sub-task Reporter: thomas liu Assignee: thomas liu Fix For: Slider 1.0.0 As Docker applications need to have Docker daemon running in the target host, it is more feasible to create functional test cases, where installation of Docker daemon in all hosts in the functional test cluster is required prior to running the test cases. Normal cases: 1) launch a single Docker container application, which uploads a file to HDFS, verify the existence of that file. need to create this Docker image. this is to test the application inside the container can access service outside the container 2) launch a single Docker container application, which is a web server, verify with http get. need to create this Docker image. this is to test the application inside the container is accessible from outside 3) launch a Docker application with multi Docker containers communicating with each other, for example, nodejs to redis, verify with http get. can reuse existing nodejs and redis Docker image. this is to test the components can access each other Exceptional cases: 1) invalid metainfo.json 2) non-existing Docker image 3) invalid 'docker run' command to start the containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-865) Create test cases for Slider-Docker support
[ https://issues.apache.org/jira/browse/SLIDER-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-865: -- Description: As Docker applications need to have Docker daemon running in the target host, it is more feasible to create functional test cases, where installation of Docker daemon in all hosts in the functional test cluster is required prior to running the test cases. Normal cases: 1) launch a single Docker container application, which uploads a file to HDFS, verify the existence of that file. need to create this Docker image. this is to test the application inside the container can access service outside the container 2) launch a single Docker container application, which is a web server, verify with http get. need to create this Docker image. this is to test the application inside the container is accessible from outside 3) launch a Docker application with multi Docker containers communicating with each other, for example, nodejs to redis, verify with http get. can reuse existing nodejs and redis Docker image. this is to test the components can access each other Exceptional cases: 1) invalid metainfo.json 2) non-existing Docker image 3) invalid 'docker run' command to start the containers was: As Docker applications need to have Docker daemon running in the target host, it is more feasible to create functional test cases, where installation of Docker daemon in all hosts in the functional test cluster is required prior to running the test cases. Normal cases: 1) launch a single Docker container application, which uploads a file to HDFS, verify the existence of that file. need to create this Docker image 2) launch a single Docker container application, which is a web server, verify with http get. need to create this Docker image 2) launch a multi Docker containers application, components of which upload different files to HDFS, verify the existence of those files. need to create those Docker images 3) launch a Docker application with multi Docker containers communicating with each other, for example, nodejs to redis, verify with http get Exceptional cases: Create test cases for Slider-Docker support --- Key: SLIDER-865 URL: https://issues.apache.org/jira/browse/SLIDER-865 Project: Slider Issue Type: Sub-task Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 As Docker applications need to have Docker daemon running in the target host, it is more feasible to create functional test cases, where installation of Docker daemon in all hosts in the functional test cluster is required prior to running the test cases. Normal cases: 1) launch a single Docker container application, which uploads a file to HDFS, verify the existence of that file. need to create this Docker image. this is to test the application inside the container can access service outside the container 2) launch a single Docker container application, which is a web server, verify with http get. need to create this Docker image. this is to test the application inside the container is accessible from outside 3) launch a Docker application with multi Docker containers communicating with each other, for example, nodejs to redis, verify with http get. can reuse existing nodejs and redis Docker image. this is to test the components can access each other Exceptional cases: 1) invalid metainfo.json 2) non-existing Docker image 3) invalid 'docker run' command to start the containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-865) Create test cases for Slider-Docker support
[ https://issues.apache.org/jira/browse/SLIDER-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-865: -- Description: As Docker applications need to have Docker daemon running in the target host, it is more feasible to create functional test cases, where installation of Docker daemon in all hosts in the functional test cluster is required prior to running the test cases. Normal cases: 1) launch a single Docker container application, which uploads a file to HDFS, verify the existence of that file. need to create this Docker image 2) launch a single Docker container application, which is a web server, verify with http get. need to create this Docker image 2) launch a multi Docker containers application, components of which upload different files to HDFS, verify the existence of those files. need to create those Docker images 3) launch a Docker application with multi Docker containers communicating with each other, for example, nodejs to redis, verify with http get Exceptional cases: Create test cases for Slider-Docker support --- Key: SLIDER-865 URL: https://issues.apache.org/jira/browse/SLIDER-865 Project: Slider Issue Type: Sub-task Reporter: thomas liu Fix For: Slider 0.80 As Docker applications need to have Docker daemon running in the target host, it is more feasible to create functional test cases, where installation of Docker daemon in all hosts in the functional test cluster is required prior to running the test cases. Normal cases: 1) launch a single Docker container application, which uploads a file to HDFS, verify the existence of that file. need to create this Docker image 2) launch a single Docker container application, which is a web server, verify with http get. need to create this Docker image 2) launch a multi Docker containers application, components of which upload different files to HDFS, verify the existence of those files. need to create those Docker images 3) launch a Docker application with multi Docker containers communicating with each other, for example, nodejs to redis, verify with http get Exceptional cases: -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-865) Create test cases for Slider-Docker support
thomas liu created SLIDER-865: - Summary: Create test cases for Slider-Docker support Key: SLIDER-865 URL: https://issues.apache.org/jira/browse/SLIDER-865 Project: Slider Issue Type: Sub-task Reporter: thomas liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-780) Support for Docker based application packaging in Slider
[ https://issues.apache.org/jira/browse/SLIDER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-780: -- Attachment: JIRA-780-002.patch This code patch is to address Gour's comments. Will attach another patch only for test cases soon Support for Docker based application packaging in Slider Key: SLIDER-780 URL: https://issues.apache.org/jira/browse/SLIDER-780 Project: Slider Issue Type: New Feature Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-780-001.patch, JIRA-780-002.patch, SlidersupportingDockersummary (1).pdf Enable Slider to deploy an application defined as Docker image, monitor its running status, fetching exported configs, and maintain its lifecycle. Please find the details of this ticket in the attachment 'SlidersupportingDockersummary(1).pdf' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-847) Add unit and functional test cases for App packages
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525106#comment-14525106 ] thomas liu commented on SLIDER-847: --- Thank you for confirming! I had run all unit test cases and fun-test cases before submitted the patch. Though there are one or two failing ones, our code change didn't cause any test case failures Please check in the patch. Thanks! Add unit and functional test cases for App packages --- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SLIDER-780) Support for Docker based application packaging in Slider
[ https://issues.apache.org/jira/browse/SLIDER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524371#comment-14524371 ] thomas liu edited comment on SLIDER-780 at 5/2/15 7:09 AM: --- This code patch is to address Gour's comments. Will attach another patch only for test cases soon I have run all unit test cases and fun-test cases. Though there are one or two failing ones, they are not because of our code change was (Author: thomas_liu): This code patch is to address Gour's comments. Will attach another patch only for test cases soon Support for Docker based application packaging in Slider Key: SLIDER-780 URL: https://issues.apache.org/jira/browse/SLIDER-780 Project: Slider Issue Type: New Feature Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-780-001.patch, JIRA-780-002.patch, SlidersupportingDockersummary (1).pdf Enable Slider to deploy an application defined as Docker image, monitor its running status, fetching exported configs, and maintain its lifecycle. Please find the details of this ticket in the attachment 'SlidersupportingDockersummary(1).pdf' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-780) Support for Docker based application packaging in Slider
[
https://issues.apache.org/jira/browse/SLIDER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522850#comment-14522850
]
thomas liu commented on SLIDER-780:
---
Thank you for your comments!
Now I'm still working on the fun-tests of this patch.
Given we need to install Docker on the target hosts, writing unit test with
minicluster is a little tricker
{quote}
I think we can get rid of variable con and declare controller at global level.
Then call controller.actionQueue.dockerManager.stop_container() in
signal_handler.
{quote}
Agree. Also seen your comment below regarding the failing test case.
{quote}
Is there a reason to use the digit 2 in the method names? If not, please rename
them appropriately.
{quote}
No there isn't...It was a result of previous temporary addStartCommand2 code
in the private branches. I just blindly merged them in. Will rename them.
{quote}
return type.toLowerCase().equals(docker);
Let's define string constant for docker in SliderUtils.java just like PYTHON
is defined.
cmdParams.put(script_type, PYTHON);
Let's define TYPE_PYTHON in AbstractComponent.java just like you have
TYPE_DOCKER.
{quote}
Agree.
{quote}
Not required now, but at some point we need to define PythonExecutionCommand
and DockerExecutionCommand to modularize the code. This will also pave the way
for supporting additional executors/containers in the future, like Flocker
(which is picking some steam as well).
{quote}
Agree. Also have this in my radar. Will simplify the params we pass to Agent as
well. Besides, PythonExecutionCommand and DockerExecutionCommand, there can be
ShellCommandExecutionCommand as well.
{quote}
Use formatter {} instead of string concat.
{quote}
Agree. Those are old logging code. Will change their level.
{quote}
Seems like we can use entrySet() as you are accessing key and value.
{quote}
Agree
{quote}
DockerContainerPort.java, DockerContainerMount.java, and DockerContainer.java
public void validate(String version) throws SliderException {
Add @Override to all validate methods.
{quote}
Agree
{quote}
Use StringBuilder
{quote}
Agree
{quote}
Some generic formatting comments:
{quote}
Sounds good
Thank you for your comments!
Will post my new patch with test cases tomorrow.
Support for Docker based application packaging in Slider
Key: SLIDER-780
URL: https://issues.apache.org/jira/browse/SLIDER-780
Project: Slider
Issue Type: New Feature
Reporter: thomas liu
Assignee: thomas liu
Fix For: Slider 0.80
Attachments: JIRA-780-001.patch, SlidersupportingDockersummary (1).pdf
Enable Slider to deploy an application defined as Docker image, monitor its
running status, fetching exported configs, and maintain its lifecycle.
Please find the details of this ticket in the attachment
'SlidersupportingDockersummary(1).pdf'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (SLIDER-822) Define new fields in communication between AgentProviderService and SliderAgent to indicate multiple addon packages
[ https://issues.apache.org/jira/browse/SLIDER-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu resolved SLIDER-822. --- Resolution: Fixed Define new fields in communication between AgentProviderService and SliderAgent to indicate multiple addon packages --- Key: SLIDER-822 URL: https://issues.apache.org/jira/browse/SLIDER-822 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-821) Enable SliderAgent to receive INSTALL and CONFIGURE commands for both master package and addon package
[ https://issues.apache.org/jira/browse/SLIDER-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu resolved SLIDER-821. --- Resolution: Fixed Enable SliderAgent to receive INSTALL and CONFIGURE commands for both master package and addon package -- Key: SLIDER-821 URL: https://issues.apache.org/jira/browse/SLIDER-821 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 SliderAgent should also maintain state for all master and addon packages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-780) Support for Docker based application packaging in Slider
[ https://issues.apache.org/jira/browse/SLIDER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-780: -- Attachment: JIRA-780-001.patch This is the code change patch that enables Slider to deploy basic Docker based applications Support for Docker based application packaging in Slider Key: SLIDER-780 URL: https://issues.apache.org/jira/browse/SLIDER-780 Project: Slider Issue Type: New Feature Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-780-001.patch, SlidersupportingDockersummary (1).pdf Enable Slider to deploy an application defined as Docker image, monitor its running status, fetching exported configs, and maintain its lifecycle. Please find the details of this ticket in the attachment 'SlidersupportingDockersummary(1).pdf' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-820) Enable AgentProviderService issue INSTALL and CONFIGURE command for addon packages
[ https://issues.apache.org/jira/browse/SLIDER-820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu resolved SLIDER-820. --- Resolution: Fixed Enable AgentProviderService issue INSTALL and CONFIGURE command for addon packages -- Key: SLIDER-820 URL: https://issues.apache.org/jira/browse/SLIDER-820 Project: Slider Issue Type: Sub-task Components: agent-provider, core Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Enable AgentProviderService issue INSTALL and CONFIGURE command for addon packages, with parameters in the commands to install and configure addon packages onto master package directories, for lib, conf, bin... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Attachment: JIRA-847-001.patch Added unit test and fun test cases in the attachment Add unit and functional test cases -- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: thomas liu Fix For: Slider 0.80 Attachments: JIRA-847-001.patch Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-819) Create addon package of Phoenix and Ranger for Hbase
[ https://issues.apache.org/jira/browse/SLIDER-819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520823#comment-14520823 ] thomas liu commented on SLIDER-819: --- Apache Phoenix add on package may help users using older version of HBase. However, this may be a very limited case. Create addon package of Phoenix and Ranger for Hbase Key: SLIDER-819 URL: https://issues.apache.org/jira/browse/SLIDER-819 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Assignee: Ted Yu Fix For: Slider 2.0.0 To develop and test Slider-773, create addon package containing Apache Phoenix and Ranger for its master Hbase package. The details of how to create add on package is described: SLIDER-834 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511472#comment-14511472 ] thomas liu commented on SLIDER-773: --- Thank you for your comment! The hbase tar there was originally used to test Slider, with add on package's python script, can copy the file over to the right directory. I should replace it with some small jars Building the zip files online is definitely a good idea. I will modify my current test cases to do that. Thanks again! Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: 773-suggest.txt, Co-processorSupport.pdf, SLIDER-773_review_comments.patch, SLIDER-773_review_comments_part2.patch, coprocessor-after.patch, coprocessor-apri-4th.patch, coprocessor.patch, coprocessor.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-773) Add co-processor support for app packages
[
https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499309#comment-14499309
]
thomas liu commented on SLIDER-773:
---
Thank you for your patch! It looks good to me.
Only one thing that I don't know why happen:
@@ -999,7 +999,7 @@ public class AgentProviderService extends
AbstractProviderService implements
} catch (SliderException e) {
log.warn(Component instance failed operation., e);
- componentStatus.applyCommandResult(CommandResult.FAILED, command);
+ componentStatus.applyCommandResult(CommandResult.FAILED, command, null);
}
log.debug(Heartbeat response: + response);
Add co-processor support for app packages
-
Key: SLIDER-773
URL: https://issues.apache.org/jira/browse/SLIDER-773
Project: Slider
Issue Type: Bug
Components: app-package, client
Affects Versions: Slider 0.60
Reporter: Sumit Mohanty
Assignee: thomas liu
Priority: Critical
Fix For: Slider 0.80
Attachments: 773-suggest.txt, Co-processorSupport.pdf,
SLIDER-773_review_comments.patch, SLIDER-773_review_comments_part2.patch,
coprocessor-after.patch, coprocessor-apri-4th.patch, coprocessor.patch,
coprocessor.patch
It is typical for applications to allow plugins/co-processors that are
essentially a set of additional jar files in the classpath and optionally a
set of config files or config changes.
Current, slider app packages can handle additional config changes/entries
very well. Additional configs files can be added as well but it is not easy
if the config files include parameters that need to be resolved by the agent.
This requires app package changes. Dropping additional jar files into the
class path is not easy and requires app package changes.
It is not efficient to modify the app package to support such plugins. App
packaging and create command should be modified such that the user can
dynamically specify additional jars, config files, configs etc.
Specific scenarios are modifying HBase to add support for Phoenix or Ranger.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (SLIDER-773) Add co-processor support for app packages
[
https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499309#comment-14499309
]
thomas liu edited comment on SLIDER-773 at 4/17/15 5:58 AM:
Thank you for your patch! It looks good to me.
Only one thing that I don't know why happen:
{quote}
@@ -999,7 +999,7 @@ public class AgentProviderService extends
AbstractProviderService implements
} catch (SliderException e) {
log.warn(Component instance failed operation., e);
- componentStatus.applyCommandResult(CommandResult.FAILED, command);
+ componentStatus.applyCommandResult(CommandResult.FAILED, command, null);
}
log.debug(Heartbeat response: + response);
{quote}
On my side, it is already
{quote}
componentStatus.applyCommandResult(CommandResult.FAILED, command, null);
{quote}
and it is not what I modified recently...
was (Author: thomas_liu):
Thank you for your patch! It looks good to me.
Only one thing that I don't know why happen:
@@ -999,7 +999,7 @@ public class AgentProviderService extends
AbstractProviderService implements
} catch (SliderException e) {
log.warn(Component instance failed operation., e);
- componentStatus.applyCommandResult(CommandResult.FAILED, command);
+ componentStatus.applyCommandResult(CommandResult.FAILED, command, null);
}
log.debug(Heartbeat response: + response);
Add co-processor support for app packages
-
Key: SLIDER-773
URL: https://issues.apache.org/jira/browse/SLIDER-773
Project: Slider
Issue Type: Bug
Components: app-package, client
Affects Versions: Slider 0.60
Reporter: Sumit Mohanty
Assignee: thomas liu
Priority: Critical
Fix For: Slider 0.80
Attachments: 773-suggest.txt, Co-processorSupport.pdf,
SLIDER-773_review_comments.patch, SLIDER-773_review_comments_part2.patch,
coprocessor-after.patch, coprocessor-apri-4th.patch, coprocessor.patch,
coprocessor.patch
It is typical for applications to allow plugins/co-processors that are
essentially a set of additional jar files in the classpath and optionally a
set of config files or config changes.
Current, slider app packages can handle additional config changes/entries
very well. Additional configs files can be added as well but it is not easy
if the config files include parameters that need to be resolved by the agent.
This requires app package changes. Dropping additional jar files into the
class path is not easy and requires app package changes.
It is not efficient to modify the app package to support such plugins. App
packaging and create command should be modified such that the user can
dynamically specify additional jars, config files, configs etc.
Specific scenarios are modifying HBase to add support for Phoenix or Ranger.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498536#comment-14498536 ] thomas liu commented on SLIDER-773: --- This is a good point. I was hesitating over this as well. So if there are add on packages to process, once they pass initial validation, pkgStatus will not be empty and contain the key for sure. If there are no packages to process, we would not visit this map at all. But I do agree that we should probably have some null pointer checking here. When I look around AgentProviderService, it looks like for several places, we are not doing this checking but confidently use the variable got from the map. For example, in handleHeartBeat(), line 891, installCmd = compCmd. That's why I followed this pattern Please let me know what you think. If necessary, we can create a ticket to fix all those cases Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: 773-suggest.txt, Co-processorSupport.pdf, SLIDER-773_review_comments.patch, coprocessor-after.patch, coprocessor-apri-4th.patch, coprocessor.patch, coprocessor.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498536#comment-14498536 ] thomas liu edited comment on SLIDER-773 at 4/16/15 7:08 PM: This is a good point. I was hesitating over this as well. Right now, if there are add on packages to process, once they pass initial validation, pkgStatus will not be empty and contain the key for sure. If there are no packages to process, we would not visit this map at all. But I do agree that we should probably have some null pointer checking here. When I look around AgentProviderService, it looks like for several places, we are not doing this checking but confidently use the variable got from the map. For example, in handleHeartBeat(), line 891, installCmd = compCmd. That's why I followed this pattern Please let me know what you think. If necessary, we can create a ticket to fix all those cases was (Author: thomas_liu): This is a good point. I was hesitating over this as well. So if there are add on packages to process, once they pass initial validation, pkgStatus will not be empty and contain the key for sure. If there are no packages to process, we would not visit this map at all. But I do agree that we should probably have some null pointer checking here. When I look around AgentProviderService, it looks like for several places, we are not doing this checking but confidently use the variable got from the map. For example, in handleHeartBeat(), line 891, installCmd = compCmd. That's why I followed this pattern Please let me know what you think. If necessary, we can create a ticket to fix all those cases Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: 773-suggest.txt, Co-processorSupport.pdf, SLIDER-773_review_comments.patch, coprocessor-after.patch, coprocessor-apri-4th.patch, coprocessor.patch, coprocessor.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SLIDER-773) Add co-processor support for app packages
[
https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498514#comment-14498514
]
thomas liu edited comment on SLIDER-773 at 4/16/15 7:02 PM:
Hi, Gour:
Thank you very much for your comments!
I tested it manually and it works for normal add on packages scenarios
Below are my responses to the code:
Potential bugs -
In line 1002 of AgentProviderService.java was the pkg or non-pkg version of
componentStatus.applyCommandResult supposed to be called? I think we should
call the non-pkg version.
} catch (SliderException e) {
log.warn(Component instance failed operation., e);
componentStatus.applyCommandResult(CommandResult.FAILED, command, null);
}
Right now, it is calling the non-pkg version (by passing in null as the third
param)
Starting with line 82 of AppDefinitionPersister.java, appDefinition.pkgName is
not changed in the following if block. Based on the logging, seems like it is
supposed to change. Is it a potential bug?
String targetName = appDefinition.pkgName;
log.debug(Initial package name: + targetName);
if (appDefinition.appDefPkgOrFolder.isDirectory()) {
log.info(Processing app package/folder {} for {},
appDefinition.appDefPkgOrFolder.getAbsolutePath(),
appDefinition.pkgName);
File tmpDir = Files.createTempDir();
File zipFile = new File(tmpDir.getCanonicalPath(), File.separator +
appDefinition.pkgName);
SliderUtils.zipFolder(appDefinition.appDefPkgOrFolder, zipFile);
src = zipFile;
targetName = appDefinition.pkgName;
}
log.debug(Final package name: + targetName);
This is an interesting case. This method was originally written by Sumit to
define the add on package file name on HDFS. However, line 82String targetName
= appDefinition.pkgName; was originally assigned some other values which
caused appConfig has a different add on package file name than the actual one
on HDFS. That's why I modified 82 to be assigned appDefinition.pkgName.
I also noticed the if block assign this name again, but it contains more logic
in case the user provided a folder, vs a file as the add on package file. so we
should leave the if block there
Agree with all of the comments below, with exceptions in between:
Themes of other issues -
Avoid unnecessary new lines in both python and java files
Avoid spaces in new lines in both python and java files
Start log message texts with a capitalized word (for consistency in Slider
code-base)
As Ted Yu already mentioned, avoid long lines (wrap beyond 80 chars)
As Steve Loughran pointed out, use formatter versions for logging instead of
string concat. It is more efficient and avoids unnecessary creation of objects
for debug level logs say, when logging is set at a higher level.
Avoid logging at info level in heartbeat flows (in both agent provider and
python side) for NOP/STATUS command scenarios. Use AgentToggleLogger.py on the
python side to see how you can log once for STATUS commands and then switch to
silent mode, until a non-STATUS command shows up again.
Avoid repetition of code for overloaded methods or constructors. Typically most
of the logic can be written in one method/constructor and call that
method/constructor with appropriate placeholder values from the other
overloaded methods/constructors.
There should be a space between if and ( between ) and } for if statements
Remove unnecessary imports before creating a patch
If a patch is a large one, typically avoid making changes to a file which has
cosmetic changes only. The cosmetic change can always be made as a separate
patch with a corresponding JIRA.
Declaration of variables or method signatures should not use implementation
classes like -
TreeMapString, State pkgStatuses = new TreeMapString, State();
instead use
MapString, State pkgStatuses = new TreeMapString, State();
I believe what you mentioned is a general principal for declaring variables.
However, in our case, the declaration must be of type TreeMap instead of a
normal Map, because we need an ordered map so that we can traverse it in order
As Ted Yu mentioned Component.MASTER_PACKAGE_NAME.equals(pkg) covers the check
for pkg.isEmpty()
In ComponentInstanceState.java as Ted Yu mentioned, you should use
pkgStatuses.entrySet() instead of pkgStatuses.keySet() as both key and value
are accessed in the loop.
In ComponentInstanceState.java is the following line required just before the
return statement (around line 234)?
nextPkgToInstall = null;
Try to define all accessor/setter methods before methods like toString,
hashCode and equals
Do not repeat getters/setters in concrete class like Component.java which are
already defined in abstract super classes
Do not checkin code with default stub codes like // TODO Auto-generated method
stub
was (Author:
[jira] [Comment Edited] (SLIDER-773) Add co-processor support for app packages
[
https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498514#comment-14498514
]
thomas liu edited comment on SLIDER-773 at 4/16/15 7:44 PM:
Hi, Gour:
Thank you very much for your comments!
I tested it manually and it works for normal add on packages scenarios
Below are my responses to the code:
{quote}
Potential bugs -
In line 1002 of AgentProviderService.java was the pkg or non-pkg version of
componentStatus.applyCommandResult supposed to be called? I think we should
call the non-pkg version.
} catch (SliderException e) {
log.warn(Component instance failed operation., e);
componentStatus.applyCommandResult(CommandResult.FAILED, command, null);
}
{quote}
Right now, it IS calling the non-pkg version (by passing in null as the third
param)
{quote}
Starting with line 82 of AppDefinitionPersister.java, appDefinition.pkgName is
not changed in the following if block. Based on the logging, seems like it is
supposed to change. Is it a potential bug?
String targetName = appDefinition.pkgName;
log.debug(Initial package name: + targetName);
if (appDefinition.appDefPkgOrFolder.isDirectory()) {
log.info(Processing app package/folder {} for {},
appDefinition.appDefPkgOrFolder.getAbsolutePath(),
appDefinition.pkgName);
File tmpDir = Files.createTempDir();
File zipFile = new File(tmpDir.getCanonicalPath(), File.separator +
appDefinition.pkgName);
SliderUtils.zipFolder(appDefinition.appDefPkgOrFolder, zipFile);
src = zipFile;
targetName = appDefinition.pkgName;
}
log.debug(Final package name: + targetName);
{quote}
This is an interesting case. This method was originally written by Sumit to
define the add on package file name on HDFS. However, line 82String targetName
= appDefinition.pkgName; was originally assigned some other values which
caused appConfig has a different add on package file name than the actual one
on HDFS. That's why I modified 82 to be assigned appDefinition.pkgName.
I also noticed the if block assign this name again, but it contains more logic
in case the user provided a folder, vs a file as the add on package file. so we
should leave the if block there
Agree with all of the comments below, with exceptions in between:
{quote}
Themes of other issues -
Avoid unnecessary new lines in both python and java files
Avoid spaces in new lines in both python and java files
Start log message texts with a capitalized word (for consistency in Slider
code-base)
As Ted Yu already mentioned, avoid long lines (wrap beyond 80 chars)
As Steve Loughran pointed out, use formatter versions for logging instead of
string concat. It is more efficient and avoids unnecessary creation of objects
for debug level logs say, when logging is set at a higher level.
Avoid logging at info level in heartbeat flows (in both agent provider and
python side) for NOP/STATUS command scenarios. Use AgentToggleLogger.py on the
python side to see how you can log once for STATUS commands and then switch to
silent mode, until a non-STATUS command shows up again.
Avoid repetition of code for overloaded methods or constructors. Typically most
of the logic can be written in one method/constructor and call that
method/constructor with appropriate placeholder values from the other
overloaded methods/constructors.
There should be a space between if and ( between ) and } for if statements
Remove unnecessary imports before creating a patch
If a patch is a large one, typically avoid making changes to a file which has
cosmetic changes only. The cosmetic change can always be made as a separate
patch with a corresponding JIRA.
Declaration of variables or method signatures should not use implementation
classes like -
TreeMapString, State pkgStatuses = new TreeMapString, State();
instead use
MapString, State pkgStatuses = new TreeMapString, State();
{quote}
I believe what you mentioned is a general principal for declaring variables.
However, in our case, the declaration must be of type TreeMap instead of a
normal Map, because we need an ordered map so that we can traverse it in order
{quote}
As Ted Yu mentioned Component.MASTER_PACKAGE_NAME.equals(pkg) covers the check
for pkg.isEmpty()
In ComponentInstanceState.java as Ted Yu mentioned, you should use
pkgStatuses.entrySet() instead of pkgStatuses.keySet() as both key and value
are accessed in the loop.
In ComponentInstanceState.java is the following line required just before the
return statement (around line 234)?
nextPkgToInstall = null;
Try to define all accessor/setter methods before methods like toString,
hashCode and equals
Do not repeat getters/setters in concrete class like Component.java which are
already defined in abstract super classes
Do not checkin code with default stub codes like // TODO
[jira] [Commented] (SLIDER-773) Add co-processor support for app packages
[
https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498514#comment-14498514
]
thomas liu commented on SLIDER-773:
---
Hi, Gour:
Thank you very much for your comments!
I tested it manually and it works for normal add on packages scenarios
Below are my responses to the code:
Potential bugs -
In line 1002 of AgentProviderService.java was the pkg or non-pkg version of
componentStatus.applyCommandResult supposed to be called? I think we should
call the non-pkg version.
} catch (SliderException e) {
log.warn(Component instance failed operation., e);
componentStatus.applyCommandResult(CommandResult.FAILED, command, null);
}
Right now, it is calling the non-pkg version (by passing in null as the third
param)
Starting with line 82 of AppDefinitionPersister.java, appDefinition.pkgName is
not changed in the following if block. Based on the logging, seems like it is
supposed to change. Is it a potential bug?
String targetName = appDefinition.pkgName;
log.debug(Initial package name: + targetName);
if (appDefinition.appDefPkgOrFolder.isDirectory()) {
log.info(Processing app package/folder {} for {},
appDefinition.appDefPkgOrFolder.getAbsolutePath(),
appDefinition.pkgName);
File tmpDir = Files.createTempDir();
File zipFile = new File(tmpDir.getCanonicalPath(), File.separator +
appDefinition.pkgName);
SliderUtils.zipFolder(appDefinition.appDefPkgOrFolder, zipFile);
src = zipFile;
targetName = appDefinition.pkgName;
}
log.debug(Final package name: + targetName);
This is an interesting case. This method was originally written by Sumit to
define the add on package file name on HDFS. However, line 82String targetName
= appDefinition.pkgName; was originally assigned some other values which
caused appConfig has a different add on package file name than the actual one
on HDFS. That's why I modified 82 to be assigned appDefinition.pkgName.
I also noticed the if block assign this name again, but it contains more logic
in case the user provided a folder, vs a file as the add on package file. so we
should leave the if block there
Agree with all of the comments below, with exceptions in between:
Themes of other issues -
Avoid unnecessary new lines in both python and java files
Avoid spaces in new lines in both python and java files
Start log message texts with a capitalized word (for consistency in Slider
code-base)
As Ted Yu already mentioned, avoid long lines (wrap beyond 80 chars)
As Steve Loughran pointed out, use formatter versions for logging instead of
string concat. It is more efficient and avoids unnecessary creation of objects
for debug level logs say, when logging is set at a higher level.
Avoid logging at info level in heartbeat flows (in both agent provider and
python side) for NOP/STATUS command scenarios. Use AgentToggleLogger.py on the
python side to see how you can log once for STATUS commands and then switch to
silent mode, until a non-STATUS command shows up again.
Avoid repetition of code for overloaded methods or constructors. Typically most
of the logic can be written in one method/constructor and call that
method/constructor with appropriate placeholder values from the other
overloaded methods/constructors.
There should be a space between if and ( between ) and } for if statements
Remove unnecessary imports before creating a patch
If a patch is a large one, typically avoid making changes to a file which has
cosmetic changes only. The cosmetic change can always be made as a separate
patch with a corresponding JIRA.
Declaration of variables or method signatures should not use implementation
classes like -
TreeMapString, State pkgStatuses = new TreeMapString, State();
instead use
MapString, State pkgStatuses = new TreeMapString, State();
I believe what you mentioned is a general principal for declaring variables.
However, in our case, the declaration must be of type TreeMap instead of a
normal Map, because we need an ordered map so that we can traverse it in order
As Ted Yu mentioned Component.MASTER_PACKAGE_NAME.equals(pkg) covers the check
for pkg.isEmpty()
In ComponentInstanceState.java as Ted Yu mentioned, you should use
pkgStatuses.entrySet() instead of pkgStatuses.keySet() as both key and value
are accessed in the loop.
In ComponentInstanceState.java is the following line required just before the
return statement (around line 234)?
nextPkgToInstall = null;
Try to define all accessor/setter methods before methods like toString,
hashCode and equals
Do not repeat getters/setters in concrete class like Component.java which are
already defined in abstract super classes
Those are overriding, given that the children class's variable type has changed
Do not checkin code with default stub codes like // TODO Auto-generated method
[jira] [Comment Edited] (SLIDER-773) Add co-processor support for app packages
[
https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498514#comment-14498514
]
thomas liu edited comment on SLIDER-773 at 4/16/15 7:43 PM:
Hi, Gour:
Thank you very much for your comments!
I tested it manually and it works for normal add on packages scenarios
Below are my responses to the code:
{quote}
Potential bugs -
In line 1002 of AgentProviderService.java was the pkg or non-pkg version of
componentStatus.applyCommandResult supposed to be called? I think we should
call the non-pkg version.
} catch (SliderException e) {
log.warn(Component instance failed operation., e);
componentStatus.applyCommandResult(CommandResult.FAILED, command, null);
}
{quote}
Right now, it is calling the non-pkg version (by passing in null as the third
param)
Starting with line 82 of AppDefinitionPersister.java, appDefinition.pkgName is
not changed in the following if block. Based on the logging, seems like it is
supposed to change. Is it a potential bug?
String targetName = appDefinition.pkgName;
log.debug(Initial package name: + targetName);
if (appDefinition.appDefPkgOrFolder.isDirectory()) {
log.info(Processing app package/folder {} for {},
appDefinition.appDefPkgOrFolder.getAbsolutePath(),
appDefinition.pkgName);
File tmpDir = Files.createTempDir();
File zipFile = new File(tmpDir.getCanonicalPath(), File.separator +
appDefinition.pkgName);
SliderUtils.zipFolder(appDefinition.appDefPkgOrFolder, zipFile);
src = zipFile;
targetName = appDefinition.pkgName;
}
log.debug(Final package name: + targetName);
This is an interesting case. This method was originally written by Sumit to
define the add on package file name on HDFS. However, line 82String targetName
= appDefinition.pkgName; was originally assigned some other values which
caused appConfig has a different add on package file name than the actual one
on HDFS. That's why I modified 82 to be assigned appDefinition.pkgName.
I also noticed the if block assign this name again, but it contains more logic
in case the user provided a folder, vs a file as the add on package file. so we
should leave the if block there
Agree with all of the comments below, with exceptions in between:
Themes of other issues -
Avoid unnecessary new lines in both python and java files
Avoid spaces in new lines in both python and java files
Start log message texts with a capitalized word (for consistency in Slider
code-base)
As Ted Yu already mentioned, avoid long lines (wrap beyond 80 chars)
As Steve Loughran pointed out, use formatter versions for logging instead of
string concat. It is more efficient and avoids unnecessary creation of objects
for debug level logs say, when logging is set at a higher level.
Avoid logging at info level in heartbeat flows (in both agent provider and
python side) for NOP/STATUS command scenarios. Use AgentToggleLogger.py on the
python side to see how you can log once for STATUS commands and then switch to
silent mode, until a non-STATUS command shows up again.
Avoid repetition of code for overloaded methods or constructors. Typically most
of the logic can be written in one method/constructor and call that
method/constructor with appropriate placeholder values from the other
overloaded methods/constructors.
There should be a space between if and ( between ) and } for if statements
Remove unnecessary imports before creating a patch
If a patch is a large one, typically avoid making changes to a file which has
cosmetic changes only. The cosmetic change can always be made as a separate
patch with a corresponding JIRA.
Declaration of variables or method signatures should not use implementation
classes like -
TreeMapString, State pkgStatuses = new TreeMapString, State();
instead use
MapString, State pkgStatuses = new TreeMapString, State();
I believe what you mentioned is a general principal for declaring variables.
However, in our case, the declaration must be of type TreeMap instead of a
normal Map, because we need an ordered map so that we can traverse it in order
As Ted Yu mentioned Component.MASTER_PACKAGE_NAME.equals(pkg) covers the check
for pkg.isEmpty()
In ComponentInstanceState.java as Ted Yu mentioned, you should use
pkgStatuses.entrySet() instead of pkgStatuses.keySet() as both key and value
are accessed in the loop.
In ComponentInstanceState.java is the following line required just before the
return statement (around line 234)?
nextPkgToInstall = null;
Try to define all accessor/setter methods before methods like toString,
hashCode and equals
Do not repeat getters/setters in concrete class like Component.java which are
already defined in abstract super classes
Do not checkin code with default stub codes like // TODO Auto-generated method
stub
was
[jira] [Updated] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-773: -- Attachment: coprocessor-after.patch Adopting Ted's comments Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: 773-suggest.txt, Co-processorSupport.pdf, coprocessor-after.patch, coprocessor-apri-4th.patch, coprocessor.patch, coprocessor.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491980#comment-14491980 ] thomas liu commented on SLIDER-773: --- Sounds good. I have incorporated all your suggestion into the latest patch 'coprocessor-after.patch' Thank you! Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: 773-suggest.txt, Co-processorSupport.pdf, coprocessor-after.patch, coprocessor-apri-4th.patch, coprocessor.patch, coprocessor.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-773) Add co-processor support for app packages
[
https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491879#comment-14491879
]
thomas liu commented on SLIDER-773:
---
Thank you for your comments.
Here are my response to them:
Some minor comments:
+log.debug(metainfo map for master and addon is: {},
+packageMetainfo.toString());
The contents of packageMetainfo have been logged in the prior while loop. Is
the above needed ?
This one is used to make sure all packages' metainfo is read, which is not
helped by the one in the loop.
The one in the loop can be ignored. I'm removing it from the loop
+log.debug(for comp + roleName + pkg +
componentStatus.getNextPkgToInstall() + issuing + command.toString());
Wrap long line. In HBase, line length is limited to 100.
Sounds good. Wrapped.
+ComponentCommand installCmd = null;
+for (ComponentCommand compCmd : comp.getCommands()) {
+ if (compCmd.getName().equals(INSTALL)) {
+installCmd = compCmd;
+ }
+}
+addInstallCommand(roleName, containerId, response, null,
+installCmd, timeout, nextPkgToInstall);
What if 'INSTALL' command isn't found in the for loop above ?
I'm following the same pattern for master package. Please refer to if
(command == Command.INSTALL) block above. Validation on the metainfo
guarantees that there must be python script or INSTALL command
+String appDef, boolean
addonPackage) throws IOException, BadConfigException {
Long line.
Wrapped
+ * @param pkg TODO
Please finish javadoc
Done
+ metainfoParser = new AddoPackageMetainfoParser();
Typo in classname above: missing 'n' between 'o' and 'P'
Fixed the name everywhere
+ log.debug(commandissued: component: {} is in {}, componentName,
currentState);
Insert space between 'd' and 'i'.
Added
+if (pkg != null !pkg.isEmpty()
+ !pkg.equals(Component.MASTER_PACKAGE_NAME)) {
The above can be simplified if you use Component.MASTER_PACKAGE_NAME.equals(pkg)
sounds good. Simplified
+ public void applyCommandResult(CommandResult result, Command command, String
pkg) {
...
+ public void applyCommandResult(CommandResult result, Command command) {
Looks like there is common code between the above two methods which can be
extracted.
sounds good. Simplified
+ for (String pkg : pkgStatuses.keySet()) {
+State pkgState = pkgStatuses.get(pkg);
Please use pkgStatuses.entrySet() since both key and value are accessed.
I think the two loops can be combined: if State.INSTALLING is encountered,
return Command.NOP. Otherwise remember the pkg until the end of loop and return
Command.INSTALL_ADDON for the pkg.
Originally, I planned to code in the way you described. Saving the iterated pkg
name while checking state in INSTALLING may be a little confusing. For better
readability, shall we use two loops to explicitly show a higher rank on
checking state in INSTALLING?
+ case INSTALL_ADDON:
+ if (this == State.INIT || this == State.INSTALL_FAILED) {
+return State.INSTALLING;
For State.INSTALL_FAILED, do we need to keep a counter to limit the number of
failures ?
The path is following what is kept for the master package currently. Let's open
another JIRA to visit INSTALL FAILURE all together.
+/**
+ *
+ */
+public abstract class AbstractMetainfoParser {
Sounds good. Added
Add co-processor support for app packages
-
Key: SLIDER-773
URL: https://issues.apache.org/jira/browse/SLIDER-773
Project: Slider
Issue Type: Bug
Components: app-package, client
Affects Versions: Slider 0.60
Reporter: Sumit Mohanty
Assignee: thomas liu
Priority: Critical
Fix For: Slider 0.80
Attachments: 773-suggest.txt, Co-processorSupport.pdf,
coprocessor-apri-4th.patch, coprocessor.patch, coprocessor.patch
It is typical for applications to allow plugins/co-processors that are
essentially a set of additional jar files in the classpath and optionally a
set of config files or config changes.
Current, slider app packages can handle additional config changes/entries
very well. Additional configs files can be added as well but it is not easy
if the config files include parameters that need to be resolved by the agent.
This requires app package changes. Dropping additional jar files into the
class path is not easy and requires app package changes.
It is not efficient to modify the app package to support such plugins. App
packaging and create command should be modified such that the user can
dynamically specify additional jars, config files, configs etc.
Specific scenarios are
[jira] [Comment Edited] (SLIDER-773) Add co-processor support for app packages
[
https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491879#comment-14491879
]
thomas liu edited comment on SLIDER-773 at 4/13/15 5:23 AM:
Thank you for your comments.
Here are my response to them:
Some minor comments:
+log.debug(metainfo map for master and addon is: {},
+packageMetainfo.toString());
The contents of packageMetainfo have been logged in the prior while loop. Is
the above needed ?
This one is used to make sure all packages' metainfo is read, which is not
helped by the one in the loop.
The one in the loop can be ignored. I'm removing it from the loop
+log.debug(for comp + roleName + pkg +
componentStatus.getNextPkgToInstall() + issuing + command.toString());
Wrap long line. In HBase, line length is limited to 100.
Sounds good. Wrapped.
+ComponentCommand installCmd = null;
+for (ComponentCommand compCmd : comp.getCommands()) {
+ if (compCmd.getName().equals(INSTALL)) {
+installCmd = compCmd;
+ }
+}
+addInstallCommand(roleName, containerId, response, null,
+installCmd, timeout, nextPkgToInstall);
What if 'INSTALL' command isn't found in the for loop above ?
I'm following the same pattern for master package. Please refer to if
(command == Command.INSTALL) block above. Validation on the metainfo
guarantees that there must be python script or INSTALL command
+String appDef, boolean
addonPackage) throws IOException, BadConfigException {
Long line.
Wrapped
+ * @param pkg TODO
Please finish javadoc
Done
+ metainfoParser = new AddoPackageMetainfoParser();
Typo in classname above: missing 'n' between 'o' and 'P'
Fixed the name everywhere
+ log.debug(commandissued: component: {} is in {}, componentName,
currentState);
Insert space between 'd' and 'i'.
Added
+if (pkg != null !pkg.isEmpty()
+ !pkg.equals(Component.MASTER_PACKAGE_NAME)) {
The above can be simplified if you use Component.MASTER_PACKAGE_NAME.equals(pkg)
when we are processing command for master package, pkg would be null or empty.
that's why the code is written like this
+ public void applyCommandResult(CommandResult result, Command command, String
pkg) {
...
+ public void applyCommandResult(CommandResult result, Command command) {
Looks like there is common code between the above two methods which can be
extracted.
sounds good. Simplified
+ for (String pkg : pkgStatuses.keySet()) {
+State pkgState = pkgStatuses.get(pkg);
Please use pkgStatuses.entrySet() since both key and value are accessed.
I think the two loops can be combined: if State.INSTALLING is encountered,
return Command.NOP. Otherwise remember the pkg until the end of loop and return
Command.INSTALL_ADDON for the pkg.
Originally, I planned to code in the way you described. Saving the iterated pkg
name while checking state in INSTALLING may be a little confusing. For better
readability, shall we use two loops to explicitly show a higher rank on
checking state in INSTALLING?
+ case INSTALL_ADDON:
+ if (this == State.INIT || this == State.INSTALL_FAILED) {
+return State.INSTALLING;
For State.INSTALL_FAILED, do we need to keep a counter to limit the number of
failures ?
The path is following what is kept for the master package currently. Let's open
another JIRA to visit INSTALL FAILURE all together.
+/**
+ *
+ */
+public abstract class AbstractMetainfoParser {
Sounds good. Added
was (Author: thomas_liu):
Thank you for your comments.
Here are my response to them:
Some minor comments:
+log.debug(metainfo map for master and addon is: {},
+packageMetainfo.toString());
The contents of packageMetainfo have been logged in the prior while loop. Is
the above needed ?
This one is used to make sure all packages' metainfo is read, which is not
helped by the one in the loop.
The one in the loop can be ignored. I'm removing it from the loop
+log.debug(for comp + roleName + pkg +
componentStatus.getNextPkgToInstall() + issuing + command.toString());
Wrap long line. In HBase, line length is limited to 100.
Sounds good. Wrapped.
+ComponentCommand installCmd = null;
+for (ComponentCommand compCmd : comp.getCommands()) {
+ if (compCmd.getName().equals(INSTALL)) {
+installCmd = compCmd;
+ }
+}
+addInstallCommand(roleName, containerId, response, null,
+installCmd, timeout, nextPkgToInstall);
What if 'INSTALL' command isn't found in the for loop above ?
I'm following the same pattern for master package. Please refer to if
(command ==
[jira] [Updated] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-773: -- Attachment: coprocessor.patch Merged Gour's latest change Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: Co-processorSupport.pdf, coprocessor-apri-4th.patch, coprocessor.patch, coprocessor.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-773: -- Attachment: coprocessor.patch Modify the code according to Steve's comments. Will check in code changes of test cases later Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: Co-processorSupport.pdf, coprocessor-apri-4th.patch, coprocessor.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-773: -- Attachment: (was: coprocessor.patch) Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: Co-processorSupport.pdf, coprocessor-apri-4th.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-773: -- Attachment: coprocessor.patch Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: Co-processorSupport.pdf, coprocessor-apri-4th.patch, coprocessor.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Description: Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist was: Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist Add unit and functional test cases -- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Fix For: Slider 0.80 Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * create application before master package installed * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-847) Add unit and functional test cases
thomas liu created SLIDER-847: - Summary: Add unit and functional test cases Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Reporter: thomas liu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLIDER-847) Add unit and functional test cases
[ https://issues.apache.org/jira/browse/SLIDER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomas liu updated SLIDER-847: -- Description: Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist Add unit and functional test cases -- Key: SLIDER-847 URL: https://issues.apache.org/jira/browse/SLIDER-847 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Fix For: Slider 0.80 Scenarios: Invalid inputs: * --addon option missing package name or zip file path * --addon zip file path doesn't exist * --addon package definition file is not a zipped file * --addon invalid package name * package zipped file doesn't contain metainfo.json/xml * package zipped file metainfo.json/xml isn't valid * python scripts in metainfo.json/xml don't exist -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SLIDER-773) Add co-processor support for app packages
[ https://issues.apache.org/jira/browse/SLIDER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388852#comment-14388852 ] thomas liu edited comment on SLIDER-773 at 4/7/15 6:48 PM: --- How to package and submit add on packages: When users would like to submit an application composed of master package and add on packages, they need to provide the master package(the existing application package) without modification as it is, and also the add on packages. The addon packages can be created the same way as master package. For submission, they still use 'slider create' command, with additional parameters: --addon package name package zip file location in local FS. The detail of packaging, especially the schema of appConfig.json, metainfo.json are defined in https://issues.apache.org/jira/browse/SLIDER-834 Current design: We are not modifying the basic architecture of Slider, either how SliderAgent communicate with SliderAM. We modify the state machine in AgentProviderService to issue INSTALL commands for add on packages. Details here: https://issues.apache.org/jira/browse/SLIDER-820 was (Author: thomas_liu): How to package and submit add on packages: Users need to create Slider application package for addon packages in a way similar to master package. The detail of packaging, especially the schema of appConfig.json, metainfo.json are defined in https://issues.apache.org/jira/browse/SLIDER-834 Current design: We are not modifying the basic architecture of Slider, either how SliderAgent communicate with SliderAM. We modify the state machine in AgentProviderService to issue INSTALL commands for add on packages. Details here: https://issues.apache.org/jira/browse/SLIDER-820 Add co-processor support for app packages - Key: SLIDER-773 URL: https://issues.apache.org/jira/browse/SLIDER-773 Project: Slider Issue Type: Bug Components: app-package, client Affects Versions: Slider 0.60 Reporter: Sumit Mohanty Assignee: thomas liu Priority: Critical Fix For: Slider 0.80 Attachments: Co-processorSupport.pdf, coprocessor-apri-4th.patch It is typical for applications to allow plugins/co-processors that are essentially a set of additional jar files in the classpath and optionally a set of config files or config changes. Current, slider app packages can handle additional config changes/entries very well. Additional configs files can be added as well but it is not easy if the config files include parameters that need to be resolved by the agent. This requires app package changes. Dropping additional jar files into the class path is not easy and requires app package changes. It is not efficient to modify the app package to support such plugins. App packaging and create command should be modified such that the user can dynamically specify additional jars, config files, configs etc. Specific scenarios are modifying HBase to add support for Phoenix or Ranger. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-834) Read addon package configurations in appConfig.json and metainfo.json into AgentProviderService
[ https://issues.apache.org/jira/browse/SLIDER-834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483765#comment-14483765 ] thomas liu commented on SLIDER-834: --- When users would like to submit an application composed of master package and add on packages, they need to provide the master package(the existing application package) without modification as it is, and also the add on packages. The addon packages can be created the same way as master package, which is composed of: metainfo.json/xml package folder package/script folder, in which they provide python scripts similar to those in the master package to instruct Slider how to install/configure the addon package. They should be zipped the same way as the master package. So now the user have: appConfig.json resources.json master package zip file addon package zip file Now when the users submit the application, they still use 'slider create' command, with additional parameters: --addon package name package zip file location in local FS. For example: if they want to submit an HBASE application with Phoenix and Ranger, they need to call: slider create hbase01 --template /Users/yarn/app/hbase/appConfig-default.json --resources /Users/yarn/app/hbase/resources-default.json --addon PHOENIX /Users/yarn/app/phoenix/Archive.zip --addon RANGER /Users/yarn/app/ranger/Archive.zip And that's all what they need Read addon package configurations in appConfig.json and metainfo.json into AgentProviderService --- Key: SLIDER-834 URL: https://issues.apache.org/jira/browse/SLIDER-834 Project: Slider Issue Type: Sub-task Components: app-package, client Reporter: thomas liu Fix For: Slider 0.80 Example metainfo.xml/json schema: metainfo schemaVersion2.0/schemaVersion package namePHOENIX/name comment Apache Phoenix is ... /comment version.../version typeADDON-PACKAGE/type minHadoopVersionXXX/minHadoopVersion components component nameALL/name commandScript scriptscripts/start_phoenix.py/script scriptTypePYTHON/scriptType timeout600/timeout /commandScript /component component nameHBASE_MASTER/name commandScript scriptscripts/start_phoenix_for_hbase_master.py/script scriptTypePYTHON/scriptType /commandScript /component ... /components configFiles configFile typexml/type fileNamephoenix.xml/fileName dictionaryNamephoenix/dictionaryName /configFile ... /configFiles /package /metainfo -- This message was sent by Atlassian JIRA (v6.3.4#6332)
