date:20161005

[jira] [Commented] (AURORA-1107) Add support for mounting task specified external volumes into containers

2016-10-05 Thread Zameer Manji (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15550496#comment-15550496
 ] 

Zameer Manji commented on AURORA-1107:
--

I'm breaking this up in to two patches. The first is the API and storage 
changes: https://reviews.apache.org/r/52588/

I will follow up with DSL changes and an e2e test.

> Add support for mounting task specified external volumes into containers
> 
>
> Key: AURORA-1107
> URL: https://issues.apache.org/jira/browse/AURORA-1107
> Project: Aurora
>  Issue Type: Task
>  Components: Docker
>Reporter: Steve Niemitz
>Assignee: Zameer Manji
>Priority: Minor
>
> The Mesos docker API allows specifying volumes on the host to mount into the 
> container when it runs.  We should expose this.  I propose:
>  - Add a volumes() set to the Docker object in base.py
>  - Add a similar set to the DockerContainer struct in api.thrift 
>  - Create a way for administrators to restrict the ability to use this.  
> Because mounts are set up by the docker daemon, they effectively allow 
> someone who can configure mounts to access anything on the machine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AURORA-1712) Debian Jessie packagaes are embedding the mesos egg build for Ubuntu trusty

2016-10-05 Thread Zameer Manji (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549776#comment-15549776
 ] 

Zameer Manji commented on AURORA-1712:
--

{noformat}
commit 640f07babdf6307a6371f04b777209a1c9eca5ce
Author: Renan DelValle 
Date:   Wed Oct 5 12:56:44 2016 -0700

Build Debian Jessie and Ubuntu Xenial specific mesos eggs

Support for making mesos native python eggs for Debian Jessie and Ubuntu 
Xenial.

Testing Done:
./make-mesos-native-egg jessie64 1.0.0 $(pwd)/test
./make-mesos-native-egg xenial64 1.0.0 $(pwd)/test

*NOTE: Have not tried installing the python eggs generated.

Bugs closed: AURORA-1712

Reviewed at https://reviews.apache.org/r/52531/

 build-support/python/make-mesos-native-egg | 71 +-
 1 file changed, 69 insertions(+), 2 deletions(-)
{noformat}

Thank you [~rdelvalle] for fixing this. This could have caused a nasty issue if 
there was ABI divergence.

> Debian Jessie packagaes are embedding the mesos egg build for Ubuntu trusty
> ---
>
> Key: AURORA-1712
> URL: https://issues.apache.org/jira/browse/AURORA-1712
> Project: Aurora
>  Issue Type: Bug
>Reporter: Stephan Erb
>Assignee: Renan DelValle
>
> The Debian packaging scripts for Trusty and Jessie are sharing the same 
> override mechanism for the pants third_party repository. We therefore end up  
> using egg-files build for Ubuntu also on Debian 
> (https://github.com/apache/aurora-packaging/blob/master/specs/debian/aurora-pants.ini)
> It seems like this is kind of working, but is clearly not optimal.
> We should extend 
> https://github.com/apache/aurora/blob/master/build-support/python/make-mesos-native-egg
>  to support Debian and then make use of it in our packaging infrastructure 
> https://github.com/apache/aurora-packaging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (AURORA-1787) `-global_container_mounts` does not appear to work with the unified containerizer

2016-10-05 Thread Zameer Manji (JIRA)


 [ 
https://issues.apache.org/jira/browse/AURORA-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zameer Manji resolved AURORA-1787.
--
Resolution: Not A Problem

Thanks for the clarification. I was able to get everything to work as intended.

> `-global_container_mounts` does not appear to work with the unified 
> containerizer
> -
>
> Key: AURORA-1787
> URL: https://issues.apache.org/jira/browse/AURORA-1787
> Project: Aurora
>  Issue Type: Bug
>Reporter: Zameer Manji
>Priority: Critical
>
> Perhaps I misunderstand how this feature is supposed to be used, but apply 
> the following patch to master:
> {noformat}
> From 1ebb5f4c5815c647e31f3253d5e5c316a0d5edd2 Mon Sep 17 00:00:00 2001
> From: Zameer Manji 
> Date: Tue, 4 Oct 2016 20:45:41 -0700
> Subject: [PATCH] Reproduce the issue.
> ---
>  examples/vagrant/upstart/aurora-scheduler.conf |  2 +-
>  src/test/sh/org/apache/aurora/e2e/run-server.sh|  4 
>  .../sh/org/apache/aurora/e2e/test_end_to_end.sh| 26 
> +++---
>  3 files changed, 18 insertions(+), 14 deletions(-)
> diff --git a/examples/vagrant/upstart/aurora-scheduler.conf 
> b/examples/vagrant/upstart/aurora-scheduler.conf
> index 91b27d7..851b5a1 100644
> --- a/examples/vagrant/upstart/aurora-scheduler.conf
> +++ b/examples/vagrant/upstart/aurora-scheduler.conf
> @@ -40,7 +40,7 @@ exec bin/aurora-scheduler \
>-native_log_file_path=/var/db/aurora \
>-backup_dir=/var/lib/aurora/backups \
>-thermos_executor_path=$DIST_DIR/thermos_executor.pex \
> -  
> -global_container_mounts=/home/vagrant/aurora/examples/vagrant/config:/home/vagrant/aurora/examples/vagrant/config:ro
>  \
> +  -global_container_mounts=/etc/rsyslog.d:rsyslog.d.container:ro \
>-thermos_executor_flags="--announcer-ensemble localhost:2181 
> --announcer-zookeeper-auth-config 
> /home/vagrant/aurora/examples/vagrant/config/announcer-auth.json 
> --mesos-containerizer-path=/usr/libexec/mesos/mesos-containerizer" \
>-allowed_container_types=MESOS,DOCKER \
>-http_authentication_mechanism=BASIC \
> diff --git a/src/test/sh/org/apache/aurora/e2e/run-server.sh 
> b/src/test/sh/org/apache/aurora/e2e/run-server.sh
> index 1fe0909..a0ee76f 100755
> --- a/src/test/sh/org/apache/aurora/e2e/run-server.sh
> +++ b/src/test/sh/org/apache/aurora/e2e/run-server.sh
> @@ -1,6 +1,10 @@
>  #!/bin/bash
>  
>  echo "Starting up server..."
> +if [ ! -d "./rsyslog.d.container" ]; then
> +  echo "Mountpoint Doesn't Exist";
> +  exit 1;
> +fi
>  while true
>  do
>echo -e "HTTP/1.1 200 OK\r\n\r\nHello from a filesystem image." | nc -l 
> "$1"
> diff --git a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 
> b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> index c93be9b..094d776 100755
> --- a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> +++ b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> @@ -514,27 +514,27 @@ trap collect_result EXIT
>  aurorabuild all
>  setup_ssh
>  
> -test_version
> -test_http_example "${TEST_JOB_ARGS[@]}"
> -test_health_check
> +# test_version
> +# test_http_example "${TEST_JOB_ARGS[@]}"
> +# test_health_check
>  
> -test_http_example_basic "${TEST_JOB_REVOCABLE_ARGS[@]}"
> +# test_http_example_basic "${TEST_JOB_REVOCABLE_ARGS[@]}"
>  
> -test_http_example_basic "${TEST_JOB_GPU_ARGS[@]}"
> +# test_http_example_basic "${TEST_JOB_GPU_ARGS[@]}"
>  
>  # build the test docker image
> -sudo docker build -t http_example -f "${TEST_ROOT}/Dockerfile.python" 
> ${TEST_ROOT}
> -test_http_example "${TEST_JOB_DOCKER_ARGS[@]}"
> +# sudo docker build -t http_example -f "${TEST_ROOT}/Dockerfile.python" 
> ${TEST_ROOT}
> +# test_http_example "${TEST_JOB_DOCKER_ARGS[@]}"
>  
>  setup_image_stores
>  test_appc_unified
> -test_docker_unified
> +# test_docker_unified
>  
> -test_admin "${TEST_ADMIN_ARGS[@]}"
> -test_basic_auth_unauthenticated  "${TEST_JOB_ARGS[@]}"
> +# test_admin "${TEST_ADMIN_ARGS[@]}"
> +# test_basic_auth_unauthenticated  "${TEST_JOB_ARGS[@]}"
>  
> -test_ephemeral_daemon_with_final 
> "${TEST_JOB_EPHEMERAL_DAEMON_WITH_FINAL_ARGS[@]}"
> +# test_ephemeral_daemon_with_final 
> "${TEST_JOB_EPHEMERAL_DAEMON_WITH_FINAL_ARGS[@]}"
>  
> -/vagrant/src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh
> -/vagrant/src/test/sh/org/apache/aurora/e2e/test_bypass_leader_redirect_end_to_end.sh
> +# /vagrant/src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh
> +# 
> /vagrant/src/test/sh/org/apache/aurora/e2e/test_bypass_leader_redirect_end_to_end.sh
>  RETCODE=0
> -- 
> 2.10.0
> {noformat}
> You can apply the patch by copying the content to a {{.patch}} file and 
> running {{git am < file.patch}}
> Run the e2e tests.
> Observe that the tests fail because the tasks fail. The tasks fail because 
> the mountpoint in their sandbox does n

[jira] [Commented] (AURORA-1787) `-global_container_mounts` does not appear to work with the unified containerizer

2016-10-05 Thread Joshua Cohen (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549442#comment-15549442
 ] 

Joshua Cohen commented on AURORA-1787:
--

Sorry, I skimmed over your full patch in the description and mostly answered 
the question based on our discussion in Slack last night.

The problem with the patch in the description is that the executor mounts 
volumes into the taskfs relative to the root of the filesystem, not relative to 
the sandbox. So when you change {{run-server.sh}} to look for 
{{./rsyslog.d.container}}, it's doing so relative to {{CWD}} which is 
{{/mnt/mesos/sandbox}}. If you look for {{/rsyslog.d.container}} it should work:

{noformat}
vagrant@aurora:~/util-linux-2.24.1$ sudo ./nsenter -m -t 13236
root@aurora:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  
rsyslog.d.container  run  sbin  srv  sys  tmp  usr  var
{noformat}

For what it's worth, if you look at the executor's logs you can also see 
exactly what it's doing with regards to mounting:

{noformat}
I1005 17:21:26.493562 13110 sandbox.py:282] Mounting rsyslog.d.container into 
task filesystem at 
/var/lib/mesos/slaves/54325286-97a6-41f3-b0ba-991cf1f733be-S0/frameworks/54325286-97a6-41f3-b0ba-991cf1f733be-/executors/thermos-vagrant-test-http_example_unified_docker-0-e1517a6c-1a48-46d0-9fc8-fdcf4d4caf75/runs/1bd5b092-d6a5-4993-a93f-03da7665b1da/taskfs/rsyslog.d.container.
{noformat}

> `-global_container_mounts` does not appear to work with the unified 
> containerizer
> -
>
> Key: AURORA-1787
> URL: https://issues.apache.org/jira/browse/AURORA-1787
> Project: Aurora
>  Issue Type: Bug
>Reporter: Zameer Manji
>Priority: Critical
>
> Perhaps I misunderstand how this feature is supposed to be used, but apply 
> the following patch to master:
> {noformat}
> From 1ebb5f4c5815c647e31f3253d5e5c316a0d5edd2 Mon Sep 17 00:00:00 2001
> From: Zameer Manji 
> Date: Tue, 4 Oct 2016 20:45:41 -0700
> Subject: [PATCH] Reproduce the issue.
> ---
>  examples/vagrant/upstart/aurora-scheduler.conf |  2 +-
>  src/test/sh/org/apache/aurora/e2e/run-server.sh|  4 
>  .../sh/org/apache/aurora/e2e/test_end_to_end.sh| 26 
> +++---
>  3 files changed, 18 insertions(+), 14 deletions(-)
> diff --git a/examples/vagrant/upstart/aurora-scheduler.conf 
> b/examples/vagrant/upstart/aurora-scheduler.conf
> index 91b27d7..851b5a1 100644
> --- a/examples/vagrant/upstart/aurora-scheduler.conf
> +++ b/examples/vagrant/upstart/aurora-scheduler.conf
> @@ -40,7 +40,7 @@ exec bin/aurora-scheduler \
>-native_log_file_path=/var/db/aurora \
>-backup_dir=/var/lib/aurora/backups \
>-thermos_executor_path=$DIST_DIR/thermos_executor.pex \
> -  
> -global_container_mounts=/home/vagrant/aurora/examples/vagrant/config:/home/vagrant/aurora/examples/vagrant/config:ro
>  \
> +  -global_container_mounts=/etc/rsyslog.d:rsyslog.d.container:ro \
>-thermos_executor_flags="--announcer-ensemble localhost:2181 
> --announcer-zookeeper-auth-config 
> /home/vagrant/aurora/examples/vagrant/config/announcer-auth.json 
> --mesos-containerizer-path=/usr/libexec/mesos/mesos-containerizer" \
>-allowed_container_types=MESOS,DOCKER \
>-http_authentication_mechanism=BASIC \
> diff --git a/src/test/sh/org/apache/aurora/e2e/run-server.sh 
> b/src/test/sh/org/apache/aurora/e2e/run-server.sh
> index 1fe0909..a0ee76f 100755
> --- a/src/test/sh/org/apache/aurora/e2e/run-server.sh
> +++ b/src/test/sh/org/apache/aurora/e2e/run-server.sh
> @@ -1,6 +1,10 @@
>  #!/bin/bash
>  
>  echo "Starting up server..."
> +if [ ! -d "./rsyslog.d.container" ]; then
> +  echo "Mountpoint Doesn't Exist";
> +  exit 1;
> +fi
>  while true
>  do
>echo -e "HTTP/1.1 200 OK\r\n\r\nHello from a filesystem image." | nc -l 
> "$1"
> diff --git a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 
> b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> index c93be9b..094d776 100755
> --- a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> +++ b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> @@ -514,27 +514,27 @@ trap collect_result EXIT
>  aurorabuild all
>  setup_ssh
>  
> -test_version
> -test_http_example "${TEST_JOB_ARGS[@]}"
> -test_health_check
> +# test_version
> +# test_http_example "${TEST_JOB_ARGS[@]}"
> +# test_health_check
>  
> -test_http_example_basic "${TEST_JOB_REVOCABLE_ARGS[@]}"
> +# test_http_example_basic "${TEST_JOB_REVOCABLE_ARGS[@]}"
>  
> -test_http_example_basic "${TEST_JOB_GPU_ARGS[@]}"
> +# test_http_example_basic "${TEST_JOB_GPU_ARGS[@]}"
>  
>  # build the test docker image
> -sudo docker build -t http_example -f "${TEST_ROOT}/Dockerfile.python" 
> ${TEST_ROOT}
> -test_http_example "${TEST_JOB_DOCKER_ARGS[@]}"
> +# sudo docker build -t http_exa

[jira] [Commented] (AURORA-1787) `-global_container_mounts` does not appear to work with the unified containerizer

2016-10-05 Thread Zameer Manji (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549299#comment-15549299
 ] 

Zameer Manji commented on AURORA-1787:
--

[~joshua.cohen]:

Just to be clear, all mountpoints must exist inside the image? I changed the 
configuration to:
{noformat}
>From 6f19dfcd71bb1dcf09c7fbd8a5f77b845deb49c8 Mon Sep 17 00:00:00 2001
From: Zameer Manji 
Date: Tue, 4 Oct 2016 20:45:41 -0700
Subject: [PATCH] Reproduce the issue.

---
 examples/vagrant/upstart/aurora-scheduler.conf |  2 +-
 src/test/sh/org/apache/aurora/e2e/run-server.sh|  4 
 .../sh/org/apache/aurora/e2e/test_end_to_end.sh| 26 +++---
 3 files changed, 18 insertions(+), 14 deletions(-)

diff --git a/examples/vagrant/upstart/aurora-scheduler.conf 
b/examples/vagrant/upstart/aurora-scheduler.conf
index 91b27d7..0ede9cb 100644
--- a/examples/vagrant/upstart/aurora-scheduler.conf
+++ b/examples/vagrant/upstart/aurora-scheduler.conf
@@ -40,7 +40,7 @@ exec bin/aurora-scheduler \
   -native_log_file_path=/var/db/aurora \
   -backup_dir=/var/lib/aurora/backups \
   -thermos_executor_path=$DIST_DIR/thermos_executor.pex \
-  
-global_container_mounts=/home/vagrant/aurora/examples/vagrant/config:/home/vagrant/aurora/examples/vagrant/config:ro
 \
+  -global_container_mounts=/etc/rsyslog.d:/etc/rsyslog.d.container:ro \
   -thermos_executor_flags="--announcer-ensemble localhost:2181 
--announcer-zookeeper-auth-config 
/home/vagrant/aurora/examples/vagrant/config/announcer-auth.json 
--mesos-containerizer-path=/usr/libexec/mesos/mesos-containerizer" \
   -allowed_container_types=MESOS,DOCKER \
   -http_authentication_mechanism=BASIC \
diff --git a/src/test/sh/org/apache/aurora/e2e/run-server.sh 
b/src/test/sh/org/apache/aurora/e2e/run-server.sh
index 1fe0909..0f078c4 100755
--- a/src/test/sh/org/apache/aurora/e2e/run-server.sh
+++ b/src/test/sh/org/apache/aurora/e2e/run-server.sh
@@ -1,6 +1,10 @@
 #!/bin/bash
 
 echo "Starting up server..."
+if [ ! -d "/etc/rsyslog.d.container" ]; then
+  echo "Mountpoint Doesn't Exist";
+  exit 1;
+fi
 while true
 do
   echo -e "HTTP/1.1 200 OK\r\n\r\nHello from a filesystem image." | nc -l "$1"
diff --git a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 
b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
index c93be9b..094d776 100755
--- a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
+++ b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
@@ -514,27 +514,27 @@ trap collect_result EXIT
 aurorabuild all
 setup_ssh
 
-test_version
-test_http_example "${TEST_JOB_ARGS[@]}"
-test_health_check
+# test_version
+# test_http_example "${TEST_JOB_ARGS[@]}"
+# test_health_check
 
-test_http_example_basic "${TEST_JOB_REVOCABLE_ARGS[@]}"
+# test_http_example_basic "${TEST_JOB_REVOCABLE_ARGS[@]}"
 
-test_http_example_basic "${TEST_JOB_GPU_ARGS[@]}"
+# test_http_example_basic "${TEST_JOB_GPU_ARGS[@]}"
 
 # build the test docker image
-sudo docker build -t http_example -f "${TEST_ROOT}/Dockerfile.python" 
${TEST_ROOT}
-test_http_example "${TEST_JOB_DOCKER_ARGS[@]}"
+# sudo docker build -t http_example -f "${TEST_ROOT}/Dockerfile.python" 
${TEST_ROOT}
+# test_http_example "${TEST_JOB_DOCKER_ARGS[@]}"
 
 setup_image_stores
 test_appc_unified
-test_docker_unified
+# test_docker_unified
 
-test_admin "${TEST_ADMIN_ARGS[@]}"
-test_basic_auth_unauthenticated  "${TEST_JOB_ARGS[@]}"
+# test_admin "${TEST_ADMIN_ARGS[@]}"
+# test_basic_auth_unauthenticated  "${TEST_JOB_ARGS[@]}"
 
-test_ephemeral_daemon_with_final 
"${TEST_JOB_EPHEMERAL_DAEMON_WITH_FINAL_ARGS[@]}"
+# test_ephemeral_daemon_with_final 
"${TEST_JOB_EPHEMERAL_DAEMON_WITH_FINAL_ARGS[@]}"
 
-/vagrant/src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh
-/vagrant/src/test/sh/org/apache/aurora/e2e/test_bypass_leader_redirect_end_to_end.sh
+# /vagrant/src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh
+# 
/vagrant/src/test/sh/org/apache/aurora/e2e/test_bypass_leader_redirect_end_to_end.sh
 RETCODE=0
-- 
2.10.0

{noformat}

And the task fails with: {{Failed to launch container: Failed to generate 
isolation script: Absolute container path '/etc/rsyslog.d.container' does not 
exist; Container destroyed while preparing isolators}}

Does this mean that the container path in a volume has to be a path that also 
exists on the host?

> `-global_container_mounts` does not appear to work with the unified 
> containerizer
> -
>
> Key: AURORA-1787
> URL: https://issues.apache.org/jira/browse/AURORA-1787
> Project: Aurora
>  Issue Type: Bug
>Reporter: Zameer Manji
>Priority: Critical
>
> Perhaps I misunderstand how this feature is supposed to be used, but apply 
> the following patch to master:
> {noformat}
> From 1ebb5f4c5815c647e31f3253d5e5c316a0d5edd2 Mon Sep 17

[jira] [Commented] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

2016-10-05 Thread Joshua Cohen (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549019#comment-15549019
 ] 

Joshua Cohen commented on AURORA-1782:
--

I *am* able to reproduce with the cmdline you provided above. Will investigate 
a fix.

> Thermos Executor is not handling apostrophe gracefully
> --
>
> Key: AURORA-1782
> URL: https://issues.apache.org/jira/browse/AURORA-1782
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
> Attachments: process with single quotes.png
>
>
> This is a regression from the behavior in 0.15.0.
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': Failed to load 
> value '{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error 
> at line 1 near: 
> {code}
> Ideally I could escape the apostrophe and have it work for Process's that run 
> 'chrooted/pivoted' by thermos or bare.  My work around has been to write a 
> templating hack for aurora's dsl.
> {code}
> # Template.render(name='foo', template='SOME_STRING_TEMPLATE')
> # Returns a Process().  I can't just base64 the entire template (that would 
> be too easy), 
> # b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace 
> apostrophe
> # with a pattern that is unlikely to be legitimately used and render a 
> 'cmdline' that will 
> # undo the hack.  Of course using the octal or "'" directly proved 
> troublesome, so I had to 
> # base64 encode the simple sed statement and decode the script on the remote 
> side.  Like
> # I said this is a complete hack.
> class Template(object):
>   class T(Struct):
> payload = Required(String)
>   @classmethod
>   def render(cls, name=None, template=None, **kwargs):
> assert name is not None
> assert template is not None
> return cls(name, template, 'template', **kwargs).process
>   def cmdline(self):
> return ''.join([
> '(mkdir .decoder && ',
> '(test -x ./.decoder/decoder || ',
> '((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) && ',
> 'chmod +x ./.decoder/decoder)) || ',
> "(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 
> 1; done)) && ",
> '((echo "{{template_payload}}" | ./.decoder/decoder) > 
> {{template_filename}})'
> ])
>   def decoder_script(self):
> return r"""#!/bin/bash
> sed "s/__APOSTROPHE__/'/g"
> """
>   def postInit(self):
> self.process.in_scope = self.in_scope  # need to override imethod
>   def __init__(self, name, template, prefix, filename=None, auto_init=True, 
> **kwargs):
> self.resolved = False
> self.process = Process(
> name="%s:%s" % (prefix, name),
> cmdline=self.cmdline(), **kwargs).bind(
> self.__class__.T(payload=template.replace("'", '__APOSTROPHE__')),
> template_filename=name if filename is None else filename,
> template_decoder=base64.b64encode(self.decoder_script()))
> if auto_init:
>   self.postInit()
>   def in_scope(self, *args, **kwargs):
> """ensure name/commandline is resolved before proceeding"""
> if self.resolved:
>   return self.process
> self.process = Process.in_scope(self.process, *args, **kwargs)
> if '{{' not in str(self.process.name()):
>   scopes = self.process.scopes()
>   for scope in scopes:
> if not hasattr(scope, 'bind'):
>   continue
> scope = scope.bind(**kwargs)
> if scope.check() and isinstance(scope, self.__class__.T):
>   self.resolved = True
>   payload = str(scope.payload())
>   print(" INFO] Memoizing {}".format(self.process.name()))
>   self.process = self.process.bind(template_payload=payload)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

2016-10-05 Thread Joshua Cohen (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549008#comment-15549008
 ] 

Joshua Cohen commented on AURORA-1782:
--

I just tried to replicate this locally and I was unable to do so. I made the 
following change to {{setup_env}} process in the [job used by our end to end 
tests](https://github.com/apache/aurora/blob/master/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora#L45-L50):

{noformat}
diff --git a/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
b/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
index c71fb81..3b75948 100644
--- a/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
+++ b/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
@@ -45,7 +45,7 @@ stage_server = Process(
 setup_env = Process(
   name = 'setup_env',
   cmdline='''cat < .thermos_profile
-export IT_WORKED=hello
+export IT_WORKED='hello'
 EOF'''
 )
{noformat}

And I was able to launch the job successfully (see attached screenshot). I'll 
need to dig more to see what's causing your issue.



> Thermos Executor is not handling apostrophe gracefully
> --
>
> Key: AURORA-1782
> URL: https://issues.apache.org/jira/browse/AURORA-1782
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
> Attachments: process with single quotes.png
>
>
> This is a regression from the behavior in 0.15.0.
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': Failed to load 
> value '{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error 
> at line 1 near: 
> {code}
> Ideally I could escape the apostrophe and have it work for Process's that run 
> 'chrooted/pivoted' by thermos or bare.  My work around has been to write a 
> templating hack for aurora's dsl.
> {code}
> # Template.render(name='foo', template='SOME_STRING_TEMPLATE')
> # Returns a Process().  I can't just base64 the entire template (that would 
> be too easy), 
> # b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace 
> apostrophe
> # with a pattern that is unlikely to be legitimately used and render a 
> 'cmdline' that will 
> # undo the hack.  Of course using the octal or "'" directly proved 
> troublesome, so I had to 
> # base64 encode the simple sed statement and decode the script on the remote 
> side.  Like
> # I said this is a complete hack.
> class Template(object):
>   class T(Struct):
> payload = Required(String)
>   @classmethod
>   def render(cls, name=None, template=None, **kwargs):
> assert name is not None
> assert template is not None
> return cls(name, template, 'template', **kwargs).process
>   def cmdline(self):
> return ''.join([
> '(mkdir .decoder && ',
> '(test -x ./.decoder/decoder || ',
> '((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) && ',
> 'chmod +x ./.decoder/decoder)) || ',
> "(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 
> 1; done)) && ",
> '((echo "{{template_payload}}" | ./.decoder/decoder) > 
> {{template_filename}})'
> ])
>   def decoder_script(self):
> return r"""#!/bin/bash
> sed "s/__APOSTROPHE__/'/g"
> """
>   def postInit(self):
> self.process.in_scope = self.in_scope  # need to override imethod
>   def __init__(self, name, template, prefix, filename=None, auto_init=True, 
> **kwargs):
> self.resolved = False
> self.process = Process(
> name="%s:%s" % (prefix, name),
> cmdline=self.cmdline(), **kwargs).bind(
> self.__class__.T(payload=template.replace("'", '__APOSTROPHE__')),
> template_filename=name if filename is None else filename,
> template_decoder=base64.b64encode(self.decoder_script()))
> if auto_init:
>   self.postInit()
>   def in_scope(self, *args, **kwargs):
> """ensure name/commandline is resolved before proceeding"""
> if self.resolved:
>   return self.process
> self.process = Process.in_scope(self.process, *args, **kwargs)
> if '{{' not in str(self.process.name()):
>   scopes = self.process.scopes()
>   for scope in scopes:
> if not hasattr(scope, 'bind'):
>   continue
> scope = scope.bind(**kwargs)
> if scope.check() and isinstance(scope, self.__class__.T):
>   self.resolved = True
>   payload = str(scope.payload())
>   print(" INFO] Memoizing {}".format(self.process.name()))
>   self.process = self.process.bind(template_payload=payload)
> {code}



--
This mes

[jira] [Updated] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

2016-10-05 Thread Joshua Cohen (JIRA)


 [ 
https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua Cohen updated AURORA-1782:
-
Attachment: process with single quotes.png

> Thermos Executor is not handling apostrophe gracefully
> --
>
> Key: AURORA-1782
> URL: https://issues.apache.org/jira/browse/AURORA-1782
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
> Attachments: process with single quotes.png
>
>
> This is a regression from the behavior in 0.15.0.
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': Failed to load 
> value '{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error 
> at line 1 near: 
> {code}
> Ideally I could escape the apostrophe and have it work for Process's that run 
> 'chrooted/pivoted' by thermos or bare.  My work around has been to write a 
> templating hack for aurora's dsl.
> {code}
> # Template.render(name='foo', template='SOME_STRING_TEMPLATE')
> # Returns a Process().  I can't just base64 the entire template (that would 
> be too easy), 
> # b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace 
> apostrophe
> # with a pattern that is unlikely to be legitimately used and render a 
> 'cmdline' that will 
> # undo the hack.  Of course using the octal or "'" directly proved 
> troublesome, so I had to 
> # base64 encode the simple sed statement and decode the script on the remote 
> side.  Like
> # I said this is a complete hack.
> class Template(object):
>   class T(Struct):
> payload = Required(String)
>   @classmethod
>   def render(cls, name=None, template=None, **kwargs):
> assert name is not None
> assert template is not None
> return cls(name, template, 'template', **kwargs).process
>   def cmdline(self):
> return ''.join([
> '(mkdir .decoder && ',
> '(test -x ./.decoder/decoder || ',
> '((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) && ',
> 'chmod +x ./.decoder/decoder)) || ',
> "(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 
> 1; done)) && ",
> '((echo "{{template_payload}}" | ./.decoder/decoder) > 
> {{template_filename}})'
> ])
>   def decoder_script(self):
> return r"""#!/bin/bash
> sed "s/__APOSTROPHE__/'/g"
> """
>   def postInit(self):
> self.process.in_scope = self.in_scope  # need to override imethod
>   def __init__(self, name, template, prefix, filename=None, auto_init=True, 
> **kwargs):
> self.resolved = False
> self.process = Process(
> name="%s:%s" % (prefix, name),
> cmdline=self.cmdline(), **kwargs).bind(
> self.__class__.T(payload=template.replace("'", '__APOSTROPHE__')),
> template_filename=name if filename is None else filename,
> template_decoder=base64.b64encode(self.decoder_script()))
> if auto_init:
>   self.postInit()
>   def in_scope(self, *args, **kwargs):
> """ensure name/commandline is resolved before proceeding"""
> if self.resolved:
>   return self.process
> self.process = Process.in_scope(self.process, *args, **kwargs)
> if '{{' not in str(self.process.name()):
>   scopes = self.process.scopes()
>   for scope in scopes:
> if not hasattr(scope, 'bind'):
>   continue
> scope = scope.bind(**kwargs)
> if scope.check() and isinstance(scope, self.__class__.T):
>   self.resolved = True
>   payload = str(scope.payload())
>   print(" INFO] Memoizing {}".format(self.process.name()))
>   self.process = self.process.bind(template_payload=payload)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

2016-10-05 Thread Joshua Cohen (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549008#comment-15549008
 ] 

Joshua Cohen edited comment on AURORA-1782 at 10/5/16 3:02 PM:
---

I just tried to replicate this locally and I was unable to do so. I made the 
following change to {{setup_env}} process in the [job used by our end to end 
tests|https://github.com/apache/aurora/blob/master/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora#L45-L50]:

{noformat}
diff --git a/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
b/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
index c71fb81..3b75948 100644
--- a/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
+++ b/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
@@ -45,7 +45,7 @@ stage_server = Process(
 setup_env = Process(
   name = 'setup_env',
   cmdline='''cat < .thermos_profile
-export IT_WORKED=hello
+export IT_WORKED='hello'
 EOF'''
 )
{noformat}

And I was able to launch the job successfully (see attached screenshot). I'll 
need to dig more to see what's causing your issue.




was (Author: joshua.cohen):
I just tried to replicate this locally and I was unable to do so. I made the 
following change to {{setup_env}} process in the [job used by our end to end 
tests](https://github.com/apache/aurora/blob/master/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora#L45-L50):

{noformat}
diff --git a/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
b/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
index c71fb81..3b75948 100644
--- a/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
+++ b/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
@@ -45,7 +45,7 @@ stage_server = Process(
 setup_env = Process(
   name = 'setup_env',
   cmdline='''cat < .thermos_profile
-export IT_WORKED=hello
+export IT_WORKED='hello'
 EOF'''
 )
{noformat}

And I was able to launch the job successfully (see attached screenshot). I'll 
need to dig more to see what's causing your issue.



> Thermos Executor is not handling apostrophe gracefully
> --
>
> Key: AURORA-1782
> URL: https://issues.apache.org/jira/browse/AURORA-1782
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
> Attachments: process with single quotes.png
>
>
> This is a regression from the behavior in 0.15.0.
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': Failed to load 
> value '{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error 
> at line 1 near: 
> {code}
> Ideally I could escape the apostrophe and have it work for Process's that run 
> 'chrooted/pivoted' by thermos or bare.  My work around has been to write a 
> templating hack for aurora's dsl.
> {code}
> # Template.render(name='foo', template='SOME_STRING_TEMPLATE')
> # Returns a Process().  I can't just base64 the entire template (that would 
> be too easy), 
> # b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace 
> apostrophe
> # with a pattern that is unlikely to be legitimately used and render a 
> 'cmdline' that will 
> # undo the hack.  Of course using the octal or "'" directly proved 
> troublesome, so I had to 
> # base64 encode the simple sed statement and decode the script on the remote 
> side.  Like
> # I said this is a complete hack.
> class Template(object):
>   class T(Struct):
> payload = Required(String)
>   @classmethod
>   def render(cls, name=None, template=None, **kwargs):
> assert name is not None
> assert template is not None
> return cls(name, template, 'template', **kwargs).process
>   def cmdline(self):
> return ''.join([
> '(mkdir .decoder && ',
> '(test -x ./.decoder/decoder || ',
> '((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) && ',
> 'chmod +x ./.decoder/decoder)) || ',
> "(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 
> 1; done)) && ",
> '((echo "{{template_payload}}" | ./.decoder/decoder) > 
> {{template_filename}})'
> ])
>   def decoder_script(self):
> return r"""#!/bin/bash
> sed "s/__APOSTROPHE__/'/g"
> """
>   def postInit(self):
> self.process.in_scope = self.in_scope  # need to override imethod
>   def __init__(self, name, template, prefix, filename=None, auto_init=True, 
> **kwargs):
> self.resolved = False
> self.process = Process(
> name="%s:%s" % (prefix, name),
> cmdline=self.cmdline(), **kwargs).bind(
> self.__class__.T

[jira] [Commented] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

2016-10-05 Thread Justin Venus (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548986#comment-15548986
 ] 

Justin Venus commented on AURORA-1782:
--

[~joshua.cohen] could the 'cmdline' be base64 encoded (after pystachio 
templating has resolved as much as it can) and sent to the executor?  I would 
imagine it should be fairly straight forward for the executor to detect and 
decode a base64 string.

Does this approach seem reasonable? 

> Thermos Executor is not handling apostrophe gracefully
> --
>
> Key: AURORA-1782
> URL: https://issues.apache.org/jira/browse/AURORA-1782
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
>
> This is a regression from the behavior in 0.15.0.
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': Failed to load 
> value '{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error 
> at line 1 near: 
> {code}
> Ideally I could escape the apostrophe and have it work for Process's that run 
> 'chrooted/pivoted' by thermos or bare.  My work around has been to write a 
> templating hack for aurora's dsl.
> {code}
> # Template.render(name='foo', template='SOME_STRING_TEMPLATE')
> # Returns a Process().  I can't just base64 the entire template (that would 
> be too easy), 
> # b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace 
> apostrophe
> # with a pattern that is unlikely to be legitimately used and render a 
> 'cmdline' that will 
> # undo the hack.  Of course using the octal or "'" directly proved 
> troublesome, so I had to 
> # base64 encode the simple sed statement and decode the script on the remote 
> side.  Like
> # I said this is a complete hack.
> class Template(object):
>   class T(Struct):
> payload = Required(String)
>   @classmethod
>   def render(cls, name=None, template=None, **kwargs):
> assert name is not None
> assert template is not None
> return cls(name, template, 'template', **kwargs).process
>   def cmdline(self):
> return ''.join([
> '(mkdir .decoder && ',
> '(test -x ./.decoder/decoder || ',
> '((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) && ',
> 'chmod +x ./.decoder/decoder)) || ',
> "(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 
> 1; done)) && ",
> '((echo "{{template_payload}}" | ./.decoder/decoder) > 
> {{template_filename}})'
> ])
>   def decoder_script(self):
> return r"""#!/bin/bash
> sed "s/__APOSTROPHE__/'/g"
> """
>   def postInit(self):
> self.process.in_scope = self.in_scope  # need to override imethod
>   def __init__(self, name, template, prefix, filename=None, auto_init=True, 
> **kwargs):
> self.resolved = False
> self.process = Process(
> name="%s:%s" % (prefix, name),
> cmdline=self.cmdline(), **kwargs).bind(
> self.__class__.T(payload=template.replace("'", '__APOSTROPHE__')),
> template_filename=name if filename is None else filename,
> template_decoder=base64.b64encode(self.decoder_script()))
> if auto_init:
>   self.postInit()
>   def in_scope(self, *args, **kwargs):
> """ensure name/commandline is resolved before proceeding"""
> if self.resolved:
>   return self.process
> self.process = Process.in_scope(self.process, *args, **kwargs)
> if '{{' not in str(self.process.name()):
>   scopes = self.process.scopes()
>   for scope in scopes:
> if not hasattr(scope, 'bind'):
>   continue
> scope = scope.bind(**kwargs)
> if scope.check() and isinstance(scope, self.__class__.T):
>   self.resolved = True
>   payload = str(scope.payload())
>   print(" INFO] Memoizing {}".format(self.process.name()))
>   self.process = self.process.bind(template_payload=payload)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

2016-10-05 Thread Joshua Cohen (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548939#comment-15548939
 ] 

Joshua Cohen commented on AURORA-1782:
--

Yes, I agree that this is definitely a major problem. There should be no 
changes required to your {{Process}} definitions to run them within a 
filesystem image. I'm unsure at this time what the fix is in the executor, but 
I'm fairly confident it can be fixed.

> Thermos Executor is not handling apostrophe gracefully
> --
>
> Key: AURORA-1782
> URL: https://issues.apache.org/jira/browse/AURORA-1782
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
>
> This is a regression from the behavior in 0.15.0.
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': Failed to load 
> value '{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error 
> at line 1 near: 
> {code}
> Ideally I could escape the apostrophe and have it work for Process's that run 
> 'chrooted/pivoted' by thermos or bare.  My work around has been to write a 
> templating hack for aurora's dsl.
> {code}
> # Template.render(name='foo', template='SOME_STRING_TEMPLATE')
> # Returns a Process().  I can't just base64 the entire template (that would 
> be too easy), 
> # b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace 
> apostrophe
> # with a pattern that is unlikely to be legitimately used and render a 
> 'cmdline' that will 
> # undo the hack.  Of course using the octal or "'" directly proved 
> troublesome, so I had to 
> # base64 encode the simple sed statement and decode the script on the remote 
> side.  Like
> # I said this is a complete hack.
> class Template(object):
>   class T(Struct):
> payload = Required(String)
>   @classmethod
>   def render(cls, name=None, template=None, **kwargs):
> assert name is not None
> assert template is not None
> return cls(name, template, 'template', **kwargs).process
>   def cmdline(self):
> return ''.join([
> '(mkdir .decoder && ',
> '(test -x ./.decoder/decoder || ',
> '((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) && ',
> 'chmod +x ./.decoder/decoder)) || ',
> "(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 
> 1; done)) && ",
> '((echo "{{template_payload}}" | ./.decoder/decoder) > 
> {{template_filename}})'
> ])
>   def decoder_script(self):
> return r"""#!/bin/bash
> sed "s/__APOSTROPHE__/'/g"
> """
>   def postInit(self):
> self.process.in_scope = self.in_scope  # need to override imethod
>   def __init__(self, name, template, prefix, filename=None, auto_init=True, 
> **kwargs):
> self.resolved = False
> self.process = Process(
> name="%s:%s" % (prefix, name),
> cmdline=self.cmdline(), **kwargs).bind(
> self.__class__.T(payload=template.replace("'", '__APOSTROPHE__')),
> template_filename=name if filename is None else filename,
> template_decoder=base64.b64encode(self.decoder_script()))
> if auto_init:
>   self.postInit()
>   def in_scope(self, *args, **kwargs):
> """ensure name/commandline is resolved before proceeding"""
> if self.resolved:
>   return self.process
> self.process = Process.in_scope(self.process, *args, **kwargs)
> if '{{' not in str(self.process.name()):
>   scopes = self.process.scopes()
>   for scope in scopes:
> if not hasattr(scope, 'bind'):
>   continue
> scope = scope.bind(**kwargs)
> if scope.check() and isinstance(scope, self.__class__.T):
>   self.resolved = True
>   payload = str(scope.payload())
>   print(" INFO] Memoizing {}".format(self.process.name()))
>   self.process = self.process.bind(template_payload=payload)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

2016-10-05 Thread Justin Venus (JIRA)


 [ 
https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Venus updated AURORA-1782:
-
Description: 
This is a regression from the behavior in 0.15.0.

This is enough to cause an execution to fail.
{code}
python -c 'import sys

if __name__ == "__main__":
  sys.exit(0)'
{code}

This is the error seen when running an inline script from a Process().
{code}
Failed to parse the flags: Failed to load flag 'command': Failed to load value 
'{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error at line 1 
near: 
{code}


Ideally I could escape the apostrophe and have it work for Process's that run 
'chrooted/pivoted' by thermos or bare.  My work around has been to write a 
templating hack for aurora's dsl.

{code}
# Template.render(name='foo', template='SOME_STRING_TEMPLATE')
# Returns a Process().  I can't just base64 the entire template (that would be 
too easy), 
# b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace 
apostrophe
# with a pattern that is unlikely to be legitimately used and render a 
'cmdline' that will 
# undo the hack.  Of course using the octal or "'" directly proved troublesome, 
so I had to 
# base64 encode the simple sed statement and decode the script on the remote 
side.  Like
# I said this is a complete hack.

class Template(object):

  class T(Struct):
payload = Required(String)

  @classmethod
  def render(cls, name=None, template=None, **kwargs):
assert name is not None
assert template is not None
return cls(name, template, 'template', **kwargs).process

  def cmdline(self):
return ''.join([
'(mkdir .decoder && ',
'(test -x ./.decoder/decoder || ',
'((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) && ',
'chmod +x ./.decoder/decoder)) || ',
"(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 
1; done)) && ",
'((echo "{{template_payload}}" | ./.decoder/decoder) > 
{{template_filename}})'
])

  def decoder_script(self):
return r"""#!/bin/bash
sed "s/__APOSTROPHE__/'/g"
"""

  def postInit(self):
self.process.in_scope = self.in_scope  # need to override imethod

  def __init__(self, name, template, prefix, filename=None, auto_init=True, 
**kwargs):
self.resolved = False

self.process = Process(
name="%s:%s" % (prefix, name),
cmdline=self.cmdline(), **kwargs).bind(
self.__class__.T(payload=template.replace("'", '__APOSTROPHE__')),
template_filename=name if filename is None else filename,
template_decoder=base64.b64encode(self.decoder_script()))
if auto_init:
  self.postInit()

  def in_scope(self, *args, **kwargs):
"""ensure name/commandline is resolved before proceeding"""
if self.resolved:
  return self.process
self.process = Process.in_scope(self.process, *args, **kwargs)
if '{{' not in str(self.process.name()):
  scopes = self.process.scopes()
  for scope in scopes:
if not hasattr(scope, 'bind'):
  continue
scope = scope.bind(**kwargs)
if scope.check() and isinstance(scope, self.__class__.T):
  self.resolved = True
  payload = str(scope.payload())
  print(" INFO] Memoizing {}".format(self.process.name()))
  self.process = self.process.bind(template_payload=payload)
{code}

  was:
When using mesos containerizer with an image, the code path taken appears to be 
stripping new line characters making it impossible to render files with 
'/bin/echo' or execute inline python scripts.  I suspect this is due to the 
introduction of shlex.split() in the method wrapped_cmdline() in the file 
src/main/python/apache/thermos/core/process.py, but I haven't investigated any 
further.

This is a regression from the behavior in 0.15.0.   


This is enough to cause an execution to fail.
{code}
python -c 'import sys

if __name__ == "__main__":
  sys.exit(0)'
{code}

This is the error seen when running an inline script from a Process().
{code}
Failed to parse the flags: Failed to load flag 'command': Failed to load value 
'{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error at line 1 
near: 
{code}



> Thermos Executor is not handling apostrophe gracefully
> --
>
> Key: AURORA-1782
> URL: https://issues.apache.org/jira/browse/AURORA-1782
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
>
> This is a regression from the behavior in 0.15.0.
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': F

[jira] [Updated] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

2016-10-05 Thread Justin Venus (JIRA)


 [ 
https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Venus updated AURORA-1782:
-
Summary: Thermos Executor is not handling apostrophe gracefully  (was: 
Thermos Executor is ignoring shell escapes in Unified Containerizer code path)

> Thermos Executor is not handling apostrophe gracefully
> --
>
> Key: AURORA-1782
> URL: https://issues.apache.org/jira/browse/AURORA-1782
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
>
> When using mesos containerizer with an image, the code path taken appears to 
> be stripping new line characters making it impossible to render files with 
> '/bin/echo' or execute inline python scripts.  I suspect this is due to the 
> introduction of shlex.split() in the method wrapped_cmdline() in the file 
> src/main/python/apache/thermos/core/process.py, but I haven't investigated 
> any further.
> This is a regression from the behavior in 0.15.0.   
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': Failed to load 
> value '{"shell":true,"value":"/bin/bash -c 'python -c import': syntax error 
> at line 1 near: 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AURORA-1781) Sandbox taskfs setup fails (groupadd error)

2016-10-05 Thread Justin Venus (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548852#comment-15548852
 ] 

Justin Venus commented on AURORA-1781:
--

[~joshua.cohen] yes selinux is enabled.
{code}
CoreOS stable (1068.9.0)
Last login: Wed Oct  5 14:20:03 2016 from 10.111.254.195
Update Strategy: No Reboots
jvenus@mesos-slave03of2 ~ $ sestatus
SELinux status: enabled
SELinuxfs mount:/sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: mcs
Current mode:   permissive
Mode from config file:  permissive
Policy MLS status:  enabled
Policy deny_unknown status: allowed
Max kernel policy version:  30
{code}

> Sandbox taskfs setup fails (groupadd error)
> ---
>
> Key: AURORA-1781
> URL: https://issues.apache.org/jira/browse/AURORA-1781
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
>
> I hit what smells like a permission issue w/ `/etc/group` when trying to use 
> a docker-image (unified containerizer setup) with mesos-1.0.0. and 
> aurora-0.16.0-rc2.  I cannot reproduce issue w/ mesos-0.28.2 and aurora-015.0.
> {code}
> Failed to initialize sandbox: Failed to create group in sandbox for task 
> image: Command '['groupadd', '-R', 
> '/var/lib/mesos/slaves/5d28d0cc-2793-4471-82d5-e67276c53f70-S2/frameworks/20160221-001235-3801519626-5050-1-/executors/thermos-nobody-prod-jenkins-0-47cc7824-565b-4265-9ab4-9ba3f364ebed/runs/a3f78288-4865-4166-8685-1ad941562f2f/taskfs',
>  '-g', '99', 'nobody']' returned non-zero exit status 10
> {code}
> {code}
> [root@mesos-master01of2 taskfs]# pwd
> /var/lib/mesos/slaves/5d28d0cc-2793-4471-82d5-e67276c53f70-S2/frameworks/20160221-001235-3801519626-5050-1-/executors/thermos-nobody-prod-jenkins-0-47cc7824-565b-4265-9ab4-9ba3f364ebed/runs/a3f78288-4865-4166-8685-1ad941562f2f/taskfs
> [root@mesos-master01of2 taskfs]# groupadd -R $PWD -g 99 nobody
> groupadd: cannot lock /etc/group; try again later.
> {code}
> Maybe related to AURORA-1761
> I'm running CoreOS with the mesos-agent (and thermos) inside docker.  Here is 
> the gist of how it's started.
> {code}
> /usr/bin/sh -c "exec /usr/bin/docker run \
> --name=mesos_slave \
> --net=host \
> --pid=host \
> --privileged \
> -v /sys:/sys \
> -v /usr/bin/docker:/usr/bin/docker:ro \
> -v /var/lib/docker:/var/lib/docker \
> -v /var/run/docker.sock:/root/docker.sock \
> -v /run/systemd/system:/run/systemd/system \
> -v /lib64/libdevmapper.so.1.02:/lib/libdevmapper.so.1.02:ro \
> -v /sys/fs/cgroup:/sys/fs/cgroup \
> -v /var/lib/mesos:/var/lib/mesos \
> -e MESOS_CONTAINERIZERS=docker,mesos \
> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins \
> -e MESOS_WORK_DIR=/var/lib/mesos \
> -e MESOS_LOGGING_LEVEL=INFO \
> -e AMAZON_REGION=us-office-2 \
> -e AVAILABILITY_ZONE=us-office-2b \
> -e MESOS_ATTRIBUTES=\"platform:linux;host:$(hostname);rack:us-office-2b\" 
> \
> -e MESOS_CLUSTER=ZeroZero \
> -e MESOS_DOCKER_SOCKET=/root/docker.sock \
> -e 
> MESOS_MASTER=zk://10.150.150.224:2181,10.150.150.225:2181,10.150.150.226:2181/mesos
>  \
> -e MESOS_LOG_DIR=/var/log/mesos \
> -e 
> MESOS_ISOLATION=\"filesystem/linux,cgroups/cpu,cgroups/mem,docker/runtime\" \
> -e MESOS_IMAGE_PROVIDERS=docker \
> -e MESOS_IMAGE_PROVISIONER_BACKEND=copy \
> -e MESOS_DOCKER_REGISTRY=http://docker-registry:31000 \
> -e MESOS_DOCKER_STORE_DIR=/var/lib/mesos/docker \
> --entrypoint=/usr/sbin/mesos-slave \
> docker-registry.thebrighttag.com:31000/mesos:latest \
> --no-systemd_enable_support \
> || rm -f /var/lib/mesos/meta/slaves/latest"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AURORA-1787) `-global_container_mounts` does not appear to work with the unified containerizer

2016-10-05 Thread Joshua Cohen (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548828#comment-15548828
 ] 

Joshua Cohen commented on AURORA-1787:
--

This is a limitation of the Mesos containerizer afaik, it requires that the 
paths specified exist before mounting. I mentioned this as part of the 
description of MESOS-5229, but it would probably be worthwhile to call this out 
as its own ticket.

You should be able to workaround this by making the container path in 
{{-global_container_mounts}} absolute, rather than relative. That will cause 
Mesos to bind mount the path on top of itself into the container's namespace 
and then the executor will happily mount that same path into the task's 
filesystem.

> `-global_container_mounts` does not appear to work with the unified 
> containerizer
> -
>
> Key: AURORA-1787
> URL: https://issues.apache.org/jira/browse/AURORA-1787
> Project: Aurora
>  Issue Type: Bug
>Reporter: Zameer Manji
>Priority: Critical
>
> Perhaps I misunderstand how this feature is supposed to be used, but apply 
> the following patch to master:
> {noformat}
> From 1ebb5f4c5815c647e31f3253d5e5c316a0d5edd2 Mon Sep 17 00:00:00 2001
> From: Zameer Manji 
> Date: Tue, 4 Oct 2016 20:45:41 -0700
> Subject: [PATCH] Reproduce the issue.
> ---
>  examples/vagrant/upstart/aurora-scheduler.conf |  2 +-
>  src/test/sh/org/apache/aurora/e2e/run-server.sh|  4 
>  .../sh/org/apache/aurora/e2e/test_end_to_end.sh| 26 
> +++---
>  3 files changed, 18 insertions(+), 14 deletions(-)
> diff --git a/examples/vagrant/upstart/aurora-scheduler.conf 
> b/examples/vagrant/upstart/aurora-scheduler.conf
> index 91b27d7..851b5a1 100644
> --- a/examples/vagrant/upstart/aurora-scheduler.conf
> +++ b/examples/vagrant/upstart/aurora-scheduler.conf
> @@ -40,7 +40,7 @@ exec bin/aurora-scheduler \
>-native_log_file_path=/var/db/aurora \
>-backup_dir=/var/lib/aurora/backups \
>-thermos_executor_path=$DIST_DIR/thermos_executor.pex \
> -  
> -global_container_mounts=/home/vagrant/aurora/examples/vagrant/config:/home/vagrant/aurora/examples/vagrant/config:ro
>  \
> +  -global_container_mounts=/etc/rsyslog.d:rsyslog.d.container:ro \
>-thermos_executor_flags="--announcer-ensemble localhost:2181 
> --announcer-zookeeper-auth-config 
> /home/vagrant/aurora/examples/vagrant/config/announcer-auth.json 
> --mesos-containerizer-path=/usr/libexec/mesos/mesos-containerizer" \
>-allowed_container_types=MESOS,DOCKER \
>-http_authentication_mechanism=BASIC \
> diff --git a/src/test/sh/org/apache/aurora/e2e/run-server.sh 
> b/src/test/sh/org/apache/aurora/e2e/run-server.sh
> index 1fe0909..a0ee76f 100755
> --- a/src/test/sh/org/apache/aurora/e2e/run-server.sh
> +++ b/src/test/sh/org/apache/aurora/e2e/run-server.sh
> @@ -1,6 +1,10 @@
>  #!/bin/bash
>  
>  echo "Starting up server..."
> +if [ ! -d "./rsyslog.d.container" ]; then
> +  echo "Mountpoint Doesn't Exist";
> +  exit 1;
> +fi
>  while true
>  do
>echo -e "HTTP/1.1 200 OK\r\n\r\nHello from a filesystem image." | nc -l 
> "$1"
> diff --git a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 
> b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> index c93be9b..094d776 100755
> --- a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> +++ b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> @@ -514,27 +514,27 @@ trap collect_result EXIT
>  aurorabuild all
>  setup_ssh
>  
> -test_version
> -test_http_example "${TEST_JOB_ARGS[@]}"
> -test_health_check
> +# test_version
> +# test_http_example "${TEST_JOB_ARGS[@]}"
> +# test_health_check
>  
> -test_http_example_basic "${TEST_JOB_REVOCABLE_ARGS[@]}"
> +# test_http_example_basic "${TEST_JOB_REVOCABLE_ARGS[@]}"
>  
> -test_http_example_basic "${TEST_JOB_GPU_ARGS[@]}"
> +# test_http_example_basic "${TEST_JOB_GPU_ARGS[@]}"
>  
>  # build the test docker image
> -sudo docker build -t http_example -f "${TEST_ROOT}/Dockerfile.python" 
> ${TEST_ROOT}
> -test_http_example "${TEST_JOB_DOCKER_ARGS[@]}"
> +# sudo docker build -t http_example -f "${TEST_ROOT}/Dockerfile.python" 
> ${TEST_ROOT}
> +# test_http_example "${TEST_JOB_DOCKER_ARGS[@]}"
>  
>  setup_image_stores
>  test_appc_unified
> -test_docker_unified
> +# test_docker_unified
>  
> -test_admin "${TEST_ADMIN_ARGS[@]}"
> -test_basic_auth_unauthenticated  "${TEST_JOB_ARGS[@]}"
> +# test_admin "${TEST_ADMIN_ARGS[@]}"
> +# test_basic_auth_unauthenticated  "${TEST_JOB_ARGS[@]}"
>  
> -test_ephemeral_daemon_with_final 
> "${TEST_JOB_EPHEMERAL_DAEMON_WITH_FINAL_ARGS[@]}"
> +# test_ephemeral_daemon_with_final 
> "${TEST_JOB_EPHEMERAL_DAEMON_WITH_FINAL_ARGS[@]}"
>  
> -/vagrant/src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh
> -/vagrant/src/test/sh/org/apache/auro

[jira] [Commented] (AURORA-1781) Sandbox taskfs setup fails (groupadd error)

2016-10-05 Thread Joshua Cohen (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548798#comment-15548798
 ] 

Joshua Cohen commented on AURORA-1781:
--

[~jvenus] Trying to find a common thread, do you have SELINUX enabled on the 
hosts where you originally saw this problem?

> Sandbox taskfs setup fails (groupadd error)
> ---
>
> Key: AURORA-1781
> URL: https://issues.apache.org/jira/browse/AURORA-1781
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
>
> I hit what smells like a permission issue w/ `/etc/group` when trying to use 
> a docker-image (unified containerizer setup) with mesos-1.0.0. and 
> aurora-0.16.0-rc2.  I cannot reproduce issue w/ mesos-0.28.2 and aurora-015.0.
> {code}
> Failed to initialize sandbox: Failed to create group in sandbox for task 
> image: Command '['groupadd', '-R', 
> '/var/lib/mesos/slaves/5d28d0cc-2793-4471-82d5-e67276c53f70-S2/frameworks/20160221-001235-3801519626-5050-1-/executors/thermos-nobody-prod-jenkins-0-47cc7824-565b-4265-9ab4-9ba3f364ebed/runs/a3f78288-4865-4166-8685-1ad941562f2f/taskfs',
>  '-g', '99', 'nobody']' returned non-zero exit status 10
> {code}
> {code}
> [root@mesos-master01of2 taskfs]# pwd
> /var/lib/mesos/slaves/5d28d0cc-2793-4471-82d5-e67276c53f70-S2/frameworks/20160221-001235-3801519626-5050-1-/executors/thermos-nobody-prod-jenkins-0-47cc7824-565b-4265-9ab4-9ba3f364ebed/runs/a3f78288-4865-4166-8685-1ad941562f2f/taskfs
> [root@mesos-master01of2 taskfs]# groupadd -R $PWD -g 99 nobody
> groupadd: cannot lock /etc/group; try again later.
> {code}
> Maybe related to AURORA-1761
> I'm running CoreOS with the mesos-agent (and thermos) inside docker.  Here is 
> the gist of how it's started.
> {code}
> /usr/bin/sh -c "exec /usr/bin/docker run \
> --name=mesos_slave \
> --net=host \
> --pid=host \
> --privileged \
> -v /sys:/sys \
> -v /usr/bin/docker:/usr/bin/docker:ro \
> -v /var/lib/docker:/var/lib/docker \
> -v /var/run/docker.sock:/root/docker.sock \
> -v /run/systemd/system:/run/systemd/system \
> -v /lib64/libdevmapper.so.1.02:/lib/libdevmapper.so.1.02:ro \
> -v /sys/fs/cgroup:/sys/fs/cgroup \
> -v /var/lib/mesos:/var/lib/mesos \
> -e MESOS_CONTAINERIZERS=docker,mesos \
> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins \
> -e MESOS_WORK_DIR=/var/lib/mesos \
> -e MESOS_LOGGING_LEVEL=INFO \
> -e AMAZON_REGION=us-office-2 \
> -e AVAILABILITY_ZONE=us-office-2b \
> -e MESOS_ATTRIBUTES=\"platform:linux;host:$(hostname);rack:us-office-2b\" 
> \
> -e MESOS_CLUSTER=ZeroZero \
> -e MESOS_DOCKER_SOCKET=/root/docker.sock \
> -e 
> MESOS_MASTER=zk://10.150.150.224:2181,10.150.150.225:2181,10.150.150.226:2181/mesos
>  \
> -e MESOS_LOG_DIR=/var/log/mesos \
> -e 
> MESOS_ISOLATION=\"filesystem/linux,cgroups/cpu,cgroups/mem,docker/runtime\" \
> -e MESOS_IMAGE_PROVIDERS=docker \
> -e MESOS_IMAGE_PROVISIONER_BACKEND=copy \
> -e MESOS_DOCKER_REGISTRY=http://docker-registry:31000 \
> -e MESOS_DOCKER_STORE_DIR=/var/lib/mesos/docker \
> --entrypoint=/usr/sbin/mesos-slave \
> docker-registry.thebrighttag.com:31000/mesos:latest \
> --no-systemd_enable_support \
> || rm -f /var/lib/mesos/meta/slaves/latest"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AURORA-1781) Sandbox taskfs setup fails (groupadd error)

2016-10-05 Thread Rogier Dikkes (JIRA)


[ 
https://issues.apache.org/jira/browse/AURORA-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548779#comment-15548779
 ] 

Rogier Dikkes commented on AURORA-1781:
---

I disabled SELINUX on a couple of hosts and since then this issue has gone 
away. The issue still persists on 1 host after selinux is disabled and i 
suspect some security setting to cause this. 

> Sandbox taskfs setup fails (groupadd error)
> ---
>
> Key: AURORA-1781
> URL: https://issues.apache.org/jira/browse/AURORA-1781
> Project: Aurora
>  Issue Type: Bug
>Affects Versions: 0.16.0
>Reporter: Justin Venus
>
> I hit what smells like a permission issue w/ `/etc/group` when trying to use 
> a docker-image (unified containerizer setup) with mesos-1.0.0. and 
> aurora-0.16.0-rc2.  I cannot reproduce issue w/ mesos-0.28.2 and aurora-015.0.
> {code}
> Failed to initialize sandbox: Failed to create group in sandbox for task 
> image: Command '['groupadd', '-R', 
> '/var/lib/mesos/slaves/5d28d0cc-2793-4471-82d5-e67276c53f70-S2/frameworks/20160221-001235-3801519626-5050-1-/executors/thermos-nobody-prod-jenkins-0-47cc7824-565b-4265-9ab4-9ba3f364ebed/runs/a3f78288-4865-4166-8685-1ad941562f2f/taskfs',
>  '-g', '99', 'nobody']' returned non-zero exit status 10
> {code}
> {code}
> [root@mesos-master01of2 taskfs]# pwd
> /var/lib/mesos/slaves/5d28d0cc-2793-4471-82d5-e67276c53f70-S2/frameworks/20160221-001235-3801519626-5050-1-/executors/thermos-nobody-prod-jenkins-0-47cc7824-565b-4265-9ab4-9ba3f364ebed/runs/a3f78288-4865-4166-8685-1ad941562f2f/taskfs
> [root@mesos-master01of2 taskfs]# groupadd -R $PWD -g 99 nobody
> groupadd: cannot lock /etc/group; try again later.
> {code}
> Maybe related to AURORA-1761
> I'm running CoreOS with the mesos-agent (and thermos) inside docker.  Here is 
> the gist of how it's started.
> {code}
> /usr/bin/sh -c "exec /usr/bin/docker run \
> --name=mesos_slave \
> --net=host \
> --pid=host \
> --privileged \
> -v /sys:/sys \
> -v /usr/bin/docker:/usr/bin/docker:ro \
> -v /var/lib/docker:/var/lib/docker \
> -v /var/run/docker.sock:/root/docker.sock \
> -v /run/systemd/system:/run/systemd/system \
> -v /lib64/libdevmapper.so.1.02:/lib/libdevmapper.so.1.02:ro \
> -v /sys/fs/cgroup:/sys/fs/cgroup \
> -v /var/lib/mesos:/var/lib/mesos \
> -e MESOS_CONTAINERIZERS=docker,mesos \
> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins \
> -e MESOS_WORK_DIR=/var/lib/mesos \
> -e MESOS_LOGGING_LEVEL=INFO \
> -e AMAZON_REGION=us-office-2 \
> -e AVAILABILITY_ZONE=us-office-2b \
> -e MESOS_ATTRIBUTES=\"platform:linux;host:$(hostname);rack:us-office-2b\" 
> \
> -e MESOS_CLUSTER=ZeroZero \
> -e MESOS_DOCKER_SOCKET=/root/docker.sock \
> -e 
> MESOS_MASTER=zk://10.150.150.224:2181,10.150.150.225:2181,10.150.150.226:2181/mesos
>  \
> -e MESOS_LOG_DIR=/var/log/mesos \
> -e 
> MESOS_ISOLATION=\"filesystem/linux,cgroups/cpu,cgroups/mem,docker/runtime\" \
> -e MESOS_IMAGE_PROVIDERS=docker \
> -e MESOS_IMAGE_PROVISIONER_BACKEND=copy \
> -e MESOS_DOCKER_REGISTRY=http://docker-registry:31000 \
> -e MESOS_DOCKER_STORE_DIR=/var/lib/mesos/docker \
> --entrypoint=/usr/sbin/mesos-slave \
> docker-registry.thebrighttag.com:31000/mesos:latest \
> --no-systemd_enable_support \
> || rm -f /var/lib/mesos/meta/slaves/latest"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AURORA-1107) Add support for mounting task specified external volumes into containers

[jira] [Commented] (AURORA-1712) Debian Jessie packagaes are embedding the mesos egg build for Ubuntu trusty

[jira] [Resolved] (AURORA-1787) `-global_container_mounts` does not appear to work with the unified containerizer

[jira] [Commented] (AURORA-1787) `-global_container_mounts` does not appear to work with the unified containerizer

[jira] [Commented] (AURORA-1787) `-global_container_mounts` does not appear to work with the unified containerizer

[jira] [Commented] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

[jira] [Commented] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

[jira] [Updated] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

[jira] [Comment Edited] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

[jira] [Commented] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

[jira] [Commented] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

[jira] [Updated] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

[jira] [Updated] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully

[jira] [Commented] (AURORA-1781) Sandbox taskfs setup fails (groupadd error)

[jira] [Commented] (AURORA-1787) `-global_container_mounts` does not appear to work with the unified containerizer

[jira] [Commented] (AURORA-1781) Sandbox taskfs setup fails (groupadd error)

[jira] [Commented] (AURORA-1781) Sandbox taskfs setup fails (groupadd error)

17 matches

Site Navigation

Mail list logo

Footer information