Re: Questions about secret handling in Mesos

[Zhitao Li]

Hi Vinod,

I filed a task https://issues.apache.org/jira/browse/MESOS-8909 for this.
If we can agree that this is something worth pursing, I'll try to post some
ideas on whether there is an efficient way to do it.

On Thu, Apr 26, 2018 at 3:32 PM, Vinod Kone  wrote:

> We do direct protobuf to JSON conversion for our API endpoints and I don't
> think we do any special case logic for `Secret` type in that conversion. So
> `value` based secrets will have their value show up in v1 (and likely v0)
> API endpoints.
>
> On Mon, Apr 23, 2018 at 9:25 AM, Zhitao Li  wrote:
>
>> Hi Alexander,
>>
>> We discovered that in our own testing thus do not plan to use the
>> environment variable. For the `volume/secret` case, I believe it's possible
>> to be careful enough so we do not log that, so it's more about whether we
>> want to promise that.
>>
>> What do you think?
>>
>> On Mon, Apr 23, 2018 at 5:13 AM, Alexander Rojas > > wrote:
>>
>>>
>>> Hey Zhitao,
>>>
>>> I sadly have to tell you that the first assumption is not correct. If
>>> you use environment based secrets, docker and verbose mode, they will get
>>> printed (see this patch https://reviews.apache.org/r/57846/). The
>>> reason is that the docker command will get logged and it might contain your
>>> secrets. You may end up with some logging line like:
>>>
>>> ```
>>> I0129 14:09:22.444318 docker.cpp:1139] Running docker -H
>>> unix:///var/run/docker.suck run --cpu-shares 25 --memory 278435456 -e
>>> ADMIN_PASSWORD=test_password …
>>> ```
>>>
>>>
>>> On 19. Apr 2018, at 19:57, Zhitao Li  wrote:
>>>
>>> Hello,
>>>
>>> We at Uber plan to use volume/secret isolator to send secrets from Uber
>>> framework to Mesos agent.
>>>
>>> For this purpose, we are referring to these documents:
>>>
>>>- File based secrets design doc
>>>
>>> 
>>>and slides
>>>
>>> 
>>>.
>>>- Apache Mesos secrets documentation
>>>
>>>
>>> Could you please confirm that the following assumptions are correct?
>>>
>>>- Mesos agent and master will never log the secret data at any
>>>logging level;
>>>- Mesos agent and master will never expose the secret data as part
>>>of any API response;
>>>- Mesos agent and master will never store the secret in any
>>>persistent storage, but only on tmpfs or ramfs;
>>>- When the secret is first downloaded on the mesos agent, it will be
>>>stored as "root" on the tmpfs/ramfs before being mounted in the container
>>>ramfs.
>>>
>>> If above assumptions are true, then I would like to see them documented
>>> in this as part of the Apache Mesos secrets documentation
>>> . Otherwise,
>>> we'd like to have a design discussion with maintainer of the isolator.
>>>
>>> We appreciate your help regarding this. Thanks!
>>>
>>> Regards,
>>> Aditya And Zhitao
>>>
>>>
>>>
>>
>>
>> --
>> Cheers,
>>
>> Zhitao Li
>>
>
>


-- 
Cheers,

Zhitao Li

Re: Questions about secret handling in Mesos

[Vinod Kone]

We do direct protobuf to JSON conversion for our API endpoints and I don't
think we do any special case logic for `Secret` type in that conversion. So
`value` based secrets will have their value show up in v1 (and likely v0)
API endpoints.

On Mon, Apr 23, 2018 at 9:25 AM, Zhitao Li  wrote:

> Hi Alexander,
>
> We discovered that in our own testing thus do not plan to use the
> environment variable. For the `volume/secret` case, I believe it's possible
> to be careful enough so we do not log that, so it's more about whether we
> want to promise that.
>
> What do you think?
>
> On Mon, Apr 23, 2018 at 5:13 AM, Alexander Rojas 
> wrote:
>
>>
>> Hey Zhitao,
>>
>> I sadly have to tell you that the first assumption is not correct. If you
>> use environment based secrets, docker and verbose mode, they will get
>> printed (see this patch https://reviews.apache.org/r/57846/). The reason
>> is that the docker command will get logged and it might contain your
>> secrets. You may end up with some logging line like:
>>
>> ```
>> I0129 14:09:22.444318 docker.cpp:1139] Running docker -H
>> unix:///var/run/docker.suck run --cpu-shares 25 --memory 278435456 -e
>> ADMIN_PASSWORD=test_password …
>> ```
>>
>>
>> On 19. Apr 2018, at 19:57, Zhitao Li  wrote:
>>
>> Hello,
>>
>> We at Uber plan to use volume/secret isolator to send secrets from Uber
>> framework to Mesos agent.
>>
>> For this purpose, we are referring to these documents:
>>
>>- File based secrets design doc
>>
>> 
>>and slides
>>
>> 
>>.
>>- Apache Mesos secrets documentation
>>
>>
>> Could you please confirm that the following assumptions are correct?
>>
>>- Mesos agent and master will never log the secret data at any
>>logging level;
>>- Mesos agent and master will never expose the secret data as part of
>>any API response;
>>- Mesos agent and master will never store the secret in any
>>persistent storage, but only on tmpfs or ramfs;
>>- When the secret is first downloaded on the mesos agent, it will be
>>stored as "root" on the tmpfs/ramfs before being mounted in the container
>>ramfs.
>>
>> If above assumptions are true, then I would like to see them documented
>> in this as part of the Apache Mesos secrets documentation
>> . Otherwise, we'd
>> like to have a design discussion with maintainer of the isolator.
>>
>> We appreciate your help regarding this. Thanks!
>>
>> Regards,
>> Aditya And Zhitao
>>
>>
>>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: Questions about secret handling in Mesos

[Zhitao Li]

Hi Alexander,

We discovered that in our own testing thus do not plan to use the
environment variable. For the `volume/secret` case, I believe it's possible
to be careful enough so we do not log that, so it's more about whether we
want to promise that.

What do you think?

On Mon, Apr 23, 2018 at 5:13 AM, Alexander Rojas 
wrote:

>
> Hey Zhitao,
>
> I sadly have to tell you that the first assumption is not correct. If you
> use environment based secrets, docker and verbose mode, they will get
> printed (see this patch https://reviews.apache.org/r/57846/). The reason
> is that the docker command will get logged and it might contain your
> secrets. You may end up with some logging line like:
>
> ```
> I0129 14:09:22.444318 docker.cpp:1139] Running docker -H
> unix:///var/run/docker.suck run --cpu-shares 25 --memory 278435456 -e
> ADMIN_PASSWORD=test_password …
> ```
>
>
> On 19. Apr 2018, at 19:57, Zhitao Li  wrote:
>
> Hello,
>
> We at Uber plan to use volume/secret isolator to send secrets from Uber
> framework to Mesos agent.
>
> For this purpose, we are referring to these documents:
>
>- File based secrets design doc
>
> 
>and slides
>
> 
>.
>- Apache Mesos secrets documentation
>
>
> Could you please confirm that the following assumptions are correct?
>
>- Mesos agent and master will never log the secret data at any logging
>level;
>- Mesos agent and master will never expose the secret data as part of
>any API response;
>- Mesos agent and master will never store the secret in any persistent
>storage, but only on tmpfs or ramfs;
>- When the secret is first downloaded on the mesos agent, it will be
>stored as "root" on the tmpfs/ramfs before being mounted in the container
>ramfs.
>
> If above assumptions are true, then I would like to see them documented in
> this as part of the Apache Mesos secrets documentation
> . Otherwise, we'd
> like to have a design discussion with maintainer of the isolator.
>
> We appreciate your help regarding this. Thanks!
>
> Regards,
> Aditya And Zhitao
>
>
>


-- 
Cheers,

Zhitao Li

Re: Questions about secret handling in Mesos

[Alexander Rojas]

Hey Zhitao,

I sadly have to tell you that the first assumption is not correct. If you use 
environment based secrets, docker and verbose mode, they will get printed (see 
this patch https://reviews.apache.org/r/57846/ 
). The reason is that the docker command 
will get logged and it might contain your secrets. You may end up with some 
logging line like:

```
I0129 14:09:22.444318 docker.cpp:1139] Running docker -H 
unix:///var/run/docker.suck  run --cpu-shares 25 
--memory 278435456 -e ADMIN_PASSWORD=test_password …
```


> On 19. Apr 2018, at 19:57, Zhitao Li  wrote:
> 
> Hello,
> 
> We at Uber plan to use volume/secret isolator to send secrets from Uber 
> framework to Mesos agent.
> 
> For this purpose, we are referring to these documents:
> File based secrets design doc  
> and
>  slides 
> .
> Apache Mesos secrets documentation 
> 
> Could you please confirm that the following assumptions are correct?
> Mesos agent and master will never log the secret data at any logging level;
> Mesos agent and master will never expose the secret data as part of any API 
> response;
> Mesos agent and master will never store the secret in any persistent storage, 
> but only on tmpfs or ramfs;
> When the secret is first downloaded on the mesos agent, it will be stored as 
> "root" on the tmpfs/ramfs before being mounted in the container ramfs.
> If above assumptions are true, then I would like to see them documented in 
> this as part of the Apache Mesos secrets documentation 
> . Otherwise, we'd like 
> to have a design discussion with maintainer of the isolator.
> 
> We appreciate your help regarding this. Thanks!
> 
> Regards,
> Aditya And Zhitao

Re: Questions about secret handling in Mesos

[Lawrence Rau]

doesn’t seem a great place for a secret; depending on how the host is handling 
swap and your tolerance for risk of leakage via ram content recovery.

..larry

> On Apr 21, 2018, at 9:02 AM, Qian Zhang  wrote:
> 
> Hi Aditya,
> 
> Yeah, you are right. `hostSecretPath` is a sub-directory under agent's 
> runtime dir, and the default value of agent's runtime dir is `/var/run/mesos` 
> which is a tmpfs. So the secret is written to tmpfs on agent host.
> 
> 
> Regards,
> Qian Zhang
> 
> On Sat, Apr 21, 2018 at 8:19 AM, Aditya Bhave  > wrote:
> Hi Qian,
> 
> Secret is written to file at hostSecretPath which is derived like this:
> 
> const string hostSecretPath = path::join(flags.runtime_dir, SECRET_DIR, 
> stringify(id::UUID::random()));
> Also,
>   const string hostSecretTmpDir = path::join(flags.runtime_dir, SECRET_DIR);
> Is the hostSecretTmpDir not located on tmpfs? The dir name alludes to this.
> 
> Thanks,
> -Aditya
> 
> On Fri, Apr 20, 2018 at 5:05 PM, Qian Zhang  > wrote:
> > When the secret is first downloaded on the mesos agent, it will be stored 
> > as "root" on the tmpfs/ramfs before being mounted in the container ramfs.
> 
> It seems the secret is not stored on the tmpfs/ramfs on the agent host, we 
> just write it into a file 
> 
>  under the agent's runtime directory, and then move it into the ramfs 
> 
>  in the container when the container is launched.
> 
> 
> Regards,
> Qian Zhang
> 
> On Fri, Apr 20, 2018 at 2:47 PM, Gilbert Song  > wrote:
> IIUC, your assumptions are all correct.
> 
> @Kapil, could you please confirm? Maybe we could improve the document at the 
> next Docathon.
> 
> Gilbert
> 
> On Thu, Apr 19, 2018 at 10:57 AM, Zhitao Li  > wrote:
> Hello,
> 
> We at Uber plan to use volume/secret isolator to send secrets from Uber
> framework to Mesos agent.
> 
> For this purpose, we are referring to these documents:
> 
>- File based secrets design doc
>
>   
> >
>and slides
>
>   
> >
>.
>- Apache Mesos secrets documentation
> >
> 
> Could you please confirm that the following assumptions are correct?
> 
>- Mesos agent and master will never log the secret data at any logging
>level;
>- Mesos agent and master will never expose the secret data as part of
>any API response;
>- Mesos agent and master will never store the secret in any persistent
>storage, but only on tmpfs or ramfs;
>- When the secret is first downloaded on the mesos agent, it will be
>stored as "root" on the tmpfs/ramfs before being mounted in the container
>ramfs.
> 
> If above assumptions are true, then I would like to see them documented in
> this as part of the Apache Mesos secrets documentation
>  >. Otherwise, we'd
> like to have a design discussion with maintainer of the isolator.
> 
> We appreciate your help regarding this. Thanks!
> 
> Regards,
> Aditya And Zhitao
> 
> 
> 
> 

Re: Questions about secret handling in Mesos

[Qian Zhang]

Hi Aditya,

Yeah, you are right. `hostSecretPath` is a sub-directory under agent's
runtime dir, and the default value of agent's runtime dir is `/var/run/mesos`
which is a tmpfs. So the secret is written to tmpfs on agent host.


Regards,
Qian Zhang

On Sat, Apr 21, 2018 at 8:19 AM, Aditya Bhave  wrote:

> Hi Qian,
>
> Secret is written to file at hostSecretPath which is derived like this:
>
> const string hostSecretPath = path::join(flags.runtime_dir, SECRET_DIR,
> stringify(id::UUID::random()));
> Also,
> const string hostSecretTmpDir = path::join(flags.runtime_dir, SECRET_DIR);
> Is the hostSecretTmpDir not located on tmpfs? The dir name alludes to
> this.
>
> Thanks,
> -Aditya
>
> On Fri, Apr 20, 2018 at 5:05 PM, Qian Zhang  wrote:
>
>> > When the secret is first downloaded on the mesos agent, it will be
>> stored as "root" on the tmpfs/ramfs before being mounted in the container
>> ramfs.
>>
>> It seems the secret is not stored on the tmpfs/ramfs on the agent host,
>> we just write it into a file
>> 
>> under the agent's runtime directory, and then move it into the ramfs
>> 
>> in the container when the container is launched.
>>
>>
>> Regards,
>> Qian Zhang
>>
>> On Fri, Apr 20, 2018 at 2:47 PM, Gilbert Song  wrote:
>>
>>> IIUC, your assumptions are all correct.
>>>
>>> @Kapil, could you please confirm? Maybe we could improve the document at
>>> the next Docathon.
>>>
>>> Gilbert
>>>
>>> On Thu, Apr 19, 2018 at 10:57 AM, Zhitao Li 
>>> wrote:
>>>
 Hello,

 We at Uber plan to use volume/secret isolator to send secrets from Uber
 framework to Mesos agent.

 For this purpose, we are referring to these documents:

- File based secrets design doc

and slides

.
- Apache Mesos secrets documentation


 Could you please confirm that the following assumptions are correct?

- Mesos agent and master will never log the secret data at any
 logging
level;
- Mesos agent and master will never expose the secret data as part of
any API response;
- Mesos agent and master will never store the secret in any
 persistent
storage, but only on tmpfs or ramfs;
- When the secret is first downloaded on the mesos agent, it will be
stored as "root" on the tmpfs/ramfs before being mounted in the
 container
ramfs.

 If above assumptions are true, then I would like to see them documented
 in
 this as part of the Apache Mesos secrets documentation
 . Otherwise,
 we'd
 like to have a design discussion with maintainer of the isolator.

 We appreciate your help regarding this. Thanks!

 Regards,
 Aditya And Zhitao

>>>
>>>
>>
>

Re: Questions about secret handling in Mesos

[Qian Zhang]

> When the secret is first downloaded on the mesos agent, it will be stored
as "root" on the tmpfs/ramfs before being mounted in the container ramfs.

It seems the secret is not stored on the tmpfs/ramfs on the agent host, we
just write it into a file

under the agent's runtime directory, and then move it into the ramfs

in the container when the container is launched.


Regards,
Qian Zhang

On Fri, Apr 20, 2018 at 2:47 PM, Gilbert Song  wrote:

> IIUC, your assumptions are all correct.
>
> @Kapil, could you please confirm? Maybe we could improve the document at
> the next Docathon.
>
> Gilbert
>
> On Thu, Apr 19, 2018 at 10:57 AM, Zhitao Li  wrote:
>
>> Hello,
>>
>> We at Uber plan to use volume/secret isolator to send secrets from Uber
>> framework to Mesos agent.
>>
>> For this purpose, we are referring to these documents:
>>
>>- File based secrets design doc
>>> TOScY87G_bMi5zBzMZmpc/edit#>
>>and slides
>>> Management%20in%20Mesos.pdf>
>>.
>>- Apache Mesos secrets documentation
>>
>>
>> Could you please confirm that the following assumptions are correct?
>>
>>- Mesos agent and master will never log the secret data at any logging
>>level;
>>- Mesos agent and master will never expose the secret data as part of
>>any API response;
>>- Mesos agent and master will never store the secret in any persistent
>>storage, but only on tmpfs or ramfs;
>>- When the secret is first downloaded on the mesos agent, it will be
>>stored as "root" on the tmpfs/ramfs before being mounted in the
>> container
>>ramfs.
>>
>> If above assumptions are true, then I would like to see them documented in
>> this as part of the Apache Mesos secrets documentation
>> . Otherwise, we'd
>> like to have a design discussion with maintainer of the isolator.
>>
>> We appreciate your help regarding this. Thanks!
>>
>> Regards,
>> Aditya And Zhitao
>>
>
>

Re: Questions about secret handling in Mesos

[Gilbert Song]

IIUC, your assumptions are all correct.

@Kapil, could you please confirm? Maybe we could improve the document at
the next Docathon.

Gilbert

On Thu, Apr 19, 2018 at 10:57 AM, Zhitao Li  wrote:

> Hello,
>
> We at Uber plan to use volume/secret isolator to send secrets from Uber
> framework to Mesos agent.
>
> For this purpose, we are referring to these documents:
>
>- File based secrets design doc
> bMi5zBzMZmpc/edit#>
>and slides
> 20Management%20in%20Mesos.pdf>
>.
>- Apache Mesos secrets documentation
>
>
> Could you please confirm that the following assumptions are correct?
>
>- Mesos agent and master will never log the secret data at any logging
>level;
>- Mesos agent and master will never expose the secret data as part of
>any API response;
>- Mesos agent and master will never store the secret in any persistent
>storage, but only on tmpfs or ramfs;
>- When the secret is first downloaded on the mesos agent, it will be
>stored as "root" on the tmpfs/ramfs before being mounted in the
> container
>ramfs.
>
> If above assumptions are true, then I would like to see them documented in
> this as part of the Apache Mesos secrets documentation
> . Otherwise, we'd
> like to have a design discussion with maintainer of the isolator.
>
> We appreciate your help regarding this. Thanks!
>
> Regards,
> Aditya And Zhitao
>

Questions about secret handling in Mesos

[Zhitao Li]

Hello,

We at Uber plan to use volume/secret isolator to send secrets from Uber
framework to Mesos agent.

For this purpose, we are referring to these documents:

   - File based secrets design doc
   

   and slides
   

   .
   - Apache Mesos secrets documentation
   

Could you please confirm that the following assumptions are correct?

   - Mesos agent and master will never log the secret data at any logging
   level;
   - Mesos agent and master will never expose the secret data as part of
   any API response;
   - Mesos agent and master will never store the secret in any persistent
   storage, but only on tmpfs or ramfs;
   - When the secret is first downloaded on the mesos agent, it will be
   stored as "root" on the tmpfs/ramfs before being mounted in the container
   ramfs.

If above assumptions are true, then I would like to see them documented in
this as part of the Apache Mesos secrets documentation
. Otherwise, we'd
like to have a design discussion with maintainer of the isolator.

We appreciate your help regarding this. Thanks!

Regards,
Aditya And Zhitao