[
https://issues.apache.org/jira/browse/MESOS-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724561#comment-16724561
]
Chun-Hung Hsiao edited comment on MESOS-9474 at 12/18/18 11:29 PM:
-------------------------------------------------------------------
{noformat}
commit e5133763adeabcea8530235c988a8f08b4cf199d
Author: Chun-Hung Hsiao <[email protected]>
Date: Fri Dec 14 11:10:13 2018 -0800
Made master to authorize `CREATE_DISK` and `DESTROY_DISK` properly.
This patch fixes a bug where the master does not respect the
authorization result for `CREATE_DISK` or`DESTROY_DISK`, causing
subsequent operations to drop if the authorization is denied.
Review: https://reviews.apache.org/r/69563{noformat}
Backported to 1.7.x:
{noformat}
commit f3e4de63def9efb1b0e6140851292ba26d8bc644{noformat}
was (Author: chhsia0):
{noformat}
commit e5133763adeabcea8530235c988a8f08b4cf199d
Author: Chun-Hung Hsiao <[email protected]>
Date: Fri Dec 14 11:10:13 2018 -0800
Made master to authorize `CREATE_DISK` and `DESTROY_DISK` properly.
This patch fixes a bug where the master does not respect the
authorization result for `CREATE_DISK` or`DESTROY_DISK`, causing
subsequent operations to drop if the authorization is denied.
Review: https://reviews.apache.org/r/69563{noformat}
> Master does not respect authorization result for `CREATE_DISK` and
> `DESTROY_DISK`.
> ----------------------------------------------------------------------------------
>
> Key: MESOS-9474
> URL: https://issues.apache.org/jira/browse/MESOS-9474
> Project: Mesos
> Issue Type: Bug
> Components: master
> Affects Versions: 1.7.0
> Reporter: Chun-Hung Hsiao
> Assignee: Chun-Hung Hsiao
> Priority: Blocker
> Labels: mesosphere, storage
> Fix For: 1.7.1, 1.8.0
>
>
> On our internal cluster with a custom authorizer module we observed the
> following problem:
> {noformat}
> I1212 14:16:58.424782 12492 master.cpp:4489] Processing ACCEPT call for
> offers: [ 98c96586-1980-4007-9651-18dd837f529a-O16233 ] on agent
> 98c96586-1980-4007-9651-18dd837f529a-S1 at slave(1)@172.31.15.78:5051
> (172.31.15.78) for framework 98c96586-1980-4007-9651-18dd837f529a-0009
> (storage)
> I1212 14:16:58.424831 12492 master.cpp:3949] Authorizing principal
> 'storage-principal' to destroy disk 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_d17d081f-c138-472d-9e0b-aa6b21e43c7c,profile-b)]:100'
> I1212 14:16:58.424913 12492 master.cpp:3949] Authorizing principal
> 'storage-principal' to destroy disk 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_c62a4e89-f902-4968-96d8-69b9efa5f35a,profile-b)]:100'
> I1212 14:16:58.424979 12492 master.cpp:3949] Authorizing principal
> 'storage-principal' to destroy disk 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_1f490a57-9e41-41a0-b7be-408147281113,profile-b)]:100'
> I1212 14:16:58.425055 12492 master.cpp:3653] Authorizing principal
> 'storage-principal' to reserve resources 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_c60ec637-3f67-4d41-bbc8-448c022c05fb,profile-b)]:100'
> I1212 14:16:58.425118 12492 master.cpp:3949] Authorizing principal
> 'storage-principal' to destroy disk 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_5ce79163-aef2-4fd4-8073-f310dec239bf,profile-b)]:100'
> I1212 14:16:58.425499 12488 authorizer.cpp:957] dstport=5050
> dstip=172.31.5.116 result=deny object="" action="DESTROY_MOUNT_DISK"
> uid="storage-principal" reason="" authorizer="mesos-master"
> timestamp=2018-12-12 14:16:58.425453056+00:00 type=audit
> I1212 14:16:58.426434 12483 master.cpp:5769] Processing DESTROY_DISK
> operation for volume disk(allocated: dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_d17d081f-c138-472d-9e0b-aa6b21e43c7c,profile-b)]:100
> from framework 98c96586-1980-4007-9651-18dd837f529a-0
> .78:5051 (172.31.15.78)
> I1212 14:16:58.426746 12483 master.cpp:5769] Processing DESTROY_DISK
> operation for volume disk(allocated: dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_c62a4e89-f902-4968-96d8-69b9efa5f35a,profile-b)]:100
> from framework 98c96586-1980-4007-9651-18dd837f529a-0
> .78:5051 (172.31.15.78)
> I1212 14:16:58.427112 12483 master.cpp:5769] Processing DESTROY_DISK
> operation for volume disk(allocated: dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_1f490a57-9e41-41a0-b7be-408147281113,profile-b)]:100
> from framework 98c96586-1980-4007-9651-18dd837f529a-0
> .78:5051 (172.31.15.78)
> W1212 14:16:58.427274 12483 master.cpp:2275] Dropping RESERVE operation from
> framework 98c96586-1980-4007-9651-18dd837f529a-0009 (storage): Not authorized
> to reserve resources as 'storage-principal'
> I1212 14:16:58.427366 12483 master.cpp:5769] Processing DESTROY_DISK
> operation for volume disk(allocated: dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_5ce79163-aef2-4fd4-8073-f310dec239bf,profile-b)]:100
> from framework 98c96586-1980-4007-9651-18dd837f529a-0009 (storage) to agent
> 98c96586-1980-4007-9651-18dd837f529a-S1 at slave(1)@172.31.15.78:5051
> (172.31.15.78){noformat}
> The authorizer module caches authorization results, and that's why there was
> only one logged authorization requset. The problem is that, the logged
> request was for {{CREATE_MOUNT_DISK}}, and the result was {{deny}}, but
> despite the denial, all {{CREATE_DISK}} operations were processed, however
> another {{RESERVE}} operation was dropped because of this denial.
> The bug is that the master pushed a authorization future in the {{futures}}
> vector in {{Master::accept}} for each {{DESTROY_DISK}}:
>
> [https://github.com/apache/mesos/blob/18356bf3f4ac730b4a798261aad042555c4a4834/src/master/master.cpp#L4599-L4601]
> However, the master never popped and checked the future in
> {{Master::_accept}}, but go ahead to process the operation:
>
> [https://github.com/apache/mesos/blob/18356bf3f4ac730b4a798261aad042555c4a4834/src/master/master.cpp#L5706]
> The future ended up mismatched with the {{RESERVE}} operation, causing it to
> be dropped.
> Updated: a similar bug would happen if we have a validation error for
> {{LAUNCH_GROUP}}. See MESOS-9480.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
