Howdy! I'm increasingly concerned by our Cloud 'Testing' environment ( resources associated with the 'apache-beam-testing' GCP project ).
The following might sound a little strange, given repetition of words [ and many meanings of 'testing']: * There is a 'production' need for ongoing testing infrastructure, tests, pipelines, builds. and * There are also a 'development' and/or 'test' needs for extending the functionality, tests, and exploring new things, before definitely adopting and making production [ ex: getting onto master ]. and * There is a very real possibility that someone 'testing' or 'developing' new features/functionality, could accidentally delete/change/modify critical resources in the solo environment -AND- We do not have super quick/easy restore paths in the event of serious trouble - ex: if things deleted. [ I assume, as I don't see terraform/other code that seems definitely easy to re-create resources ... BUT, I could be missing and not understanding this point ]. There could be benefits [ also costs/burdens, or pros/cons to consider ] from separating functions/environments to minimize when/how people are manually using resources/roles with permissions to modify the critical infrastructure. I have some ideas for steps to address [ and where I could step in to help implement ], but before starting to formalize ideas and strategies [ longer write-ups, etc ], wanted to open for discussion to ensure whether this is thinking that makes sense, that the community believes would be valuable. If done well, longer-term I imagine most of the to-be-solution would have little effect on actual users of environments [ including developers/committers/etc ] and mostly would be around better separation and more attention paid to controls/branches/roles/permissions. *Can people share thoughts around potentially having at least one more environment to add additional safety around our critical code testing infrastructure, to keep that further distinct from workflows/process/code that is still being developed?* Do advise, Cheers, Austin
