pan3793 commented on PR #8412: URL: https://github.com/apache/hadoop/pull/8412#issuecomment-4276656967
> 2. Spark example blocks some of their workflows from running on fork repos with conditionals [like this one](https://github.com/apache/spark/blob/a2efe5fdcf9767ee8f113f7487de1a8092d2e7c6/.github/workflows/build_non_ansi.yml#L33): `if github.repository == 'apache/spark'`. > > This PR is missing 2, right? @ajfabbri yes, but this is irrelevant to security, the intention here is to save resources, I remember in those days, the forked repos also auto-run cron jobs by default. - note, the contributor has full control over their forked repo, so they can remove `if github.repository == 'apache/spark'` condition in their forked repo if they want. spark has a lot of profiles (java/scala/python/arrow/pandas versions, ANSI/non-ANSI, sbt/maven, etc.), a full combination produces a large matrix, so it selects a part of that to run on PR and push, and schedules daily jobs (you can find those jobs status in the README.md) on `apache/spark` repo for other combinations. > Spark uses these workflows from privileged context on the official repo (workflow "cron" triggers that can't be triggered by forks) as well as allowing them to run on forks. The thing I dislike about their example is it lacks clear separation of privilege. I think we could do better on that. I'm not sure what your definition of "privileged". I think it's a normal use case of GHA, the same workflow can be triggered by different events from different contexts, we just need to be careful with one case - a workflow can be triggered on the upstream context, and the workflow consumes untrusted code from PR. > Are you thinking that we 1. build the hadoop-build (dev-support) container. 2. run maven build in that container. 3. build pre-installed images (like hadoop-trunk, hadoop-, [hadoop-3](https://issues.apache.org/jira/browse/HADOOP-3).5, etc.) 4. run tests on those installed images? I don't think this is something we want to do, for unit tests. Generally, testing requires more dev dependencies, which might not be required by the runtime, for example, in Maven, we can define dependencies in compile, runtime, test scopes, when runs UT, it pulls the runtime + test scopes deps into the classpath, similar things apply to native libs. TBH, I didn't see many production use cases that deploy Hadoop in containers, and obviously, the current "pre-installed images" are mostly used for downstream projects for testing, and it only covers a few cases, at least kerberized YARN is likely not to work - for hadoop 3.4.x, official hadoop bin tgz was built on ubuntu 20.04, with openssl 1.x, while the pre-installed images is based on the ubuntu 22.04, which only has openssl 3.x in apt repo, IIRC it will fail the kerberized linux container to start. while building some smoking/integration tests like [spark kubernetes/integration-tests ](https://github.com/apache/spark/blob/branch-4.1/resource-managers/kubernetes/integration-tests/README.md) based on the "pre-installed images" might be a good supplement in the future, but obviously, this is out of the scope of the current goals -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
