[
https://issues.apache.org/jira/browse/SPARK-43367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yikun Jiang updated SPARK-43367:
--------------------------------
Description:
rm /bin/sh && ln -sv /bin/bash /bin/sh
Does spark expect /bin/sh to be bash (in posix mode) and not just a generic
posix compliant shell? Is there a bug with using dash? Minimally, this should
do a dpkg --divert to ensure that users in dependent images don't accidentally
revert it with package updates.
[1]
https://github.com/docker-library/official-images/pull/13089?notification_referrer_id=NT_kwDOABp-orI0MzIwMzMwNzY5OjE3MzYzNTQ#issuecomment-1533540388
[2] related origina change:
https://github.com/apache-spark-on-k8s/spark/pull/444/files
was:
rm /bin/sh && ln -sv /bin/bash /bin/sh
Does spark expect /bin/sh to be bash (in posix mode) and not just a generic
posix compliant shell? Is there a bug with using dash? Minimally, this should
do a dpkg --divert to ensure that users in dependent images don't accidentally
revert it with package updates.
env | grep SPARK_JAVA_OPT_ |...
This is susceptible to a few bugs particularly around newlines in values. I see
two ways around it.
ensuring the matching name is at the start, -E '^SPARK_JAVA_OPT_', and running
all the commands with null-terminated input and output: env -0 and -z on the
other commands
or bash variable prefix expansion
for v in "${!SPARK_JAVA_OPT_@}"; do
SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" )
done
[1]
https://github.com/docker-library/official-images/pull/13089?notification_referrer_id=NT_kwDOABp-orI0MzIwMzMwNzY5OjE3MzYzNTQ#issuecomment-1533540388
[2] related origina change:
https://github.com/apache-spark-on-k8s/spark/pull/444/files
> Recover sh support in Dockerfiles
> ----------------------------------
>
> Key: SPARK-43367
> URL: https://issues.apache.org/jira/browse/SPARK-43367
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Docker
> Affects Versions: 3.5.0
> Reporter: Yikun Jiang
> Priority: Major
>
> rm /bin/sh && ln -sv /bin/bash /bin/sh
> Does spark expect /bin/sh to be bash (in posix mode) and not just a generic
> posix compliant shell? Is there a bug with using dash? Minimally, this should
> do a dpkg --divert to ensure that users in dependent images don't
> accidentally revert it with package updates.
> [1]
> https://github.com/docker-library/official-images/pull/13089?notification_referrer_id=NT_kwDOABp-orI0MzIwMzMwNzY5OjE3MzYzNTQ#issuecomment-1533540388
> [2] related origina change:
> https://github.com/apache-spark-on-k8s/spark/pull/444/files
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
