Base Docker image caching broken in CI

[Hyukjin Kwon]

Hi all,

ghcr is flaky now, so we will have to wait for a couple of days and see if
it gets fixed up soon.
See also https://github.com/apache/spark/pull/39490#issuecomment-1378190658
Thanks Yikun for taking a look at this.