[ https://issues.apache.org/jira/browse/KAFKA-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357273#comment-17357273 ]
Abhijit Mane commented on KAFKA-12847: -------------------------------------- Thanks [~chia7712] I guess my detailed explanation was not clear enough. System tests failed to start when it failed to build *ducker-ak-openjdk-8* image while trying on rhel / ubuntu on x86 due to bash UID conflict. It's *+not+* about numerical value of UID but string literal "UID" itself whose value can't be altered in bash. I also tried on RHEL/Ubuntu on x86 and IBM Power (ppc64le) systems with bash shell. "echo $UID" will resolve to user id of logged in user. For ex: -------------------------------------------------------------------------------- # useradd testuser && id testuser uid=1005(testuser) gid=1005(testuser) groups=1005(testuser) # echo $UID 1005 # UID=390 -bash: UID: readonly variable # UID_DUCKER=390 // accepted as no conflict with bash built-in values -------------------------------------------------------------------------------- A simple name change in Dockerfile from UID to UID_DUCKER fixes the issue so there is no conflict. Are you able to run sysTests as is after cloning and without making any changes as the README states? (https://github.com/apache/kafka/tree/trunk/tests#readme) Let me know. > Dockerfile needed for kafka system tests needs changes > ------------------------------------------------------ > > Key: KAFKA-12847 > URL: https://issues.apache.org/jira/browse/KAFKA-12847 > Project: Kafka > Issue Type: Bug > Components: system tests > Affects Versions: 2.8.0, 2.7.1 > Environment: Issue tested in environments below but is independent of > h/w arch. or Linux flavor: - > 1.) RHEL-8.3 on x86_64 > 2.) RHEL-8.3 on IBM Power (ppc64le) > 3.) apache/kafka branch tested: trunk (master) > Reporter: Abhijit Mane > Assignee: Abhijit Mane > Priority: Major > Labels: easyfix > Attachments: Dockerfile.upstream > > > Hello, > I tried apache/kafka system tests as per documentation: - > ([https://github.com/apache/kafka/tree/trunk/tests#readme|https://github.com/apache/kafka/tree/trunk/tests#readme_]) > ========================================================= > PROBLEM > ~~~~~~ > 1.) As root user, clone kafka github repo and start "kafka system tests" > # git clone [https://github.com/apache/kafka.git] > # cd kafka > # ./gradlew clean systemTestLibs > # bash tests/docker/run_tests.sh > 2.) Dockerfile issue - > [https://github.com/apache/kafka/blob/trunk/tests/docker/Dockerfile] > This file has an *UID* entry as shown below: - > ----------- > ARG *UID*="1000" > RUN useradd -u $*UID* ducker > // {color:#de350b}*Error during docker build*{color} => useradd: UID 0 is not > unique, root user id is 0 > ----------- > I ran everything as root which means the built-in bash environment variable > 'UID' always > resolves to 0 and can't be changed. Hence, the docker build fails. The issue > should be seen even if run as non-root. > 3.) Next, as root, as per README, I ran: - > server:/kafka> *bash tests/docker/run_tests.sh* > The ducker tool builds the container images & switches to user '*ducker*' > inside the container > & maps kafka root dir ('kafka') from host to '/opt/kafka-dev' in the > container. > Ref: > [https://github.com/apache/kafka/blob/trunk/tests/docker/ducker-ak|https://github.com/apache/kafka/blob/trunk/tests/docker/ducker-ak] > Ex: docker run -d *-v "${kafka_dir}:/opt/kafka-dev"* <img_name> > This fails as the 'ducker' user has *no write permissions* to create files > under 'kafka' root dir. Hence, it needs to be made writeable. > // *chmod -R a+w kafka* > – needed as container is run as 'ducker' and needs write access since kafka > root volume from host is mapped to container as "/opt/kafka-dev" where the > 'ducker' user writes logs > ========================================================= > ========================================================= > *FIXES needed* > ~~~~~~~~~ > 1.) Dockerfile - > [https://github.com/apache/kafka/blob/trunk/tests/docker/Dockerfile] > Change 'UID' to '*UID_DUCKER*'. > This won't conflict with built in bash env. var UID and the docker image > build should succeed. > ----------- > ARG *UID_DUCKER*="1000" > RUN useradd -u $*UID_DUCKER* ducker > // *{color:#57d9a3}No Error{color}* => No conflict with built-in UID > ----------- > 2.) README needs an update where we must ensure the kafka root dir from where > the tests > are launched is writeable to allow the 'ducker' user to create results/logs. > # chmod -R a+w kafka > With this, I was able to get the docker images built and system tests started > successfully. > ========================================================= > Also, I wonder whether or not upstream Dockerfile & System tests are part of > CI/CD and get tested for every PR. If so, this issue should have been caught. > > *Question to kafka SME* > ------------------------- > Do you believe this is a valid problem with the Dockerfile and the fix is > acceptable? > Please let me know and I am happy to submit a PR with this fix. > Thanks, > Abhijit -- This message was sent by Atlassian Jira (v8.3.4#803005)