[jira] [Updated] (SPARK-31165) Multiple wrong references in Dockerfile for k8s
[ https://issues.apache.org/jira/browse/SPARK-31165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Dimolarov updated SPARK-31165: -- Description: I am currently trying to follow the k8s instructions for Spark: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I clone apache/spark on GitHub on the master branch I saw multiple wrong folder references after trying to build my Docker image: *Issue 1: The comments in the Dockerfile reference the wrong folder for the Dockerfile:* {code:java} # If this docker file is being used in the context of building your images from a Spark # distribution, the docker build command should be invoked from the top level directory # of the Spark distribution. E.g.: # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} Well that docker build command simply won't run. I only got the following to run: {code:java} docker build -t spark:latest -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . {code} which is the actual path to the Dockerfile. *Issue 2: jars folder does not exist* After I read the tutorial I of course build spark first as per the instructions with: {code:java} ./build/mvn -Pkubernetes -DskipTests clean package{code} Nonetheless, in the Dockerfile I get this error when building: {code:java} Step 5/18 : COPY jars /opt/spark/jars COPY failed: stat /var/lib/docker/tmp/docker-builder402673637/jars: no such file or directory{code} for which I may have found a similar issue here: [https://stackoverflow.com/questions/52451538/spark-for-kubernetes-test-on-mac] I am new to Spark but I assume that this jars folder - if the build step would actually make it and I ran the maven build of the master branch successfully with the command I mentioned above - would exist in the root folder of the project. Turns out it's here: spark/assembly/target/scala-2.12/jars *Issue 3: missing entrypoint.sh and decom.sh due to wrong reference* While Issue 2 remains unresolved as I can't wrap my head around the missing jars folder (bin and sbin got copied successfully after I made a dummy jars folder) I then got stuck on these 2 steps: {code:java} COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY kubernetes/dockerfiles/spark/decom.sh /opt/{code} with: {code:java} Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY failed: stat /var/lib/docker/tmp/docker-builder638219776/kubernetes/dockerfiles/spark/entrypoint.sh: no such file or directory{code} which makes sense since the path should actually be: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh resource-managers/kubernetes/docker/src/main/dockerfiles/spark/decom.sh *Issue 4: /tests/ has been renamed in /integration-tests/* **And the location is wrong. {code:java} COPY kubernetes/tests /opt/spark/tests {code} has to be changed to: {code:java} COPY resource-managers/kubernetes/integration-tests /opt/spark/tests{code} *Remark* I only created one issue since this seems like somebody cleaned up the repo and forgot to change these. Am I missing something here? If I am, I apologise in advance since I am new to the Spark project. I also saw that some of these references were handled through vars in previous branches: [https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile] (e.g. 2.4) but that also does not run out of the box. I am also really not sure about the affected versions since that was not transparent enough for me on GH - feel free to edit that field :) Thanks in advance! was: I am currently trying to follow the k8s instructions for Spark: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I clone apache/spark on GitHub on the master branch I saw multiple wrong folder references after trying to build my Docker image: *Issue 1: The comments in the Dockerfile reference the wrong folder for the Dockerfile:* {code:java} # If this docker file is being used in the context of building your images from a Spark # distribution, the docker build command should be invoked from the top level directory # of the Spark distribution. E.g.: # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} Well that docker build command simply won't run. I only got the following to run: {code:java} docker build -t spark:latest -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . {code} which is the actual path to the Dockerfile. *Issue 2: jars folder does not exist* After I read the tutorial I of course build spark first as per the instructions with: {code:java} ./build/mvn -Pkubernetes -DskipTests clean package{code} Nonetheless, in the Dockerfile I get this error when building: {code:java} Step 5/18 : COPY jars /opt
[jira] [Updated] (SPARK-31165) Multiple wrong references in Dockerfile for k8s
[ https://issues.apache.org/jira/browse/SPARK-31165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Dimolarov updated SPARK-31165: -- Description: I am currently trying to follow the k8s instructions for Spark: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I clone apache/spark on GitHub on the master branch I saw multiple wrong folder references after trying to build my Docker image: *Issue 1: The comments in the Dockerfile reference the wrong folder for the Dockerfile:* {code:java} # If this docker file is being used in the context of building your images from a Spark # distribution, the docker build command should be invoked from the top level directory # of the Spark distribution. E.g.: # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} Well that docker build command simply won't run. I only got the following to run: {code:java} docker build -t spark:latest -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . {code} which is the actual path to the Dockerfile. *Issue 2: jars folder does not exist* After I read the tutorial I of course build spark first as per the instructions with: {code:java} ./build/mvn -Pkubernetes -DskipTests clean package{code} Nonetheless, in the Dockerfile I get this error when building: {code:java} Step 5/18 : COPY jars /opt/spark/jars COPY failed: stat /var/lib/docker/tmp/docker-builder402673637/jars: no such file or directory{code} for which I may have found a similar issue here: [https://stackoverflow.com/questions/52451538/spark-for-kubernetes-test-on-mac] I am new to Spark but I assume that this jars folder - if the build step would actually make it and I ran the maven build of the master branch successfully with the command I mentioned above - would exist in the root folder of the project. Turns out it's here: spark/assembly/target/scala-2.12/jars *Issue 3: missing entrypoint.sh and decom.sh due to wrong reference* While Issue 2 remains unresolved as I can't wrap my head around the missing jars folder (bin and sbin got copied successfully after I made a dummy jars folder) I then got stuck on these 2 steps: {code:java} COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY kubernetes/dockerfiles/spark/decom.sh /opt/{code} with: {code:java} Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY failed: stat /var/lib/docker/tmp/docker-builder638219776/kubernetes/dockerfiles/spark/entrypoint.sh: no such file or directory{code} which makes sense since the path should actually be: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh resource-managers/kubernetes/docker/src/main/dockerfiles/spark/decom.sh *Remark* I only created one issue since this seems like somebody cleaned up the repo and forgot to change these. Am I missing something here? If I am, I apologise in advance since I am new to the Spark project. I also saw that some of these references were handled through vars in previous branches: [https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile] (e.g. 2.4) but that also does not run out of the box. I am also really not sure about the affected versions since that was not transparent enough for me on GH - feel free to edit that field :) Thanks in advance! was: I am currently trying to follow the k8s instructions for Spark: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I clone apache/spark on GitHub on the master branch I saw multiple wrong folder references after trying to build my Docker image: *Issue 1: The comments in the Dockerfile reference the wrong folder for the Dockerfile:* {code:java} # If this docker file is being used in the context of building your images from a Spark # distribution, the docker build command should be invoked from the top level directory # of the Spark distribution. E.g.: # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} Well that docker build command simply won't run. I only got the following to run: {code:java} docker build -t spark:latest -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . {code} which is the actual path to the Dockerfile. *Issue 2: jars folder does not exist* After I read the tutorial I of course build spark first as per the instructions with: {code:java} ./build/mvn -Pkubernetes -DskipTests clean package{code} Nonetheless, in the Dockerfile I get this error when building: {code:java} Step 5/18 : COPY jars /opt/spark/jars COPY failed: stat /var/lib/docker/tmp/docker-builder402673637/jars: no such file or directory{code} for which I may have found a similar issue here: [https://stackoverflow.com/questions/52451538/spark-for-kubernetes-test-on-mac] I am new to
[jira] [Updated] (SPARK-31165) Multiple wrong references in Dockerfile for k8s
[ https://issues.apache.org/jira/browse/SPARK-31165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Dimolarov updated SPARK-31165: -- Description: I am currently trying to follow the k8s instructions for Spark: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I clone apache/spark on GitHub on the master branch I saw multiple wrong folder references after trying to build my Docker image: *Issue 1: The comments in the Dockerfile reference the wrong folder for the Dockerfile:* {code:java} # If this docker file is being used in the context of building your images from a Spark # distribution, the docker build command should be invoked from the top level directory # of the Spark distribution. E.g.: # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} Well that docker build command simply won't run. I only got the following to run: {code:java} docker build -t spark:latest -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . {code} which is the actual path to the Dockerfile. *Issue 2: jars folder does not exist* After I read the tutorial I of course build spark first as per the instructions with: {code:java} ./build/mvn -Pkubernetes -DskipTests clean package{code} Nonetheless, in the Dockerfile I get this error when building: {code:java} Step 5/18 : COPY jars /opt/spark/jars COPY failed: stat /var/lib/docker/tmp/docker-builder402673637/jars: no such file or directory{code} for which I may have found a similar issue here: [https://stackoverflow.com/questions/52451538/spark-for-kubernetes-test-on-mac] I am new to Spark but I assume that this jars folder - if the build step would actually make it and I ran the maven build of the master branch successfully with the command I mentioned above - would exist in the root folder of the project. *Issue 3: missing entrypoint.sh and decom.sh due to wrong reference* While Issue 2 remains unresolved as I can't wrap my head around the missing jars folder (bin and sbin got copied successfully after I made a dummy jars folder) I then got stuck on these 2 steps: {code:java} COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY kubernetes/dockerfiles/spark/decom.sh /opt/{code} with: {code:java} Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY failed: stat /var/lib/docker/tmp/docker-builder638219776/kubernetes/dockerfiles/spark/entrypoint.sh: no such file or directory{code} which makes sense since the path should actually be: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh resource-managers/kubernetes/docker/src/main/dockerfiles/spark/decom.sh *Remark* I only created one issue since this seems like somebody cleaned up the repo and forgot to change these. Am I missing something here? If I am, I apologise in advance since I am new to the Spark project. I also saw that some of these references were handled through vars in previous branches: [https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile] (e.g. 2.4) but that also does not run out of the box. I am also really not sure about the affected versions since that was not transparent enough for me on GH - feel free to edit that field :) I can also create a PR and change these but I need help with Issue 2 and the jar files since I am not sure what the correct path for that one is. Would love some help on this :) Thanks in advance! was: I am currently trying to follow the k8s instructions for Spark: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I clone apache/spark on GitHub on the master branch I saw multiple wrong folder references after trying to build my Docker image: *Issue 1: The comments in the Dockerfile reference the wrong folder for the Dockerfile:* {code:java} # If this docker file is being used in the context of building your images from a Spark # distribution, the docker build command should be invoked from the top level directory # of the Spark distribution. E.g.: # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} Well that docker build command simply won't run. I only got the following to run: {code:java} docker build -t spark:latest -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . {code} which is the actual path to the Dockerfile. *Issue 2: jars folder does not exist* After I read the tutorial I of course build spark first as per the instructions with: {code:java} ./build/mvn -Pkubernetes -DskipTests clean package{code} Nonetheless, in the Dockerfile I get this error when building: {code:java} Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY failed: stat /var/lib/docker/tmp/docker-builder638219776/kubernetes/dockerfiles/spark
[jira] [Updated] (SPARK-31165) Multiple wrong references in Dockerfile for k8s
[ https://issues.apache.org/jira/browse/SPARK-31165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Dimolarov updated SPARK-31165: -- Affects Version/s: 2.4.5 > Multiple wrong references in Dockerfile for k8s > > > Key: SPARK-31165 > URL: https://issues.apache.org/jira/browse/SPARK-31165 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core >Affects Versions: 2.4.5, 3.0.0 >Reporter: Nikolay Dimolarov >Priority: Minor > > I am currently trying to follow the k8s instructions for Spark: > [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I > clone apache/spark on GitHub on the master branch I saw multiple wrong folder > references after trying to build my Docker image: > > *Issue 1: The comments in the Dockerfile reference the wrong folder for the > Dockerfile:* > {code:java} > # If this docker file is being used in the context of building your images > from a Spark # distribution, the docker build command should be invoked from > the top level directory # of the Spark distribution. E.g.: # docker build -t > spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} > Well that docker build command simply won't run. I only got the following to > run: > {code:java} > docker build -t spark:latest -f > resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . > {code} > which is the actual path to the Dockerfile. > > *Issue 2: jars folder does not exist* > After I read the tutorial I of course build spark first as per the > instructions with: > {code:java} > ./build/mvn -Pkubernetes -DskipTests clean package{code} > Nonetheless, in the Dockerfile I get this error when building: > {code:java} > Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ > COPY failed: stat > /var/lib/docker/tmp/docker-builder638219776/kubernetes/dockerfiles/spark/entrypoint.sh: > no such file or directory{code} > for which I may have found a similar issue here: > [https://stackoverflow.com/questions/52451538/spark-for-kubernetes-test-on-mac] > I am new to Spark but I assume that this jars folder - if the build step > would actually make it and I ran the maven build of the master branch > successfully with the command I mentioned above - would exist in the root > folder of the project. > > *Issue 3: missing entrypoint.sh and decom.sh due to wrong reference* > While Issue 2 remains unresolved as I can't wrap my head around the missing > jars folder (bin and sbin got copied successfully after I made a dummy jars > folder) I then got stuck on these 2 steps: > {code:java} > COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY > kubernetes/dockerfiles/spark/decom.sh /opt/{code} > > with: > > {code:java} > Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ > COPY failed: stat > /var/lib/docker/tmp/docker-builder638219776/kubernetes/dockerfiles/spark/entrypoint.sh: > no such file or directory{code} > > which makes sense since the path should actually be: > > resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh > resource-managers/kubernetes/docker/src/main/dockerfiles/spark/decom.sh > > *Remark* > > I only created one issue since this seems like somebody cleaned up the repo > and forgot to change these. Am I missing something here? If I am, I apologise > in advance since I am new to the Spark project. I also saw that some of these > references were handled through vars in previous branches: > [https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile] > (e.g. 2.4) but that also does not run out of the box. > > I am also really not sure about the affected versions since that was not > transparent enough for me on GH - feel free to edit that field :) > > I can also create a PR and change these but I need help with Issue 2 and the > jar files since I am not sure what the correct path for that one is. Would > love some help on this :) > > Thanks in advance! > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31165) Multiple wrong references in Dockerfile for k8s
[ https://issues.apache.org/jira/browse/SPARK-31165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Dimolarov updated SPARK-31165: -- Description: I am currently trying to follow the k8s instructions for Spark: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I clone apache/spark on GitHub on the master branch I saw multiple wrong folder references after trying to build my Docker image: *Issue 1: The comments in the Dockerfile reference the wrong folder for the Dockerfile:* {code:java} # If this docker file is being used in the context of building your images from a Spark # distribution, the docker build command should be invoked from the top level directory # of the Spark distribution. E.g.: # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} Well that docker build command simply won't run. I only got the following to run: {code:java} docker build -t spark:latest -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . {code} which is the actual path to the Dockerfile. *Issue 2: jars folder does not exist* After I read the tutorial I of course build spark first as per the instructions with: {code:java} ./build/mvn -Pkubernetes -DskipTests clean package{code} Nonetheless, in the Dockerfile I get this error when building: {code:java} Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY failed: stat /var/lib/docker/tmp/docker-builder638219776/kubernetes/dockerfiles/spark/entrypoint.sh: no such file or directory{code} for which I may have found a similar issue here: [https://stackoverflow.com/questions/52451538/spark-for-kubernetes-test-on-mac] I am new to Spark but I assume that this jars folder - if the build step would actually make it and I ran the maven build of the master branch successfully with the command I mentioned above - would exist in the root folder of the project. *Issue 3: missing entrypoint.sh and decom.sh due to wrong reference* While Issue 2 remains unresolved as I can't wrap my head around the missing jars folder (bin and sbin got copied successfully after I made a dummy jars folder) I then got stuck on these 2 steps: {code:java} COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY kubernetes/dockerfiles/spark/decom.sh /opt/{code} with: {code:java} Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY failed: stat /var/lib/docker/tmp/docker-builder638219776/kubernetes/dockerfiles/spark/entrypoint.sh: no such file or directory{code} which makes sense since the path should actually be: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh resource-managers/kubernetes/docker/src/main/dockerfiles/spark/decom.sh *Remark* I only created one issue since this seems like somebody cleaned up the repo and forgot to change these. Am I missing something here? If I am, I apologise in advance since I am new to the Spark project. I also saw that some of these references were handled through vars in previous branches: [https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile] (e.g. 2.4) but that also does not run out of the box. I am also really not sure about the affected versions since that was not transparent enough for me on GH - feel free to edit that field :) I can also create a PR and change these but I need help with Issue 2 and the jar files since I am not sure what the correct path for that one is. Would love some help on this :) Thanks in advance! was: I am currently trying to follow the k8s instructions for Spark: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] and when I clone apache/spark on GitHub on the master branch I saw multiple wrong folder references after trying to build my Docker image: *Issue 1: The comments in the Dockerfile state have the wrong folder for the Dockerfile:* {code:java} # If this docker file is being used in the context of building your images from a Spark # distribution, the docker build command should be invoked from the top level directory # of the Spark distribution. E.g.: # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .{code} Well that docker build command simply won't run. I only got the following to run: {code:java} docker build -t spark:latest -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . {code} which is the actual path to the Dockerfile. *Issue 2: jars folder does not exist* After I read the tutorial I of course build spark first as per the instructions with: {code:java} ./build/mvn -Pkubernetes -DskipTests clean package{code} Nonetheless, in the Dockerfile I get this error when building: {code:java} Step 8/18 : COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/ COPY failed: stat /var