[jira] [Comment Edited] (SPARK-27365) Spark Jenkins supports testing GPU-aware scheduling features
[
https://issues.apache.org/jira/browse/SPARK-27365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820549#comment-16820549
]
Calvin Park edited comment on SPARK-27365 at 4/17/19 10:34 PM:
---
Simple Jenkinsfile
{code:java}
pipeline {
agent {
dockerfile {
label 'docker-gpu'
args '--runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all'
}
}
stages {
stage('smi') {
steps {
sh 'nvidia-smi'
}
}
}
}{code}
with Dockerfile
{code:java}
FROM ubuntu:xenial
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
curl \
flake8 \
git-core \
openjdk-8-jdk \
python2.7 \
python-pip \
wget
RUN pip install \
requests \
numpy
# Build script looks for javac in jre dir
ENV JAVA_HOME "/usr/lib/jvm/java-8-openjdk-amd64"
#
http://spark.apache.org/docs/latest/building-spark.html#setting-up-mavens-memory-usage
# We have a pretty beefy server
ENV MAVEN_OPTS "-Xmx20g -XX:ReservedCodeCacheSize=2g"
{code}
was (Author: calvinatnvidia):
Simple Jenkinsfile
{code:java}
pipeline {
agent {
dockerfile {
label 'docker-gpu'
args '--runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all'
}
}
stages {
stage('smi') {
steps {
sh 'nvidia-smi'
}
}
}
}{code}
with Dockerfile
{code:java}
FROM ubuntu:xenial
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
curl \
flake8 \
git-core \
openjdk-8-jdk \
python2.7 \
python-pip \
wget
RUN DEBIAN_FRONTEND=noninteractive pip install \
requests \
numpy
# Build script looks for javac in jre dir
ENV JAVA_HOME "/usr/lib/jvm/java-8-openjdk-amd64"
#
http://spark.apache.org/docs/latest/building-spark.html#setting-up-mavens-memory-usage
# We have a pretty beefy server
ENV MAVEN_OPTS "-Xmx20g -XX:ReservedCodeCacheSize=2g"
{code}
> Spark Jenkins supports testing GPU-aware scheduling features
>
>
> Key: SPARK-27365
> URL: https://issues.apache.org/jira/browse/SPARK-27365
> Project: Spark
> Issue Type: Story
> Components: jenkins
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Priority: Major
>
> Upgrade Spark Jenkins to install GPU cards and run GPU integration tests
> triggered by "GPU" in PRs.
> cc: [~afeng] [~shaneknapp]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27365) Spark Jenkins supports testing GPU-aware scheduling features
[
https://issues.apache.org/jira/browse/SPARK-27365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820549#comment-16820549
]
Calvin Park edited comment on SPARK-27365 at 4/17/19 10:34 PM:
---
Simple Jenkinsfile
{code:java}
pipeline {
agent {
dockerfile {
label 'docker-gpu'
args '--runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all'
}
}
stages {
stage('smi') {
steps {
sh 'nvidia-smi'
}
}
}
}{code}
with Dockerfile
{code:java}
FROM ubuntu:xenial
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
curl \
flake8 \
git-core \
openjdk-8-jdk \
python2.7 \
python-pip \
wget
RUN DEBIAN_FRONTEND=noninteractive pip install \
requests \
numpy
# Build script looks for javac in jre dir
ENV JAVA_HOME "/usr/lib/jvm/java-8-openjdk-amd64"
#
http://spark.apache.org/docs/latest/building-spark.html#setting-up-mavens-memory-usage
# We have a pretty beefy server
ENV MAVEN_OPTS "-Xmx20g -XX:ReservedCodeCacheSize=2g"
{code}
was (Author: calvinatnvidia):
Simple Jenkinsfile
{code:java}
pipeline {
agent {
dockerfile {
label 'docker-gpu'
args '--runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all'
}
}
stages {
stage('smi') {
steps {
sh 'nvidia-smi'
}
}
}
}{code}
with Dockerfile
FROM ubuntu:xenialRUN apt-get update && \DEBIAN_FRONTEND=noninteractive
apt-get install -y \curl \flake8 \git-core \openjdk-8-jdk \
python2.7 \python-pip \wgetRUN DEBIAN_FRONTEND=noninteractive pip
install \requests \numpy# Build script looks for javac in jre dirENV
JAVA_HOME "/usr/lib/jvm/java-8-openjdk-amd64"#
http://spark.apache.org/docs/latest/building-spark.html#setting-up-mavens-memory-usage#
We have a pretty beefy serverENV MAVEN_OPTS "-Xmx20g
-XX:ReservedCodeCacheSize=2g"
> Spark Jenkins supports testing GPU-aware scheduling features
>
>
> Key: SPARK-27365
> URL: https://issues.apache.org/jira/browse/SPARK-27365
> Project: Spark
> Issue Type: Story
> Components: jenkins
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Priority: Major
>
> Upgrade Spark Jenkins to install GPU cards and run GPU integration tests
> triggered by "GPU" in PRs.
> cc: [~afeng] [~shaneknapp]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
