[
https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958056#comment-16958056
]
Shane Knapp edited comment on SPARK-29106 at 10/23/19 5:29 PM:
---------------------------------------------------------------
> For pyspark test, you mentioned we didn't install any python debs for
> testing. Is there any "requirements.txt" or "test-requirements.txt" in the
> spark repo? I'm failed to find them. When we test the pyspark before, we just
> realize that we need to install numpy package with pip, because when we exec
> the pyspark test scripts, the fail message noticed us. So you mentioned
> "pyspark testing debs" before, you mean that we should figure all out
> manually? Is there any kind suggest from your side?
i manage the jenkins configs via ansible, and python specifically through
anaconda. anaconda was my initial choice for package management because we
need to support multiple python versions (2.7, 3.x, pypy) and specific package
versions for each python version itself.
sadly there is no official ARM anaconda python distribution, which is a BIG
hurdle for this project.
i also don't use requirements.txt and pip to do the initial python env setup as
pip is flakier than i like, and the conda envs just work a LOT better.
see:
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#building-identical-conda-environments
i could check in the specific python package configs in to the spark repo, but
they're specific to our worker configsn and even though the worker deployment
process is automated (via ansible) there is ALWAYS some stupid dependency loop
that pops up and requires manual intervention.
another issue is that i do NOT want any builds installing/updating/creating
either python environments OR packages. builds should NEVER EVER modify the
bare-metal (or VM) system-level configs.
so, to summarize what needs to happen to get the python tests up and running:
1) there is no conda distribution for the ARM architecture, meaning...
2) i need to use venv to install everything...
3) which means i need to use pip/requirements.txt, which is known to be flaky...
4) and the python packages for ARM are named differently than x86...
5) or don't exist...
6) or are the wrong version...
7) meaning that setting up and testing three different python versions with
differing package names and versions makes this a lot of trial and error.
i would like to get this done asap, but i will need to carve some serious time
to get my brain wrapped around the
> For sparkR test, we compile a higher R version 3.6.1 by fix many lib
> dependency, and make it work. And exec the R test script, till to all of them
> return pass. So we wonder the difficult about the test when we truelly in
> amplab, could you please share more to us?
i have a deep and comprehensive hatred of installing and setting up R. i've
attached a couple of files showing the packages installed, their versions, and
some of the ansible snippets i use to do the initial install.
https://issues.apache.org/jira/secure/attachment/12983856/R-ansible.yml
https://issues.apache.org/jira/secure/attachment/12983857/R-libs.txt
just like you, i need to go back and manually fix lib dependency and version
errors once the initial setup is complete.
this is why i have a deep and comprehensive hatred of installing and setting up
R.
> For current periodic jobs, you said it will be triggered 2 times per day.
> Each build will cost most 11 hours. I have a thought about the next job
> deployment, wish to know your thought about it. My thought is we can setup 2
> jobs per day, one is the current maven UT test triggered by SCM changes(11h),
> the other will run the pyspark and sparkR tests also triggered by SCM
> changes(including spark build and tests, may cost 5-6 hours). How about this?
> We can talk and discuss if we don't realize how difficult to do these now.
yeah, i am amenable to having a second ARM build. i'd be curious as to the
impact on the VM's performance when we have two builds running simultaneously.
if i have some time today i'll experiment.
shane
was (Author: shaneknapp):
> For pyspark test, you mentioned we didn't install any python debs for
> testing. Is there any "requirements.txt" or "test-requirements.txt" in the
> spark repo? I'm failed to find them. When we test the pyspark before, we just
> realize that we need to install numpy package with pip, because when we exec
> the pyspark test scripts, the fail message noticed us. So you mentioned
> "pyspark testing debs" before, you mean that we should figure all out
> manually? Is there any kind suggest from your side?
i manage the jenkins configs via ansible, and python specifically through
anaconda. anaconda was my initial choice for package management because we
need to support multiple python versions (2.7, 3.x, pypy) and specific package
versions for each python version itself.
sadly there is no official ARM anaconda python distribution, which is a BIG
hurdle for this project.
i also don't use requirements.txt and pip to do the initial python env setup as
pip is flakier than i like, and the conda envs just work a LOT better.
see:
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#building-identical-conda-environments
i could check in the specific python package configs in to the spark repo, but
they're specific to our worker configsn and even though the worker deployment
process is automated (via ansible) there is ALWAYS some stupid dependency loop
that pops up and requires manual intervention.
another issue is that i do NOT want any builds installing/updating/creating
either python environments OR packages. builds should NEVER EVER modify the
bare-metal (or VM) system-level configs.
so, to summarize what needs to happen to get the python tests up and running:
1) there is no conda distribution for the ARM architecture, meaning...
2) i need to use venv to install everything...
3) which means i need to use pip/requirements.txt, which is known to be flaky...
4) and the python packages for ARM are named differently than x86...
5) or don't exist...
6) or are the wrong version...
7) meaning that setting up and testing three different python versions with
differing package names and versions makes this a lot of trial and error.
i would like to get this done asap, but i will need to carve some serious time
to get my brain wrapped around the
> For sparkR test, we compile a higher R version 3.6.1 by fix many lib
> dependency, and make it work. And exec the R test script, till to all of them
> return pass. So we wonder the difficult about the test when we truelly in
> amplab, could you please share more to us?
i have a deep and comprehensive hatred of installing and setting up R. i'll
attach a couple of files showing the packages installed, their versions, and
some of the ansible snippets i use to do the initial install.
just like you, i need to go back and manually fix lib dependency and version
errors once the initial setup is complete.
this is why i have a deep and comprehensive hatred of installing and setting up
R.
> For current periodic jobs, you said it will be triggered 2 times per day.
> Each build will cost most 11 hours. I have a thought about the next job
> deployment, wish to know your thought about it. My thought is we can setup 2
> jobs per day, one is the current maven UT test triggered by SCM changes(11h),
> the other will run the pyspark and sparkR tests also triggered by SCM
> changes(including spark build and tests, may cost 5-6 hours). How about this?
> We can talk and discuss if we don't realize how difficult to do these now.
yeah, i am amenable to having a second ARM build. i'd be curious as to the
impact on the VM's performance when we have two builds running simultaneously.
if i have some time today i'll experiment.
shane
> Add jenkins arm test for spark
> ------------------------------
>
> Key: SPARK-29106
> URL: https://issues.apache.org/jira/browse/SPARK-29106
> Project: Spark
> Issue Type: Test
> Components: Tests
> Affects Versions: 3.0.0
> Reporter: huangtianhua
> Priority: Minor
> Attachments: R-ansible.yml, R-libs.txt
>
>
> Add arm test jobs to amplab jenkins for spark.
> Till now we made two arm test periodic jobs for spark in OpenLab, one is
> based on master with hadoop 2.7(similar with QA test of amplab jenkins),
> other one is based on a new branch which we made on date 09-09, see
> [http://status.openlabtesting.org/builds/job/spark-master-unit-test-hadoop-2.7-arm64]
> and
> [http://status.openlabtesting.org/builds/job/spark-unchanged-branch-unit-test-hadoop-2.7-arm64.|http://status.openlabtesting.org/builds/job/spark-unchanged-branch-unit-test-hadoop-2.7-arm64]
> We only have to care about the first one when integrate arm test with amplab
> jenkins.
> About the k8s test on arm, we have took test it, see
> [https://github.com/theopenlab/spark/pull/17], maybe we can integrate it
> later.
> And we plan test on other stable branches too, and we can integrate them to
> amplab when they are ready.
> We have offered an arm instance and sent the infos to shane knapp, thanks
> shane to add the first arm job to amplab jenkins :)
> The other important thing is about the leveldbjni
> [https://github.com/fusesource/leveldbjni,|https://github.com/fusesource/leveldbjni/issues/80]
> spark depends on leveldbjni-all-1.8
> [https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8],
> we can see there is no arm64 supporting. So we build an arm64 supporting
> release of leveldbjni see
> [https://mvnrepository.com/artifact/org.openlabtesting.leveldbjni/leveldbjni-all/1.8],
> but we can't modified the spark pom.xml directly with something like
> 'property'/'profile' to choose correct jar package on arm or x86 platform,
> because spark depends on some hadoop packages like hadoop-hdfs, the packages
> depend on leveldbjni-all-1.8 too, unless hadoop release with new arm
> supporting leveldbjni jar. Now we download the leveldbjni-al-1.8 of
> openlabtesting and 'mvn install' to use it when arm testing for spark.
> PS: The issues found and fixed:
> SPARK-28770
> [https://github.com/apache/spark/pull/25673]
>
> SPARK-28519
> [https://github.com/apache/spark/pull/25279]
>
> SPARK-28433
> [https://github.com/apache/spark/pull/25186]
>
> SPARK-28467
> [https://github.com/apache/spark/pull/25864]
>
> SPARK-29286
> [https://github.com/apache/spark/pull/26021]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
