[GitHub] [incubator-sedona] jiayuasu commented on pull request #494: [SEDONA-7] Build Sedona for Spark 2.4, 3.0 and Scala 2.11, 2.12
jiayuasu commented on pull request #494: URL: https://github.com/apache/incubator-sedona/pull/494#issuecomment-742123181 @Imbruced I have fixed the Python 3.9 issue. It turns out that we only need to do `sudo apt-get install libgeos-dev`. Now I will merge the PR. You can go ahead and open a PR for your faster Adapter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #494: [SEDONA-7] Build Sedona for Spark 2.4, 3.0 and Scala 2.11, 2.12
jiayuasu commented on pull request #494: URL: https://github.com/apache/incubator-sedona/pull/494#issuecomment-741675706 @Imbruced I have successfully made Sedona run on Spark 2.4.7 + Python 3.7. In fact, I am glad that the test failed before. There was a bug in the root pom.xml (sedona-parent). It packaged a wrong jackson into the compiled Sedona jar. It was introduced by PR https://github.com/apache/incubator-sedona/pull/471 This bug will sometimes cause the Scala / Java / Python Sedona fail in the Spark cluster mode. Once I removed this dependency, all test passed. Now as you can see in the GitHub CI test result, 6 checks have passed. The only thing left is the test on Spark 3.0.1 + Python 3.9. Based on my initial test https://github.com/apache/incubator-sedona/runs/1521112458 , the error is `OSError: Could not find library geos_c or load any of its variants ['libgeos_c.so.1', 'libgeos_c.so']` It looks like some of the Sedona Python packages need to be updated. If you think Spark 3.0.1 + Python 3.9 is something easy to fix, please let me know the solution. If you think this will take some time, I will directly merge this PR and leave Python 3.9 support for the future work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #494: [SEDONA-7] Build Sedona for Spark 2.4, 3.0 and Scala 2.11, 2.12
jiayuasu commented on pull request #494: URL: https://github.com/apache/incubator-sedona/pull/494#issuecomment-741636788 @Imbruced Test on PySpark 2.4.7 + Python 3.7 still failed. Please see https://github.com/apache/incubator-sedona/runs/1522711633?check_suite_focus=true It uses the correct PySpark version 2.4.7, and Spark binary version 2.4.7. I use `pipenv graph` to print out all installed packages. PySpark in Pipfile is also set to `>=2.4.0` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #494: [SEDONA-7] Build Sedona for Spark 2.4, 3.0 and Scala 2.11, 2.12
jiayuasu commented on pull request #494: URL: https://github.com/apache/incubator-sedona/pull/494#issuecomment-741557234 > Should I rebase to this branch when I will create PR with faster Adapter for python ? Or wait until it will be merged ? It should be ready tomorrow (docs update remains). Let me try to merge this PR first. I believe it will be done today or tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #494: [SEDONA-7] Build Sedona for Spark 2.4, 3.0 and Scala 2.11, 2.12
jiayuasu commented on pull request #494: URL: https://github.com/apache/incubator-sedona/pull/494#issuecomment-740517601 @Imbruced I also made a few changes on Python Adapter and Python. You can take a look: 1. PythonAdapter uses a Scala API which is compatible for both Scala 2.11 and 2.12 2. Fix the get version bug in Sedona Python. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #494: [SEDONA-7] Build Sedona for Spark 2.4, 3.0 and Scala 2.11, 2.12
jiayuasu commented on pull request #494: URL: https://github.com/apache/incubator-sedona/pull/494#issuecomment-740502328 @Imbruced I have tested Sedona Python on Spark 2.4.7 + Python 3.7. It failed. I also tested it on Spark 3.0.1 + Python 3.7. It passed. 1. Does Seodona Python support Spark 2.4? 2. If not, what change do you need to support Spark 2.4? Can we simply change the PySpark version in PipFile? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
