Re: Time for 2.1.3
Quick update for everybody: I was trying to deal with the release scripts to get them to work with 2.1; there were some fixes needed, and on top of that Maven Central changed something over the weekend which made Java 7 unhappy. I actually was able to create an RC1 after many tries and tweaking, but there's also currently at least one blocker that I think we should pick up. So right now I'm waiting for that. So heads up that RC2 will be the first one put up for a vote (I'll explain that in the vote e-mail so people who missed this one don't scratch their heads for too long). On Tue, Jun 12, 2018 at 4:27 PM, Marcelo Vanzin wrote: > Hey all, > > There are some fixes that went into 2.1.3 recently that probably > deserve a release. So as usual, please take a look if there's anything > else you'd like on that release, otherwise I'd like to start with the > process by early next week. > > I'll go through jira to see what's the status of things targeted at > that release, but last I checked there wasn't anything on the radar. > > Thanks! > > -- > Marcelo -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: Jenkins build errors
Those still appear to be env problems. I don't know why it is so
persistent. Does it all pass locally? Retrigger tests again and see what
happens.
On Tue, Jun 19, 2018, 2:53 AM Petar Zecevic wrote:
>
> Thanks, but unfortunately, it died again. Now at pyspark tests:
>
>
>
> Running PySpark tests
>
> Running PySpark tests. Output is in
> /home/jenkins/workspace/SparkPullRequestBuilder@2/python/unit-tests.log
> Will test against the following Python executables: ['python2.7',
> 'python3.4', 'pypy']
> Will test the following Python modules: ['pyspark-core', 'pyspark-ml',
> 'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming']
> Will skip PyArrow related features against Python executable 'python2.7' in
> 'pyspark-sql' module. PyArrow >= 0.8.0 is required; however, PyArrow was not
> found.
> Will skip Pandas related features against Python executable 'python2.7' in
> 'pyspark-sql' module. Pandas >= 0.19.2 is required; however, Pandas 0.16.0
> was found.
> Will test PyArrow related features against Python executable 'python3.4' in
> 'pyspark-sql' module.
> Will test Pandas related features against Python executable 'python3.4' in
> 'pyspark-sql' module.
> Will skip PyArrow related features against Python executable 'pypy' in
> 'pyspark-sql' module. PyArrow >= 0.8.0 is required; however, PyArrow was not
> found.
> Will skip Pandas related features against Python executable 'pypy' in
> 'pyspark-sql' module. Pandas >= 0.19.2 is required; however, Pandas was not
> found.
> Starting test(python2.7): pyspark.mllib.tests
> Starting test(pypy): pyspark.sql.tests
> Starting test(pypy): pyspark.streaming.tests
> Starting test(pypy): pyspark.tests
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> ...
> [Stage 0:> (0 + 1) /
> 1]
>
> ..
> [Stage 0:> (0 + 4) /
> 4]
>
> .
> [Stage 0:> (0 + 4) /
> 4]
>
> ..
> [Stage 0:> (0 + 4) /
> 4]
>
>
> [Stage 0:> (0 + 4) /
> 4]
>
>
> [Stage 0:> (0 + 4) /
> 4]
>
>
> [Stage 0:> (0 + 4) /
> 4]
>
>
> [Stage 0:>(0 + 32) /
> 32]...
> [Stage 10:> (0 + 1) /
> 1]
>
> .
> [Stage 0:> (0 + 4) /
> 4]
>
> .s
> [Stage 0:> (0 + 1) /
> 1]
>
> .
> [Stage 0:> (0 + 4) /
> 4]
> [Stage 0:==>(1 + 3) /
> 4]
>
> .
> [Stage 0:> (0 + 4) /
> 4]
>
> ..
> [Stage 0:> (0 + 2) /
> 2]
>
> .
> [Stage 29:===> (3 + 1) /
> 4]
>
> ..
> [Stage 79:> (0 + 1) /
> 1]
>
> ..
> [Stage 83:>(0 + 4) /
> 10]
> [Stage 83:==> (4 + 4) /
> 10]
> [Stage 83:=> (8 + 2) /
> 10]
>
> ..cc: cc: no input filesno input files
>
> cc: no input files
> cc: no input files
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/usr/lib64/pypy-2.5.1/lib-python/2.7/threading.py", line 806, in
> __bootstrap_inner
> self.run()
> File "/usr/lib64/pypy-2.5.1/lib-python/2.7/threading.py", line 759, in run
> self.__target(*self.__args, **self.__kwargs)
> File
> "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/rdd.py"
> ,
> line 771, in pipe_objs
> out.write(s.encode('utf-8'))
> IOError: [Errno 32] Broken pipe: ''
>
> cc: no input files
> cc: no input files
> cc: no input files
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/usr/lib64/pypy-2.5.1/lib-python/2.7/threading.py", line 806, in
> __bootstrap_inner
> self.run()
> File "/usr/lib64/pypy-2.5.1/lib-python/2.7/threading.py", line 759, in run
> self.__target(*self.__args, **self.__kwargs)
> File
> "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/rdd.py"
> ,
> line 771, in pipe_objs
> out.write(s.encode('utf-8'))
> IOError: [Errno 32] Broken pipe: ''
>
> Exception in
Asking for reviewing PRs regarding structured streaming
Hi Spark devs, I have couple of pull requests for structured streaming which are getting older and fading out from earlier pages in PR pages. https://github.com/apache/spark/pull/21469 https://github.com/apache/spark/pull/21357 https://github.com/apache/spark/pull/21222 Two of them are in a kind of approval by couple of folks, but no approval from committers yet. One of them needs rebase and I would be happy to do it after reviewing or in progress of reviewing. Getting reviewed in time would be critical for contributors to be honest, so I'd like to ask dev mailing list to review my PRs. Thanks in advance, Jungtaek Lim (HeartSaVioR)
Re: Jenkins build errors
Thanks, but unfortunately, it died again. Now at pyspark tests: Running PySpark tests Running PySpark tests. Output is in /home/jenkins/workspace/SparkPullRequestBuilder@2/python/unit-tests.log Will test against the following Python executables: ['python2.7', 'python3.4', 'pypy'] Will test the following Python modules: ['pyspark-core', 'pyspark-ml', 'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming'] Will skip PyArrow related features against Python executable 'python2.7' in 'pyspark-sql' module. PyArrow >= 0.8.0 is required; however, PyArrow was not found. Will skip Pandas related features against Python executable 'python2.7' in 'pyspark-sql' module. Pandas >= 0.19.2 is required; however, Pandas 0.16.0 was found. Will test PyArrow related features against Python executable 'python3.4' in 'pyspark-sql' module. Will test Pandas related features against Python executable 'python3.4' in 'pyspark-sql' module. Will skip PyArrow related features against Python executable 'pypy' in 'pyspark-sql' module. PyArrow >= 0.8.0 is required; however, PyArrow was not found. Will skip Pandas related features against Python executable 'pypy' in 'pyspark-sql' module. Pandas >= 0.19.2 is required; however, Pandas was not found. Starting test(python2.7): pyspark.mllib.tests Starting test(pypy): pyspark.sql.tests Starting test(pypy): pyspark.streaming.tests Starting test(pypy): pyspark.tests Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). ... [Stage 0:> (0 + 1) / 1] .. [Stage 0:> (0 + 4) / 4] . [Stage 0:> (0 + 4) / 4] .. [Stage 0:> (0 + 4) / 4] [Stage 0:> (0 + 4) / 4] [Stage 0:> (0 + 4) / 4] [Stage 0:> (0 + 4) / 4] [Stage 0:>(0 + 32) / 32]... [Stage 10:> (0 + 1) / 1] . [Stage 0:> (0 + 4) / 4] .s [Stage 0:> (0 + 1) / 1] . [Stage 0:> (0 + 4) / 4] [Stage 0:==>(1 + 3) / 4] . [Stage 0:> (0 + 4) / 4] .. [Stage 0:> (0 + 2) / 2] . [Stage 29:===> (3 + 1) / 4] .. [Stage 79:> (0 + 1) / 1] .. [Stage 83:>(0 + 4) / 10] [Stage 83:==> (4 + 4) / 10] [Stage 83:=> (8 + 2) / 10] ..cc: cc: no input filesno input files cc: no input files cc: no input files Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/pypy-2.5.1/lib-python/2.7/threading.py", line 806, in __bootstrap_inner self.run() File
