GitHub user Suresh-Patibandla opened a pull request:
https://github.com/apache/spark/pull/22248
TestPull
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22248.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22248
commit c5b8d54c61780af6e9e157e6c855718df972efad
Author: Chris Martin
Date: 2018-07-28T15:40:10Z
[SPARK-24950][SQL] DateTimeUtilsSuite daysToMillis and millisToDays fails
w/java 8 181-b13
## What changes were proposed in this pull request?
- Update DateTimeUtilsSuite so that when testing roundtripping in
daysToMillis and millisToDays multiple skipdates can be specified.
- Updated test so that both new years eve 2014 and new years day 2015 are
skipped for kiribati time zones. This is necessary as java versions pre
181-b13 considered new years day 2015 to be skipped while susequent versions
corrected this to new years eve.
## How was this patch tested?
Unit tests
Author: Chris Martin
Closes #21901 from d80tb7/SPARK-24950_datetimeUtilsSuite_failures.
commit 8fe5d2c393f035b9e82ba42202421c9ba66d6c78
Author: Kazuaki Ishizaki
Date: 2018-07-29T13:31:16Z
[SPARK-24956][Build][test-maven] Upgrade maven version to 3.5.4
## What changes were proposed in this pull request?
This PR updates maven version from 3.3.9 to 3.5.4. The current build
process uses mvn 3.3.9 that was release on 2015, which looks pretty old.
We met [an issue](https://issues.apache.org/jira/browse/SPARK-24895) to
need the maven 3.5.2 or later.
The release note of the 3.5.4 is
[here](https://maven.apache.org/docs/3.5.4/release-notes.html). Note version
3.4 was skipped.
From [the release note of the
3.5.0](https://maven.apache.org/docs/3.5.0/release-notes.html), the followings
are new features:
1. ANSI color logging for improved output visibility
1. add support for module name != artifactId in every calculated URLs
(project, SCM, site): special project.directory property
1. create a slf4j-simple provider extension that supports level color
rendering
1. ModelResolver interface enhancement: addition of
resolveModel(Dependency) supporting version ranges
## How was this patch tested?
Existing tests
Author: Kazuaki Ishizaki
Closes #21905 from kiszk/SPARK-24956.
commit 2c54aae1bc2fa3da26917c89e6201fb2108d9fab
Author: liulijia
Date: 2018-07-29T20:13:00Z
[SPARK-24809][SQL] Serializing LongToUnsafeRowMap in executor may result in
data error
When join key is long or int in broadcast join, Spark will use
`LongToUnsafeRowMap` to store key-values of the table witch will be
broadcasted. But, when `LongToUnsafeRowMap` is broadcasted to executors, and it
is too big to hold in memory, it will be stored in disk. At that time, because
`write` uses a variable `cursor` to determine how many bytes in `page` of
`LongToUnsafeRowMap` will be write out and the `cursor` was not restore when
deserializing, executor will write out nothing from page into disk.
## What changes were proposed in this pull request?
Restore cursor value when deserializing.
Author: liulijia
Closes #21772 from liutang123/SPARK-24809.
commit 3695ba57731a669ed20e7f676edee602c292fbed
Author: Xingbo Jiang
Date: 2018-07-30T01:58:28Z
[MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSuite and
TaskSchedulerImplSuite
## What changes were proposed in this pull request?
In the `afterEach()` method of both `TastSetManagerSuite` and
`TaskSchedulerImplSuite`, `super.afterEach()` shall be called at the end,
because it shall stop the SparkContext.
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93706/testReport/org.apache.spark.scheduler/TaskSchedulerImplSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
The test failure is caused by the above reason, the newly added
`barrierCoordinator` required `rpcEnv` which has been stopped before
`TaskSchedulerImpl` doing cleanup.
## How was this patch tested?
Existing tests.
Author: Xingbo Jiang
Closes #21908 from jiangxb1987/afterEach.
commit 3210121fed0ba256667f18f990c1a11d32c306ea
Author:
