GitHub user litao1223 opened a pull request:
https://github.com/apache/spark/pull/22935
Branch 2.2
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/litao1223/spark branch-2.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22935.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22935
commit 399aa016e8f44fea4e5ef4b71a9a80484dd755f8
Author: Xingbo Jiang
Date: 2017-07-11T13:52:54Z
[SPARK-21366][SQL][TEST] Add sql test for window functions
## What changes were proposed in this pull request?
Add sql test for window functions, also remove uncecessary test cases in
`WindowQuerySuite`.
## How was this patch tested?
Added `window.sql` and the corresponding output file.
Author: Xingbo Jiang
Closes #18591 from jiangxb1987/window.
(cherry picked from commit 66d21686556681457aab6e44e19f5614c5635f0c)
Signed-off-by: Wenchen Fan
commit cb6fc89ba20a427fa7d66fa5036b17c1a5d5d87f
Author: Eric Vandenberg
Date: 2017-07-12T06:49:15Z
[SPARK-21219][CORE] Task retry occurs on same executor due to race coâ¦
â¦ndition with blacklisting
There's a race condition in the current TaskSetManager where a failed task
is added for retry (addPendingTask), and can asynchronously be assigned to an
executor *prior* to the blacklist state (updateBlacklistForFailedTask), the
result is the task might re-execute on the same executor. This is particularly
problematic if the executor is shutting down since the retry task immediately
becomes a lost task (ExecutorLostFailure). Another side effect is that the
actual failure reason gets obscured by the retry task which never actually
executed. There are sample logs showing the issue in the
https://issues.apache.org/jira/browse/SPARK-21219
The fix is to change the ordering of the addPendingTask and
updatingBlackListForFailedTask calls in TaskSetManager.handleFailedTask
Implemented a unit test that verifies the task is black listed before it is
added to the pending task. Ran the unit test without the fix and it fails.
Ran the unit test with the fix and it passes.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Author: Eric Vandenberg
Closes #18427 from ericvandenbergfb/blacklistFix.
## What changes were proposed in this pull request?
This is a backport of the fix to SPARK-21219, already checked in as 96d58f2.
## How was this patch tested?
Ran TaskSetManagerSuite tests locally.
Author: Eric Vandenberg
Closes #18604 from jsoltren/branch-2.2.
commit 39eba3053ac99f03d9df56471bae5fc5cc9f4462
Author: Kohki Nishio
Date: 2017-07-13T00:22:40Z
[SPARK-18646][REPL] Set parent classloader as null for ExecutorClassLoader
## What changes were proposed in this pull request?
`ClassLoader` will preferentially load class from `parent`. Only when
`parent` is null or the load failed, that it will call the overridden
`findClass` function. To avoid the potential issue caused by loading class
using inappropriate class loader, we should set the `parent` of `ClassLoader`
to null, so that we can fully control which class loader is used.
This is take over of #17074, the primary author of this PR is taroplus .
Should close #17074 after this PR get merged.
## How was this patch tested?
Add test case in `ExecutorClassLoaderSuite`.
Author: Kohki Nishio
Author: Xingbo Jiang
Closes #18614 from jiangxb1987/executor_classloader.
(cherry picked from commit e08d06b37bc96cc48fec1c5e40f73e0bca09c616)
Signed-off-by: Wenchen Fan
commit cf0719b5e99333b28bb4066b304dbcf8400c80ea
Author: Wenchen Fan
Date: 2017-07-13T00:34:42Z
Revert "[SPARK-18646][REPL] Set parent classloader as null for
ExecutorClassLoader"
This reverts commit 39eba3053ac99f03d9df56471bae5fc5cc9f4462.
commit bfe3ba86936ffaabff9f89d03018eb368d246b4d
Author: jerryshao
Date: 2017-07-13T22:25:38Z
[SPARK-21376][YARN] Fix yarn client token expire issue when cleaning the
staging files in long running scenario
## What changes were proposed in this pull request?
This issue happens in long running application with
