GitHub user speful opened a pull request:
https://github.com/apache/spark/pull/22118
Branch 2.2
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-2.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22118.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22118
commit 86609a95af4b700e83638b7416c7e3706c2d64c6
Author: Liang-Chi Hsieh
Date: 2017-08-08T08:12:41Z
[SPARK-21567][SQL] Dataset should work with type alias
If we create a type alias for a type workable with Dataset, the type alias
doesn't work with Dataset.
A reproducible case looks like:
object C {
type TwoInt = (Int, Int)
def tupleTypeAlias: TwoInt = (1, 1)
}
Seq(1).toDS().map(_ => ("", C.tupleTypeAlias))
It throws an exception like:
type T1 is not a class
scala.ScalaReflectionException: type T1 is not a class
at
scala.reflect.api.Symbols$SymbolApi$class.asClass(Symbols.scala:275)
...
This patch accesses the dealias of type in many places in `ScalaReflection`
to fix it.
Added test case.
Author: Liang-Chi Hsieh
Closes #18813 from viirya/SPARK-21567.
(cherry picked from commit ee1304199bcd9c1d5fc94f5b06fdd5f6fe7336a1)
Signed-off-by: Wenchen Fan
commit e87ffcaa3e5b75f8d313dc995e4801063b60cd5c
Author: Wenchen Fan
Date: 2017-08-08T08:32:49Z
Revert "[SPARK-21567][SQL] Dataset should work with type alias"
This reverts commit 86609a95af4b700e83638b7416c7e3706c2d64c6.
commit d0233145208eb6afcd9fe0c1c3a9dbbd35d7727e
Author: pgandhi
Date: 2017-08-09T05:46:06Z
[SPARK-21503][UI] Spark UI shows incorrect task status for a killed
Executor Process
The executor tab on Spark UI page shows task as completed when an executor
process that is running that task is killed using the kill command.
Added the case ExecutorLostFailure which was previously not there, thus,
the default case would be executed in which case, task would be marked as
completed. This case will consider all those cases where executor connection to
Spark Driver was lost due to killing the executor process, network connection
etc.
## How was this patch tested?
Manually Tested the fix by observing the UI change before and after.
Before:
https://user-images.githubusercontent.com/8190/28482929-571c9cea-6e30-11e7-93dd-728de5cdea95.png;>
After:
https://user-images.githubusercontent.com/8190/28482964-8649f5ee-6e30-11e7-91bd-2eb2089c61cc.png;>
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Author: pgandhi
Author: pgandhi999
Closes #18707 from pgandhi999/master.
(cherry picked from commit f016f5c8f6c6aae674e9905a5c0b0bede09163a4)
Signed-off-by: Wenchen Fan
commit 7446be3328ea75a5197b2587e3a8e2ca7977726b
Author: WeichenXu
Date: 2017-08-09T06:44:10Z
[SPARK-21523][ML] update breeze to 0.13.2 for an emergency bugfix in strong
wolfe line search
## What changes were proposed in this pull request?
Update breeze to 0.13.1 for an emergency bugfix in strong wolfe line search
https://github.com/scalanlp/breeze/pull/651
## How was this patch tested?
N/A
Author: WeichenXu
Closes #18797 from WeichenXu123/update-breeze.
(cherry picked from commit b35660dd0e930f4b484a079d9e2516b0a7dacf1d)
Signed-off-by: Yanbo Liang
commit f6d56d2f1c377000921effea2b1faae15f9cae82
Author: Shixiong Zhu
Date: 2017-08-09T06:49:33Z
[SPARK-21596][SS] Ensure places calling HDFSMetadataLog.get check the
return value
Same PR as #18799 but for branch 2.2. Main discussion the other PR.
When I was investigating a flaky test, I realized that many places don't
check the return value of `HDFSMetadataLog.get(batchId: Long): Option[T]`. When
a batch is supposed to be there, the caller just ignores None rather than
throwing an error. If some bug causes a query doesn't generate a batch metadata
file, this behavior will hide it and allow the query continuing to run and
finally delete metadata logs and make it hard to debug.
This PR ensures that places calling HDFSMetadataLog.get always check the
