GitHub user deepaksonu opened a pull request:
https://github.com/apache/spark/pull/21507
Branch 1.6
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-1.6
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21507.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21507
commit cdfb2a1410aa799596c8b751187dbac28b2cc678
Author: Wenchen Fan
Date: 2016-02-04T00:13:23Z
[SPARK-13101][SQL][BRANCH-1.6] nullability of array type element should not
fail analysis of encoder
nullability should only be considered as an optimization rather than part
of the type system, so instead of failing analysis for mismatch nullability, we
should pass analysis and add runtime null check.
backport https://github.com/apache/spark/pull/11035 to 1.6
Author: Wenchen Fan
Closes #11042 from cloud-fan/branch-1.6.
commit 2f390d3066297466d98e17a78c5433f37f70cc95
Author: Yuhao Yang
Date: 2016-02-04T05:19:44Z
[ML][DOC] fix wrong api link in ml onevsrest
minor fix for api link in ml onevsrest
Author: Yuhao Yang
Closes #11068 from hhbyyh/onevsrestDoc.
(cherry picked from commit c2c956bcd1a75fd01868ee9ad2939a6d3de52bc2)
Signed-off-by: Xiangrui Meng
commit a907c7c64887833770cd593eecccf53620de59b7
Author: Shixiong Zhu
Date: 2016-02-04T20:43:16Z
[SPARK-13195][STREAMING] Fix NoSuchElementException when a state is not set
but timeoutThreshold is defined
Check the state Existence before calling get.
Author: Shixiong Zhu
Closes #11081 from zsxwing/SPARK-13195.
(cherry picked from commit 8e2f296306131e6c7c2f06d6672995d3ff8ab021)
Signed-off-by: Shixiong Zhu
commit 3ca5dc3072d0d96ba07d102e9104cbbb177c352b
Author: Bill Chambers
Date: 2016-02-05T22:35:39Z
[SPARK-13214][DOCS] update dynamicAllocation documentation
Author: Bill Chambers
Closes #11094 from anabranch/dynamic-docs.
(cherry picked from commit 66e1383de2650a0f06929db8109a02e32c5eaf6b)
Signed-off-by: Andrew Or
commit 9b30096227263f77fc67ed8f12fb2911c3256774
Author: Davies Liu
Date: 2016-02-08T20:08:58Z
[SPARK-13210][SQL] catch OOM when allocate memory and expand array
There is a bug when we try to grow the buffer, OOM is ignore wrongly (the
assert also skipped by JVM), then we try grow the array again, this one will
trigger spilling free the current page, the current record we inserted will be
invalid.
The root cause is that JVM has less free memory than MemoryManager thought,
it will OOM when allocate a page without trigger spilling. We should catch the
OOM, and acquire memory again to trigger spilling.
And also, we could not grow the array in `insertRecord` of `InMemorySorter`
(it was there just for easy testing).
Author: Davies Liu
Closes #11095 from davies/fix_expand.
commit 82fa86470682cb4fcd4b3d5351167e4a936b8494
Author: Steve Loughran
Date: 2016-02-09T19:01:47Z
[SPARK-12807][YARN] Spark External Shuffle not working in Hadoop clusters
with Jackson 2.2.3
Patch to
1. Shade jackson 2.x in spark-yarn-shuffle JAR: core, databind, annotation
2. Use maven antrun to verify the JAR has the renamed classes
Being Maven-based, I don't know if the verification phase kicks in on an
SBT/jenkins build. It will on a `mvn install`
Author: Steve Loughran
Closes #10780 from steveloughran/stevel/patches/SPARK-12807-master-shuffle.
(cherry picked from commit 34d0b70b309f16af263eb4e6d7c36e2ea170bc67)
Signed-off-by: Marcelo Vanzin
commit 89818cbf808137201d2558eaab312264d852cf00
Author: Liang-Chi Hsieh
Date: 2016-02-10T01:10:55Z
[SPARK-10524][ML] Use the soft prediction to order categories' bins
JIRA: https://issues.apache.org/jira/browse/SPARK-10524
Currently we use the hard prediction (`ImpurityCalculator.predict`) to
order categories' bins. But we should use the soft prediction.
Author: Liang-Chi Hsieh
Author: Liang-Chi Hsieh
Author: Joseph K. Bradley
Closes #8734 from viirya/dt-soft-centroids.
(cherry picked from commit 9267bc68fab65c6a798e065a1dbe0f5171df3077)
Signed-off-by: Joseph K. Bradley
commit 93f1d91755475a242456fe06e57bfca10f4d722f
