Repository: spark
Updated Branches:
refs/heads/branch-1.6 47e58322c -> 1b75cd6d3
[SPARK-10186][SQL][FOLLOW-UP] simplify test
Author: Wenchen Fan
Closes #9783 from cloud-fan/postgre.
(cherry picked from commit 8019f66df5c65e21d6e4e7e8fbfb7d0471ba3e37)
Signed-off-by: Reynold Xin
Project: h
Repository: spark
Updated Branches:
refs/heads/master 9154f89be -> 8019f66df
[SPARK-10186][SQL][FOLLOW-UP] simplify test
Author: Wenchen Fan
Closes #9783 from cloud-fan/postgre.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/
Repository: spark
Updated Branches:
refs/heads/branch-1.6 dbb2ea06d -> 47e58322c
[SPARK-11728] Replace example code in ml-ensembles.md using include_example
JIRA issue https://issues.apache.org/jira/browse/SPARK-11728.
The ml-ensembles.md file contains `OneVsRestExample`. Instead of writing n
Repository: spark
Updated Branches:
refs/heads/master 2f191c66b -> 9154f89be
[SPARK-11728] Replace example code in ml-ensembles.md using include_example
JIRA issue https://issues.apache.org/jira/browse/SPARK-11728.
The ml-ensembles.md file contains `OneVsRestExample`. Instead of writing new
Repository: spark
Updated Branches:
refs/heads/master 67a5132c2 -> 2f191c66b
[SPARK-11643] [SQL] parse year with leading zero
Support the years between 0 <= year < 1000
Author: Davies Liu
Closes #9701 from davies/leading_zero.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Com
Repository: spark
Updated Branches:
refs/heads/branch-1.6 3f63f08f9 -> dbb2ea06d
[SPARK-11643] [SQL] parse year with leading zero
Support the years between 0 <= year < 1000
Author: Davies Liu
Closes #9701 from davies/leading_zero.
(cherry picked from commit 2f191c66b668fc97f82f44fd8336b6a4
Repository: spark
Updated Branches:
refs/heads/master 446738e51 -> 67a5132c2
[SPARK-7013][ML][TEST] Add unit test for spark.ml StandardScaler
I have added unit test for ML's StandardScaler By comparing with R's output,
please review for me.
Thx.
Author: RoyGaoVLIS
Closes #6665 from RoyGao
Repository: spark
Updated Branches:
refs/heads/branch-1.6 737f07172 -> 3f63f08f9
[SPARK-7013][ML][TEST] Add unit test for spark.ml StandardScaler
I have added unit test for ML's StandardScaler By comparing with R's output,
please review for me.
Thx.
Author: RoyGaoVLIS
Closes #6665 from Ro
Repository: spark
Updated Branches:
refs/heads/branch-1.6 c13f72316 -> 737f07172
[SPARK-11761] Prevent the call to StreamingContext#stop() in the listener bus's
thread
See discussion toward the tail of https://github.com/apache/spark/pull/9723
>From zsxwing :
```
The user should not call stop
Repository: spark
Updated Branches:
refs/heads/master 8fb775ba8 -> 446738e51
[SPARK-11761] Prevent the call to StreamingContext#stop() in the listener bus's
thread
See discussion toward the tail of https://github.com/apache/spark/pull/9723
>From zsxwing :
```
The user should not call stop or
Repository: spark
Updated Branches:
refs/heads/master 91f4b6f2d -> 8fb775ba8
[SPARK-11755][R] SparkR should export "predict"
The bug described at
[SPARK-11755](https://issues.apache.org/jira/browse/SPARK-11755), after
exporting ```predict``` we can both get the help information from the Spar
Repository: spark
Updated Branches:
refs/heads/branch-1.6 6bb613c02 -> c13f72316
[SPARK-11755][R] SparkR should export "predict"
The bug described at
[SPARK-11755](https://issues.apache.org/jira/browse/SPARK-11755), after
exporting ```predict``` we can both get the help information from the
Repository: spark
Updated Branches:
refs/heads/branch-1.6 cb8dbf6a1 -> 6bb613c02
[SPARK-11797][SQL] collect, first, and take should use encoders for
serialization
They were previously using Spark's default serializer for serialization.
Author: Reynold Xin
Closes #9787 from rxin/SPARK-11797
Repository: spark
Updated Branches:
refs/heads/master 98be8169f -> 91f4b6f2d
[SPARK-11797][SQL] collect, first, and take should use encoders for
serialization
They were previously using Spark's default serializer for serialization.
Author: Reynold Xin
Closes #9787 from rxin/SPARK-11797.
Repository: spark
Updated Branches:
refs/heads/branch-1.6 9a1ae6a1b -> cb8dbf6a1
[SPARK-11737] [SQL] Fix serialization of UTF8String with Kyro
The default implementation of serialization UTF8String with Kyro may be not
correct (BYTE_ARRAY_OFFSET could be different across JVM)
Author: Davies
Repository: spark
Updated Branches:
refs/heads/branch-1.5 f33e277f9 -> f7a7230f3
[SPARK-11737] [SQL] Fix serialization of UTF8String with Kyro
The default implementation of serialization UTF8String with Kyro may be not
correct (BYTE_ARRAY_OFFSET could be different across JVM)
Author: Davies
Repository: spark
Updated Branches:
refs/heads/master e33053ee0 -> 98be8169f
[SPARK-11737] [SQL] Fix serialization of UTF8String with Kyro
The default implementation of serialization UTF8String with Kyro may be not
correct (BYTE_ARRAY_OFFSET could be different across JVM)
Author: Davies Liu
Repository: spark
Updated Branches:
refs/heads/branch-1.6 eddc7a58d -> 9a1ae6a1b
[SPARK-11583] [CORE] MapStatus Using RoaringBitmap More Properly
This PR upgrade the version of RoaringBitmap to 0.5.10, to optimize the memory
layout, will be much smaller when most of blocks are empty.
This PR
Repository: spark
Updated Branches:
refs/heads/master bf25f9bdf -> e33053ee0
[SPARK-11583] [CORE] MapStatus Using RoaringBitmap More Properly
This PR upgrade the version of RoaringBitmap to 0.5.10, to optimize the memory
layout, will be much smaller when most of blocks are empty.
This PR is
Repository: spark
Updated Branches:
refs/heads/master ed8d1531f -> bf25f9bdf
[SPARK-11016] Move RoaringBitmap to explicit Kryo serializer
Fix the serialization of RoaringBitmap with Kyro serializer
This PR came from https://github.com/metamx/spark/pull/1, thanks to drcrallen
Author: Davies L
Repository: spark
Updated Branches:
refs/heads/branch-1.6 683e34ed9 -> eddc7a58d
[SPARK-11016] Move RoaringBitmap to explicit Kryo serializer
Fix the serialization of RoaringBitmap with Kyro serializer
This PR came from https://github.com/metamx/spark/pull/1, thanks to drcrallen
Author: Davi
Repository: spark
Updated Branches:
refs/heads/branch-1.6 a7fcc3117 -> 683e34ed9
[SPARK-11793][SQL] Dataset should set the resolved encoders internally for maps.
I also wrote a test case -- but unfortunately the test case is not working due
to SPARK-11795.
Author: Reynold Xin
Closes #9784
Repository: spark
Updated Branches:
refs/heads/master 75a292291 -> ed8d1531f
[SPARK-11793][SQL] Dataset should set the resolved encoders internally for maps.
I also wrote a test case -- but unfortunately the test case is not working due
to SPARK-11795.
Author: Reynold Xin
Closes #9784 from
Repository: spark
Updated Branches:
refs/heads/branch-1.6 3133d8bd1 -> a7fcc3117
[SPARK-9065][STREAMING][PYSPARK] Add MessageHandler for Kafka Python API
Fixed the merge conflicts in #7410
Closes #7410
Author: Shixiong Zhu
Author: jerryshao
Author: jerryshao
Closes #9742 from zsxwing/pr7
Repository: spark
Updated Branches:
refs/heads/master b362d50fc -> 75a292291
[SPARK-9065][STREAMING][PYSPARK] Add MessageHandler for Kafka Python API
Fixed the merge conflicts in #7410
Closes #7410
Author: Shixiong Zhu
Author: jerryshao
Author: jerryshao
Closes #9742 from zsxwing/pr7410.
Repository: spark
Updated Branches:
refs/heads/branch-1.5 e26dc9642 -> f33e277f9
[HOTFIX][STREAMING] Add mockito to fix the compilation error
Added mockito to the test scope to fix the compilation error in branch 1.5
Author: Shixiong Zhu
Closes #9782 from zsxwing/1.5-hotfix.
Project: htt
Repository: spark
Updated Branches:
refs/heads/master 52c734b58 -> b362d50fc
[SPARK-11726] Throw exception on timeout when waiting for REST server response
Author: Jacek Lewandowski
Closes #9692 from jacek-lewandowski/SPARK-11726.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Repository: spark
Updated Branches:
refs/heads/branch-1.6 7e17ce5b6 -> 3133d8bd1
[SPARK-11726] Throw exception on timeout when waiting for REST server response
Author: Jacek Lewandowski
Closes #9692 from jacek-lewandowski/SPARK-11726.
Project: http://git-wip-us.apache.org/repos/asf/spark/r
Repository: spark
Updated Branches:
refs/heads/branch-1.6 3471244f7 -> 7e17ce5b6
[SPARK-11771][YARN][TRIVIAL] maximum memory in yarn is controlled by two params
have both in error msg
When we exceed the max memory tell users to increase both params instead of
just the one.
Author: Holden Ka
Repository: spark
Updated Branches:
refs/heads/master 3720b1480 -> 52c734b58
[SPARK-11771][YARN][TRIVIAL] maximum memory in yarn is controlled by two params
have both in error msg
When we exceed the max memory tell users to increase both params instead of
just the one.
Author: Holden Karau
Repository: spark
Updated Branches:
refs/heads/branch-1.6 89a2ddf77 -> 3471244f7
[SPARK-11790][STREAMING][TESTS] Increase the connection timeout
Sometimes, EmbeddedZookeeper may need more than 6 seconds to setup up in a slow
Jenkins worker. So just increase the timeout, it won't increase the
Repository: spark
Updated Branches:
refs/heads/master e29656f8e -> 3720b1480
[SPARK-11790][STREAMING][TESTS] Increase the connection timeout
Sometimes, EmbeddedZookeeper may need more than 6 seconds to setup up in a slow
Jenkins worker. So just increase the timeout, it won't increase the test
Repository: spark
Updated Branches:
refs/heads/branch-1.6 33ae9c74a -> 89a2ddf77
[MINOR] Correct comments in JavaDirectKafkaWordCount
Author: Rohan Bhanderi
Closes #9781 from RohanBhanderi/patch-3.
(cherry picked from commit e29656f8e7fa19686b448292e20d8bbf07ab9f11)
Signed-off-by: Andrew Or
Repository: spark
Updated Branches:
refs/heads/master 965245d08 -> e29656f8e
[MINOR] Correct comments in JavaDirectKafkaWordCount
Author: Rohan Bhanderi
Closes #9781 from RohanBhanderi/patch-3.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/
Repository: spark
Updated Branches:
refs/heads/master 928d63162 -> 965245d08
[SPARK-9552] Add force control for killExecutors to avoid false killing for
those busy executors
By using the dynamic allocation, sometimes it occurs false killing for those
busy executors. Some executors with assig
Repository: spark
Updated Branches:
refs/heads/branch-1.6 78dc07cdf -> 33ae9c74a
[SPARK-9552] Add force control for killExecutors to avoid false killing for
those busy executors
By using the dynamic allocation, sometimes it occurs false killing for those
busy executors. Some executors with a
Repository: spark
Updated Branches:
refs/heads/branch-1.6 fa9d56f9e -> 78dc07cdf
[SPARK-11764][ML] make Param.jsonEncode/jsonDecode support Vector
This PR makes the default read/write work with simple transformers/estimators
that have params of type `Param[Vector]`. jkbradley
Author: Xiangru
Repository: spark
Updated Branches:
refs/heads/branch-1.5 bdcbbdac6 -> e26dc9642
[SPARK-11740][STREAMING] Fix the race condition of two checkpoints in a batch
We will do checkpoint when generating a batch and completing a batch. When the
processing time of a batch is greater than the batch in
Repository: spark
Updated Branches:
refs/heads/branch-1.6 1a5dfb706 -> fa9d56f9e
[SPARK-11740][STREAMING] Fix the race condition of two checkpoints in a batch
We will do checkpoint when generating a batch and completing a batch. When the
processing time of a batch is greater than the batch in
Repository: spark
Updated Branches:
refs/heads/master 936bc0bcb -> 928d63162
[SPARK-11740][STREAMING] Fix the race condition of two checkpoints in a batch
We will do checkpoint when generating a batch and completing a batch. When the
processing time of a batch is greater than the batch interv
Repository: spark
Updated Branches:
refs/heads/branch-1.5 cf36cdba2 -> bdcbbdac6
[SPARK-11786][CORE] Tone down messages from akka error monitor.
There events happen normally during the app's lifecycle, so printing
out ERROR logs all the time is misleading, and can actually affect usability
of
Repository: spark
Updated Branches:
refs/heads/branch-1.6 a529427a1 -> 1a5dfb706
[SPARK-11786][CORE] Tone down messages from akka error monitor.
There events happen normally during the app's lifecycle, so printing
out ERROR logs all the time is misleading, and can actually affect usability
of
Repository: spark
Updated Branches:
refs/heads/master 3e9e63802 -> 936bc0bcb
[SPARK-11786][CORE] Tone down messages from akka error monitor.
There events happen normally during the app's lifecycle, so printing
out ERROR logs all the time is misleading, and can actually affect usability
of inte
Repository: spark
Updated Branches:
refs/heads/master 6eb7008b7 -> 3e9e63802
[SPARK-11764][ML] make Param.jsonEncode/jsonDecode support Vector
This PR makes the default read/write work with simple transformers/estimators
that have params of type `Param[Vector]`. jkbradley
Author: Xiangrui Me
Repository: spark
Updated Branches:
refs/heads/master 328eb49e6 -> 6eb7008b7
[SPARK-11763][ML] Add save,load to LogisticRegression Estimator
Add save/load to LogisticRegression Estimator, and refactor tests a little to
make it easier to add similar support to other Estimator, Model pairs.
Mo
Repository: spark
Updated Branches:
refs/heads/branch-1.6 c0ada5b93 -> a529427a1
[SPARK-11763][ML] Add save,load to LogisticRegression Estimator
Add save/load to LogisticRegression Estimator, and refactor tests a little to
make it easier to add similar support to other Estimator, Model pairs.
Repository: spark
Updated Branches:
refs/heads/branch-1.6 b468f8171 -> c0ada5b93
[SPARK-11729] Replace example code in ml-linear-methods.md using include_example
JIRA link: https://issues.apache.org/jira/browse/SPARK-11729
Author: Xusen Yin
Closes #9713 from yinxusen/SPARK-11729.
(cherry p
Repository: spark
Updated Branches:
refs/heads/master fa603e08d -> 328eb49e6
[SPARK-11729] Replace example code in ml-linear-methods.md using include_example
JIRA link: https://issues.apache.org/jira/browse/SPARK-11729
Author: Xusen Yin
Closes #9713 from yinxusen/SPARK-11729.
Project: htt
Repository: spark
Updated Branches:
refs/heads/branch-1.6 5047e675e -> b468f8171
[SPARK-11732] Removes some MiMa false positives
This adds an extra filter for private or protected classes. We only filter for
package private right now.
Author: Timothy Hunter
Closes #9697 from thunterdb/spar
Repository: spark
Updated Branches:
refs/heads/master 5aca6ad00 -> fa603e08d
[SPARK-11732] Removes some MiMa false positives
This adds an extra filter for private or protected classes. We only filter for
package private right now.
Author: Timothy Hunter
Closes #9697 from thunterdb/spark-11
Repository: spark
Updated Branches:
refs/heads/branch-1.6 e7f901647 -> 5047e675e
[SPARK-11767] [SQL] limit the size of caced batch
Currently the size of cached batch in only controlled by `batchSize` (default
value is 1), which does not work well with the size of serialized columns
(for
Repository: spark
Updated Branches:
refs/heads/master d98d1cb00 -> 5aca6ad00
[SPARK-11767] [SQL] limit the size of caced batch
Currently the size of cached batch in only controlled by `batchSize` (default
value is 1), which does not work well with the size of serialized columns
(for exam
Repository: spark
Updated Branches:
refs/heads/branch-1.6 88431fb3e -> e7f901647
[SPARK-11769][ML] Add save, load to all basic Transformers
This excludes Estimators and ones which include Vector and other non-basic
types for Params or data. This adds:
* Bucketizer
* DCT
* HashingTF
* Interac
Repository: spark
Updated Branches:
refs/heads/master d92514966 -> d98d1cb00
[SPARK-11769][ML] Add save, load to all basic Transformers
This excludes Estimators and ones which include Vector and other non-basic
types for Params or data. This adds:
* Bucketizer
* DCT
* HashingTF
* Interaction
Repository: spark
Updated Branches:
refs/heads/branch-1.6 933f025ce -> 88431fb3e
[SPARK-10186][SQL] support postgre array type in JDBCRDD
Add ARRAY support to `PostgresDialect`.
Nested ARRAY is not allowed for now because it's hard to get the array
dimension info. See
http://stackoverflow.c
Repository: spark
Updated Branches:
refs/heads/master 0158ff773 -> d92514966
[SPARK-10186][SQL] support postgre array type in JDBCRDD
Add ARRAY support to `PostgresDialect`.
Nested ARRAY is not allowed for now because it's hard to get the array
dimension info. See
http://stackoverflow.com/q
Repository: spark
Updated Branches:
refs/heads/branch-1.6 167ea61a6 -> 933f025ce
[SPARK-8658][SQL][FOLLOW-UP] AttributeReference's equals method compares all
the members
Based on the comment of cloud-fan in https://github.com/apache/spark/pull/9216,
update the AttributeReference's hashCode f
Repository: spark
Updated Branches:
refs/heads/master 7b1407c7b -> 0158ff773
[SPARK-8658][SQL][FOLLOW-UP] AttributeReference's equals method compares all
the members
Based on the comment of cloud-fan in https://github.com/apache/spark/pull/9216,
update the AttributeReference's hashCode funct
Repository: spark
Updated Branches:
refs/heads/branch-1.6 a3f0c77c9 -> 167ea61a6
[SPARK-11089][SQL] Adds option for disabling multi-session in Thrift server
This PR adds a new option `spark.sql.hive.thriftServer.singleSession` for
disabling multi-session support in the Thrift server.
Note th
Repository: spark
Updated Branches:
refs/heads/master e8833dd12 -> 7b1407c7b
[SPARK-11089][SQL] Adds option for disabling multi-session in Thrift server
This PR adds a new option `spark.sql.hive.thriftServer.singleSession` for
disabling multi-session support in the Thrift server.
Note that t
Repository: spark
Updated Branches:
refs/heads/branch-1.6 d07a7e1e6 -> a3f0c77c9
[SPARK-11679][SQL] Invoking method " apply(fields:
java.util.List[StructField])" in "StructType" gets ClassCastException
In the previous method, fields.toArray will cast java.util.List[StructField]
into Array[Ob
Repository: spark
Updated Branches:
refs/heads/master 21fac5434 -> e8833dd12
[SPARK-11679][SQL] Invoking method " apply(fields:
java.util.List[StructField])" in "StructType" gets ClassCastException
In the previous method, fields.toArray will cast java.util.List[StructField]
into Array[Object
Repository: spark
Updated Branches:
refs/heads/branch-1.6 0f39c2d07 -> d07a7e1e6
[SPARK-11766][MLLIB] add toJson/fromJson to Vector/Vectors
This is to support JSON serialization of Param[Vector] in the pipeline API. It
could be used for other purposes too. The schema is the same as `VectorUDT
Repository: spark
Updated Branches:
refs/heads/master cc567b663 -> 21fac5434
[SPARK-11766][MLLIB] add toJson/fromJson to Vector/Vectors
This is to support JSON serialization of Param[Vector] in the pipeline API. It
could be used for other purposes too. The schema is the same as `VectorUDT`.
Repository: spark
Updated Branches:
refs/heads/master 6fc2740eb -> cc567b663
[SPARK-11695][CORE] Set s3a credentials
Set s3a credentials when creating a new default hadoop configuration.
Author: Chris Bannister
Closes #9663 from Zariel/set-s3a-creds.
Project: http://git-wip-us.apache.org/
Repository: spark
Updated Branches:
refs/heads/branch-1.6 ba1a19020 -> 0f39c2d07
[SPARK-11695][CORE] Set s3a credentials
Set s3a credentials when creating a new default hadoop configuration.
Author: Chris Bannister
Closes #9663 from Zariel/set-s3a-creds.
(cherry picked from commit cc567b66
Repository: spark
Updated Branches:
refs/heads/branch-1.6 c049fa4bd -> ba1a19020
[SPARK-11744][LAUNCHER] Fix print version throw exception when using pyspark
shell
Exception details can be seen here
(https://issues.apache.org/jira/browse/SPARK-11744).
Author: jerryshao
Closes #9721 from j
Repository: spark
Updated Branches:
refs/heads/master 15cc36b77 -> 6fc2740eb
[SPARK-11744][LAUNCHER] Fix print version throw exception when using pyspark
shell
Exception details can be seen here
(https://issues.apache.org/jira/browse/SPARK-11744).
Author: jerryshao
Closes #9721 from jerry
Repository: spark
Updated Branches:
refs/heads/branch-1.6 62ad81a14 -> c049fa4bd
[SPARK-11191][SQL][FOLLOW-UP] Cleans up unnecessary anonymous
HiveFunctionRegistry
According to discussion in PR #9664, the anonymous `HiveFunctionRegistry` in
`HiveContext` can be removed now.
Author: Cheng Li
Repository: spark
Updated Branches:
refs/heads/branch-1.6 29f18b730 -> 62ad81a14
[SPARK-11694][SQL] Backports #9754
The main purpose of this PR is to backport
https://github.com/apache/spark/pull/9754
I added several commits but they are identical with the PR.
I will cc liancheng just to fi
Repository: spark
Updated Branches:
refs/heads/master 7276fa9aa -> 15cc36b77
[SPARK-11779][DOCS] Fix reference to deprecated MESOS_NATIVE_LIBRARY
MESOS_NATIVE_LIBRARY was renamed in favor of MESOS_NATIVE_JAVA_LIBRARY. This
commit fixes the reference in the documentation.
Author: Philipp Hoff
Repository: spark
Updated Branches:
refs/heads/branch-1.6 b1a2abb23 -> 29f18b730
[SPARK-11779][DOCS] Fix reference to deprecated MESOS_NATIVE_LIBRARY
MESOS_NATIVE_LIBRARY was renamed in favor of MESOS_NATIVE_JAVA_LIBRARY. This
commit fixes the reference in the documentation.
Author: Philipp
Repository: spark
Updated Branches:
refs/heads/branch-1.6 2ae1fa074 -> b1a2abb23
[SPARK-11751] Doc describe error in the "Spark Streaming Programming Guide" page
In the **[Task Launching
Overheads](http://spark.apache.org/docs/latest/streaming-programming-guide.html#task-launching-overheads)*
Repository: spark
Updated Branches:
refs/heads/master fa13301ae -> 7276fa9aa
[SPARK-11751] Doc describe error in the "Spark Streaming Programming Guide" page
In the **[Task Launching
Overheads](http://spark.apache.org/docs/latest/streaming-programming-guide.html#task-launching-overheads)**
s
Repository: spark
Updated Branches:
refs/heads/branch-1.5 51fc152b7 -> cf36cdba2
[SPARK-11191][SQL][FOLLOW-UP] Cleans up unnecessary anonymous
HiveFunctionRegistry
According to discussion in PR #9664, the anonymous `HiveFunctionRegistry` in
`HiveContext` can be removed now.
Author: Cheng Li
Repository: spark
Updated Branches:
refs/heads/master d79d8b08f -> fa13301ae
[SPARK-11191][SQL][FOLLOW-UP] Cleans up unnecessary anonymous
HiveFunctionRegistry
According to discussion in PR #9664, the anonymous `HiveFunctionRegistry` in
`HiveContext` can be removed now.
Author: Cheng Lian
76 matches
Mail list logo
