[spark] branch master updated (3668291 -> 51d2917)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3668291 [SPARK-30452][ML][PYSPARK][FOLLOWUP] Change IsotonicRegressionModel.numFeatures to property add 51d2917 [SPARK-30505][DOCS] Deprecate Avro option `ignoreExtension` in sql-data-sources-avro.md No new revisions were added by this update. Summary of changes: docs/sql-data-sources-avro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3668291 -> 51d2917)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3668291 [SPARK-30452][ML][PYSPARK][FOLLOWUP] Change IsotonicRegressionModel.numFeatures to property add 51d2917 [SPARK-30505][DOCS] Deprecate Avro option `ignoreExtension` in sql-data-sources-avro.md No new revisions were added by this update. Summary of changes: docs/sql-data-sources-avro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0f3d744c -> 3668291)
This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0f3d744c [MINOR][TESTS] Remove unsupported `header` option in AvroSuite add 3668291 [SPARK-30452][ML][PYSPARK][FOLLOWUP] Change IsotonicRegressionModel.numFeatures to property No new revisions were added by this update. Summary of changes: python/pyspark/ml/regression.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0f3d744c -> 3668291)
This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0f3d744c [MINOR][TESTS] Remove unsupported `header` option in AvroSuite add 3668291 [SPARK-30452][ML][PYSPARK][FOLLOWUP] Change IsotonicRegressionModel.numFeatures to property No new revisions were added by this update. Summary of changes: python/pyspark/ml/regression.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch 2.0 created (now 0f3d744c)
This is an automated email from the ASF dual-hosted git repository. zsxwing pushed a change to branch 2.0 in repository https://gitbox.apache.org/repos/asf/spark.git. at 0f3d744c [MINOR][TESTS] Remove unsupported `header` option in AvroSuite No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (db7262a -> 0f3d744c)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from db7262a [SPARK-30509][SQL] Fix deprecation log warning in Avro schema inferring add 0f3d744c [MINOR][TESTS] Remove unsupported `header` option in AvroSuite No new revisions were added by this update. Summary of changes: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 2 -- 1 file changed, 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0c6bd3b -> db7262a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0c6bd3b [SPARK-27142][SQL] Provide REST API for SQL information add db7262a [SPARK-30509][SQL] Fix deprecation log warning in Avro schema inferring No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/avro/AvroUtils.scala | 9 .../org/apache/spark/sql/avro/AvroSuite.scala | 24 ++ 2 files changed, 29 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0c6bd3b -> db7262a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0c6bd3b [SPARK-27142][SQL] Provide REST API for SQL information add db7262a [SPARK-30509][SQL] Fix deprecation log warning in Avro schema inferring No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/avro/AvroUtils.scala | 9 .../org/apache/spark/sql/avro/AvroSuite.scala | 24 ++ 2 files changed, 29 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9320011 -> 0c6bd3b)
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9320011 [SPARK-9478][ML][PYSPARK] Add sample weights to Random Forest add 0c6bd3b [SPARK-27142][SQL] Provide REST API for SQL information No new revisions were added by this update. Summary of changes: .../spark/sql/execution/ui/SQLAppStatusStore.scala | 4 + .../status/api/v1/sql/ApiSqlRootResource.scala | 16 ++-- .../spark/status/api/v1/sql/SqlResource.scala | 101 + .../org/apache/spark/status/api/v1/sql/api.scala | 20 +++- 4 files changed, 127 insertions(+), 14 deletions(-) copy common/unsafe/src/main/java/org/apache/spark/unsafe/KVIterator.java => sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/ApiSqlRootResource.scala (73%) create mode 100644 sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/SqlResource.scala copy core/src/main/scala/org/apache/spark/metrics/sink/package.scala => sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/api.scala (65%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9320011 -> 0c6bd3b)
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9320011 [SPARK-9478][ML][PYSPARK] Add sample weights to Random Forest add 0c6bd3b [SPARK-27142][SQL] Provide REST API for SQL information No new revisions were added by this update. Summary of changes: .../spark/sql/execution/ui/SQLAppStatusStore.scala | 4 + .../status/api/v1/sql/ApiSqlRootResource.scala | 16 ++-- .../spark/status/api/v1/sql/SqlResource.scala | 101 + .../org/apache/spark/status/api/v1/sql/api.scala | 20 +++- 4 files changed, 127 insertions(+), 14 deletions(-) copy common/unsafe/src/main/java/org/apache/spark/unsafe/KVIterator.java => sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/ApiSqlRootResource.scala (73%) create mode 100644 sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/SqlResource.scala copy core/src/main/scala/org/apache/spark/metrics/sink/package.scala => sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/api.scala (65%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (176b696 -> 9320011)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 176b696 [SPARK-30423][SQL] Deprecate UserDefinedAggregateFunction add 9320011 [SPARK-9478][ML][PYSPARK] Add sample weights to Random Forest No new revisions were added by this update. Summary of changes: .../ml/classification/RandomForestClassifier.scala | 26 +++ .../ml/regression/RandomForestRegressor.scala | 23 ++--- .../apache/spark/ml/tree/impl/BaggedPoint.scala| 6 ++-- .../RandomForestClassifierSuite.scala | 35 +++ .../ml/regression/RandomForestRegressorSuite.scala | 39 ++ .../spark/ml/tree/impl/BaggedPointSuite.scala | 3 +- python/pyspark/ml/classification.py| 24 ++--- python/pyspark/ml/regression.py| 28 +--- 8 files changed, 161 insertions(+), 23 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a2aa966 -> 176b696)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a2aa966 [SPARK-29544][SQL] optimize skewed partition based on data size add 176b696 [SPARK-30423][SQL] Deprecate UserDefinedAggregateFunction No new revisions were added by this update. Summary of changes: docs/sql-getting-started.md| 23 ++-- .../sql/JavaUserDefinedUntypedAggregation.java | 116 +++-- .../sql/UserDefinedUntypedAggregation.scala| 62 +-- .../org/apache/spark/sql/UDFRegistration.scala | 4 + .../org/apache/spark/sql/expressions/udaf.scala| 4 + 5 files changed, 105 insertions(+), 104 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2688fae -> a2aa966)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2688fae [SPARK-30498][ML][PYSPARK] Fix some ml parity issues between python and scala add a2aa966 [SPARK-29544][SQL] optimize skewed partition based on data size No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/MapOutputTracker.scala | 62 +++-- .../org/apache/spark/shuffle/ShuffleManager.scala | 10 +- .../spark/shuffle/sort/SortShuffleManager.scala| 9 +- .../org/apache/spark/sql/internal/SQLConf.scala| 30 +++ .../spark/sql/execution/ShuffledRowRDD.scala | 23 +- .../execution/adaptive/AdaptiveSparkPlanExec.scala | 4 + .../execution/adaptive/LocalShuffledRowRDD.scala | 6 +- .../adaptive/OptimizeLocalShuffleReader.scala | 2 +- .../execution/adaptive/OptimizeSkewedJoin.scala| 293 + .../adaptive/ReduceNumShufflePartitions.scala | 125 + .../execution/adaptive/SkewedShuffledRowRDD.scala | 78 ++ .../execution/exchange/ShuffleExchangeExec.scala | 15 +- .../ReduceNumShufflePartitionsSuite.scala | 5 +- .../adaptive/AdaptiveQueryExecSuite.scala | 147 +++ 14 files changed, 703 insertions(+), 106 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/SkewedShuffledRowRDD.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2688fae -> a2aa966)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2688fae [SPARK-30498][ML][PYSPARK] Fix some ml parity issues between python and scala add a2aa966 [SPARK-29544][SQL] optimize skewed partition based on data size No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/MapOutputTracker.scala | 62 +++-- .../org/apache/spark/shuffle/ShuffleManager.scala | 10 +- .../spark/shuffle/sort/SortShuffleManager.scala| 9 +- .../org/apache/spark/sql/internal/SQLConf.scala| 30 +++ .../spark/sql/execution/ShuffledRowRDD.scala | 23 +- .../execution/adaptive/AdaptiveSparkPlanExec.scala | 4 + .../execution/adaptive/LocalShuffledRowRDD.scala | 6 +- .../adaptive/OptimizeLocalShuffleReader.scala | 2 +- .../execution/adaptive/OptimizeSkewedJoin.scala| 293 + .../adaptive/ReduceNumShufflePartitions.scala | 125 + .../execution/adaptive/SkewedShuffledRowRDD.scala | 78 ++ .../execution/exchange/ShuffleExchangeExec.scala | 15 +- .../ReduceNumShufflePartitionsSuite.scala | 5 +- .../adaptive/AdaptiveQueryExecSuite.scala | 147 +++ 14 files changed, 703 insertions(+), 106 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/SkewedShuffledRowRDD.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4462756 -> 2688fae)
This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4462756 [SPARK-30325][CORE] markPartitionCompleted cause task status inconsistent add 2688fae [SPARK-30498][ML][PYSPARK] Fix some ml parity issues between python and scala No new revisions were added by this update. Summary of changes: python/pyspark/ml/classification.py | 8 -- python/pyspark/ml/clustering.py | 4 --- python/pyspark/ml/feature.py| 42 +++ python/pyspark/ml/regression.py | 1 - python/pyspark/ml/tuning.py | 50 +++-- 5 files changed, 18 insertions(+), 87 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e0efd21 -> 4462756)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e0efd21 [SPARK-30292][SQL] Throw Exception when invalid string is cast to numeric type in ANSI mode add 4462756 [SPARK-30325][CORE] markPartitionCompleted cause task status inconsistent No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (88fc8db -> e0efd21)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 88fc8db [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests add e0efd21 [SPARK-30292][SQL] Throw Exception when invalid string is cast to numeric type in ANSI mode No new revisions were added by this update. Summary of changes: .../org/apache/spark/unsafe/types/UTF8String.java | 46 + .../spark/sql/catalyst/expressions/Cast.scala | 59 +- .../spark/sql/catalyst/expressions/CastSuite.scala | 221 ++--- .../sql-tests/results/postgreSQL/float4.sql.out| 20 +- .../sql-tests/results/postgreSQL/float8.sql.out| 20 +- .../sql-tests/results/postgreSQL/text.sql.out | 10 +- .../results/postgreSQL/window_part2.sql.out| 11 +- .../results/postgreSQL/window_part4.sql.out| 9 +- 8 files changed, 278 insertions(+), 118 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (88fc8db -> e0efd21)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 88fc8db [SPARK-30482][SQL][CORE][TESTS] Add sub-class of `AppenderSkeleton` reusable in tests add e0efd21 [SPARK-30292][SQL] Throw Exception when invalid string is cast to numeric type in ANSI mode No new revisions were added by this update. Summary of changes: .../org/apache/spark/unsafe/types/UTF8String.java | 46 + .../spark/sql/catalyst/expressions/Cast.scala | 59 +- .../spark/sql/catalyst/expressions/CastSuite.scala | 221 ++--- .../sql-tests/results/postgreSQL/float4.sql.out| 20 +- .../sql-tests/results/postgreSQL/float8.sql.out| 20 +- .../sql-tests/results/postgreSQL/text.sql.out | 10 +- .../results/postgreSQL/window_part2.sql.out| 11 +- .../results/postgreSQL/window_part4.sql.out| 9 +- 8 files changed, 278 insertions(+), 118 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
