date:20211227

[spark] branch master updated (780cf3c -> 67c39a0)

2021-12-27 Thread wenchen

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 780cf3c  [SPARK-37747][BUILD][FOLLOWUP] Update 
MapStatusesSerDeserBenchmark results
 add 67c39a0  [SPARK-37716][SQL] Improve error messages when a LateralJoin 
has non-deterministic expressions

No new revisions were added by this update.

Summary of changes:
 .../sql/catalyst/analysis/CheckAnalysis.scala  | 17 ++-
 .../resources/sql-tests/inputs/join-lateral.sql| 10 
 .../sql-tests/results/join-lateral.sql.out | 56 +-
 3 files changed, 81 insertions(+), 2 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (67c39a0 -> 75f919e)

2021-12-27 Thread wenchen

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 67c39a0  [SPARK-37716][SQL] Improve error messages when a LateralJoin 
has non-deterministic expressions
 add 75f919e  [SPARK-37732][SQL][TEST] Improve the implement of JDBCV2Suite

No new revisions were added by this update.

Summary of changes:
 .../org/apache/spark/sql/jdbc/JDBCV2Suite.scala| 59 --
 1 file changed, 22 insertions(+), 37 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (d8920ae -> 6c69262)

2021-12-27 Thread wenchen

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from d8920ae  [SPARK-35703][SQL] Relax constraint for bucket join and 
remove HashClusteredDistribution
 add 6c69262  [SPARK-37496][SQL][FOLLOWUP] DataFrameWriter.saveAsTable 
should take write options in CTAS and RTAS

No new revisions were added by this update.

Summary of changes:
 .../org/apache/spark/sql/DataFrameWriter.scala |  4 +-
 .../sql/connector/DataSourceV2DataFrameSuite.scala | 48 +-
 2 files changed, 49 insertions(+), 3 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-37676][SQL] Support ANSI Aggregation Function: percentile_cont

2021-12-27 Thread maxgekk

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new 73789da  [SPARK-37676][SQL] Support ANSI Aggregation Function: 
percentile_cont
73789da is described below

commit 73789da962c9037bde21a53fb5826b10475658fe
Author: Jiaan Geng 
AuthorDate: Mon Dec 27 16:12:43 2021 +0300

[SPARK-37676][SQL] Support ANSI Aggregation Function: percentile_cont

### What changes were proposed in this pull request?
`PERCENTILE_CONT` is an ANSI aggregate functions.

The mainstream database supports `percentile_cont` show below:
**Postgresql**
https://www.postgresql.org/docs/9.4/functions-aggregate.html
**Teradata**
https://docs.teradata.com/r/kmuOwjp1zEYg98JsB8fu_A/cPkFySIBORL~M938Zv07Cg
**Snowflake**
https://docs.snowflake.com/en/sql-reference/functions/percentile_cont.html
**Oracle**

https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/PERCENTILE_CONT.html#GUID-CA259452-A565-41B3-A4F4-DD74B66CEDE0
**H2**
http://www.h2database.com/html/functions-aggregate.html#percentile_cont
**Sybase**

https://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc01776.1601/doc/html/san1278453109663.html
**Exasol**

https://docs.exasol.com/sql_references/functions/alphabeticallistfunctions/percentile_cont.htm
**RedShift**
https://docs.aws.amazon.com/redshift/latest/dg/r_PERCENTILE_CONT.html
**Yellowbrick**
https://www.yellowbrick.com/docs/2.2/ybd_sqlref/percentile_cont.html
**Mariadb**
https://mariadb.com/kb/en/percentile_cont/
**Phoenix**
http://phoenix.incubator.apache.org/language/functions.html#percentile_cont
**Singlestore**

https://docs.singlestore.com/db/v7.6/en/reference/sql-reference/window-functions/percentile_cont-and-median.html

### Why are the changes needed?
`PERCENTILE_CONT` is very useful. Exposing the expression can make the 
migration from other systems to Spark SQL easier.

### Does this PR introduce _any_ user-facing change?
'Yes'. New feature.

### How was this patch tested?
New tests.

Closes #34936 from beliefer/SPARK-37676.

Authored-by: Jiaan Geng 
Signed-off-by: Max Gekk 
---
 docs/sql-ref-ansi-compliance.md|  2 ++
 .../apache/spark/sql/catalyst/parser/SqlBase.g4|  6 
 .../spark/sql/catalyst/parser/AstBuilder.scala | 15 +++-
 .../sql/catalyst/parser/PlanParserSuite.scala  | 24 -
 .../test/resources/sql-tests/inputs/group-by.sql   | 18 +-
 .../resources/sql-tests/results/group-by.sql.out   | 41 +-
 6 files changed, 102 insertions(+), 4 deletions(-)

diff --git a/docs/sql-ref-ansi-compliance.md b/docs/sql-ref-ansi-compliance.md
index 7b5bde4..1b4a778 100644
--- a/docs/sql-ref-ansi-compliance.md
+++ b/docs/sql-ref-ansi-compliance.md
@@ -494,6 +494,7 @@ Below is a list of all the keywords in Spark SQL.
 |PARTITIONED|non-reserved|non-reserved|non-reserved|
 |PARTITIONS|non-reserved|non-reserved|non-reserved|
 |PERCENT|non-reserved|non-reserved|non-reserved|
+|PERCENTILE_CONT|reserved|non-reserved|non-reserved|
 |PIVOT|non-reserved|non-reserved|non-reserved|
 |PLACING|non-reserved|non-reserved|non-reserved|
 |POSITION|non-reserved|non-reserved|reserved|
@@ -594,5 +595,6 @@ Below is a list of all the keywords in Spark SQL.
 |WHERE|reserved|non-reserved|reserved|
 |WINDOW|non-reserved|non-reserved|reserved|
 |WITH|reserved|non-reserved|reserved|
+|WITHIN|reserved|non-reserved|reserved|
 |YEAR|non-reserved|non-reserved|non-reserved|
 |ZONE|non-reserved|non-reserved|non-reserved|
diff --git 
a/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 
b/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
index 6511489..5037520 100644
--- 
a/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
+++ 
b/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
@@ -888,6 +888,8 @@ primaryExpression
FROM srcStr=valueExpression ')' 
#trim
 | OVERLAY '(' input=valueExpression PLACING replace=valueExpression
   FROM position=valueExpression (FOR length=valueExpression)? ')'  
#overlay
+| PERCENTILE_CONT '(' percentage=valueExpression ')'
+  WITHIN GROUP '(' ORDER BY sortItem ')'   
#percentile
 ;
 
 constant
@@ -1475,6 +1477,7 @@ nonReserved
 | PARTITION
 | PARTITIONED
 | PARTITIONS
+| PERCENTILE_CONT
 | PERCENTLIT
 | PIVOT
 | PLACING
@@ -1570,6 +1573,7 @@ nonReserved
 | WHERE
 | WINDOW
 | WITH
+| WITHIN
 | YEAR
 | ZONE
 //--DEFAULT-NON-RESERVED-END
@@ -1747,6 +1751,7 @

[spark] branch master updated (73789da -> f4d6a0f)

2021-12-27 Thread viirya

This is an automated email from the ASF dual-hosted git repository.

viirya pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 73789da  [SPARK-37676][SQL] Support ANSI Aggregation Function: 
percentile_cont
 add f4d6a0f  [SPARK-37746][CORE] log4j2-defaults.properties is not working 
since log4j 2 is always initialized by default

No new revisions were added by this update.

Summary of changes:
 .../org/apache/spark/log4j2-defaults.properties |  2 +-
 .../main/scala/org/apache/spark/internal/Logging.scala  | 17 +++--
 2 files changed, 12 insertions(+), 7 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations

2021-12-27 Thread dongjoon

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new 285d8dd  [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
285d8dd is described below

commit 285d8dd9f0b8ee8b85603b4cc0f656c452bff629
Author: Dongjoon Hyun 
AuthorDate: Mon Dec 27 17:30:22 2021 -0800

[SPARK-37755][CORE] Optimize RocksDB KVStore configurations

### What changes were proposed in this pull request?

This PR aims to optimize RocksDB KVStore configurations to improve the 
performance up to 4 times.

### Why are the changes needed?

Currently, `RocksDB` KVStore performance is not optimized.

**REFERENCE (LevelDB) **
```
[INFO] Running org.apache.spark.util.kvstore.LevelDBBenchmark
count   meanmin max 
95th
dbClose 4   0.802   0.777   0.824   
0.824
dbCreation  4   71.029  2.528   278.049 
278.049
naturalIndexCreateIterator  10240.006   0.002   1.559   
0.007
naturalIndexDescendingCreateIterator10240.003   0.003   0.043   
0.004
naturalIndexDescendingIteration 10240.007   0.005   0.053   
0.008
naturalIndexIteration   10240.007   0.005   0.080   
0.009
randomDeleteIndexed 10240.020   0.017   0.148   
0.024
randomDeletesNoIndex10240.025   0.008   14.739  
0.013
randomUpdatesIndexed10240.079   0.033   23.527  
0.075
randomUpdatesNoIndex10240.078   0.072   0.584   
0.082
randomWritesIndexed 10240.121   0.043   42.030  
0.141
randomWritesNoIndex 10240.086   0.059   1.204   
0.090
refIndexCreateIterator  10240.003   0.002   0.033   
0.004
refIndexDescendingCreateIterator10240.002   0.002   0.018   
0.003
refIndexDescendingIteration 10240.007   0.006   0.045   
0.008
refIndexIteration   10240.009   0.006   0.261   
0.010
sequentialDeleteIndexed 10240.019   0.016   0.090   
0.024
sequentialDeleteNoIndex 10240.013   0.010   0.083   
0.014
sequentialUpdatesIndexed10240.031   0.027   0.560   
0.035
sequentialUpdatesNoIndex10240.028   0.024   0.618   
0.032
sequentialWritesIndexed 10240.037   0.031   1.301   
0.041
sequentialWritesNoIndex 10240.038   0.026   1.699   
0.065
```

**AS-IS (RocksDB)**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count   meanmin max 
95th
dbClose 4   0.621   0.455   0.830   
0.830
dbCreation  4   93.812  4.566   365.368 
365.368
naturalIndexCreateIterator  10240.005   0.002   1.349   
0.007
naturalIndexDescendingCreateIterator10240.006   0.005   0.054   
0.007
naturalIndexDescendingIteration 10240.007   0.005   0.034   
0.009
naturalIndexIteration   10240.007   0.005   0.062   
0.009
randomDeleteIndexed 10240.138   0.079   0.636   
0.186
randomDeletesNoIndex10240.104   0.058   0.223   
0.143
randomUpdatesIndexed10240.203   0.145   23.849  
0.243
randomUpdatesNoIndex10240.161   0.122   0.509   
0.196
randomWritesIndexed 10240.239   0.149   42.780  
0.279
randomWritesNoIndex 10240.143   0.080   1.593   
0.221
refIndexCreateIterator  10240.003   0.003   0.032   
0.005
refIndexDescendingCreateIterator10240.003   0.003   0.028   
0.005
refIndexDescendingIteration 10240.008   0.006   0.038   
0.009
refIndexIteration   10240.009   0.007   0.265   
0.011
sequentialDeleteIndexed 10240.122   0.106   0.251   
0.150
sequentialDeleteNoIndex 10240.101   0.059   0.315   
0.121
sequentialUpdatesIndexed10240.147   0.109   0.812   
0.197
sequentialUpdatesNoIndex10240.143   0.112   0.896   
0.184
sequentialWritesIndexed 10240.170   0.130   1.715   
0.235
sequentialWritesNoIndex 1024

[spark] branch master updated (285d8dd -> 371e307)

2021-12-27 Thread gurwls223

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 285d8dd  [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
 add 371e307  [SPARK-37730][PYTHON] Replace use of 
MPLPlot._add_legend_handle with MPLPlot._append_legend_handles_labels

No new revisions were added by this update.

Summary of changes:
 python/pyspark/pandas/plot/matplotlib.py | 22 --
 1 file changed, 20 insertions(+), 2 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (371e307 -> 80dca37)

2021-12-27 Thread dongjoon

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 371e307  [SPARK-37730][PYTHON] Replace use of 
MPLPlot._add_legend_handle with MPLPlot._append_legend_handles_labels
 add 80dca37  [SPARK-37760][BUILD] Upgrade SBT to 1.6.0

No new revisions were added by this update.

Summary of changes:
 dev/appveyor-install-dependencies.ps1 | 2 +-
 project/build.properties  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (80dca37 -> 78899e3)

2021-12-27 Thread gurwls223

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 80dca37  [SPARK-37760][BUILD] Upgrade SBT to 1.6.0
 add 78899e3  [SPARK-37756][PYTHON] Enable matplotlib test for pandas API 
on Spark

No new revisions were added by this update.

Summary of changes:
 .github/workflows/build_and_test.yml | 3 +++
 1 file changed, 3 insertions(+)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-37728][SQL] Reading nested columns with ORC vectorized reader can cause ArrayIndexOutOfBoundsException

2021-12-27 Thread dongjoon

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new d245aef  [SPARK-37728][SQL] Reading nested columns with ORC vectorized 
reader can cause ArrayIndexOutOfBoundsException
d245aef is described below

commit d245aefe87cc974fe7fc37587d09bec12c1bf911
Author: Yimin 
AuthorDate: Mon Dec 27 22:09:51 2021 -0800

[SPARK-37728][SQL] Reading nested columns with ORC vectorized reader can 
cause ArrayIndexOutOfBoundsException

### What changes were proposed in this pull request?

When an OrcColumnarBatchReader is created, method initBatch will be called 
only once. In method initBatch:

`orcVectorWrappers[i] = OrcColumnVectorUtils.toOrcColumnVector(dt, 
wrap.batch().cols[colId]);`

When the second argument of toOrcColumnVector is a 
ListColumnVector/MapColumnVector, orcVectorWrappers[i] is initialized with the 
ListColumnVector or MapColumnVector's offsets and lengths.

However, when method nextBatch of OrcColumnarBatchReader is called, method 
ensureSize of ColumnVector (and its subclasses, like MultiValuedColumnVector) 
could be called, then the ListColumnVector/MapColumnVector's offsets and 
lengths could refer to new array objects. This could result in the 
ArrayIndexOutOfBoundsException.

This PR makes OrcArrayColumnVector.getArray and OrcMapColumnVector.getMap 
always get offsets and lengths from the underlying ColumnVector, which can 
resolve this issue.

### Why are the changes needed?

Bugfix

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Pass the CIs with the newly added test case.

Closes #35002 from yym1995/fix-nested.

Lead-authored-by: Yimin 
Co-authored-by: Yimin <26797163+yym1...@users.noreply.github.com>
Signed-off-by: Dongjoon Hyun 
---
 .../datasources/orc/OrcArrayColumnVector.java  | 13 +
 .../execution/datasources/orc/OrcColumnVector.java |  2 +-
 .../datasources/orc/OrcColumnVectorUtils.java  |  6 ++
 .../datasources/orc/OrcMapColumnVector.java| 13 +
 .../execution/datasources/orc/OrcQuerySuite.scala  | 22 ++
 5 files changed, 35 insertions(+), 21 deletions(-)

diff --git 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcArrayColumnVector.java
 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcArrayColumnVector.java
index 6e13e97..b0c818f 100644
--- 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcArrayColumnVector.java
+++ 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcArrayColumnVector.java
@@ -18,6 +18,7 @@
 package org.apache.spark.sql.execution.datasources.orc;
 
 import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.ListColumnVector;
 
 import org.apache.spark.sql.types.ArrayType;
 import org.apache.spark.sql.types.DataType;
@@ -31,26 +32,22 @@ import org.apache.spark.unsafe.types.UTF8String;
  */
 public class OrcArrayColumnVector extends OrcColumnVector {
   private final OrcColumnVector data;
-  private final long[] offsets;
-  private final long[] lengths;
 
   OrcArrayColumnVector(
   DataType type,
   ColumnVector vector,
-  OrcColumnVector data,
-  long[] offsets,
-  long[] lengths) {
+  OrcColumnVector data) {
 
 super(type, vector);
 
 this.data = data;
-this.offsets = offsets;
-this.lengths = lengths;
   }
 
   @Override
   public ColumnarArray getArray(int rowId) {
-return new ColumnarArray(data, (int) offsets[rowId], (int) lengths[rowId]);
+int offsets = (int) ((ListColumnVector) baseData).offsets[rowId];
+int lengths = (int) ((ListColumnVector) baseData).lengths[rowId];
+return new ColumnarArray(data, offsets, lengths);
   }
 
   @Override
diff --git 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
index 0becd25..7fe1b30 100644
--- 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
+++ 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
@@ -29,7 +29,7 @@ import org.apache.spark.sql.vectorized.ColumnarBatch;
  * this column vector is used to adapt Hive ColumnVector with Spark 
ColumnarVector.
  */
 public abstract class OrcColumnVector extends 
org.apache.spark.sql.vectorized.ColumnVector {
-  private final ColumnVector baseData;
+  protected final ColumnVector baseData;
   private int batchSize;
 
   OrcColumnVector(DataType type, ColumnVector vector) {
diff --git 
a/sql/core/src/main/java/org/apache/spark/sql/execution/da

[spark] branch master updated: [SPARK-37715][SQL][BUILD][TESTS] Remove `ojdbc6` dependency and update docker-integration test docs

2021-12-27 Thread dongjoon

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new 624d3b7  [SPARK-37715][SQL][BUILD][TESTS] Remove `ojdbc6` dependency 
and update docker-integration test docs
624d3b7 is described below

commit 624d3b7c49abbf6f7bd82fa519b9270e56c660ee
Author: Danny Guinther 
AuthorDate: Mon Dec 27 22:39:50 2021 -0800

[SPARK-37715][SQL][BUILD][TESTS] Remove `ojdbc6` dependency and update 
docker-integration test docs

### What changes were proposed in this pull request?

This PR aims to remove unused `ojdbc6` dependency because we are using 
`ojdbc8` in `pom.xml`.
In addition, this updates two image names in documentation.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

Pass the GitHub Action Docker test.

Closes #34979 from tdg5/just-the-docker-stuff.

Authored-by: Danny Guinther 
Signed-off-by: Dongjoon Hyun 
---
 .../scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala  | 3 ++-
 .../org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala | 3 ++-
 project/SparkBuild.scala   | 3 +--
 3 files changed, 5 insertions(+), 4 deletions(-)

diff --git 
a/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
 
b/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
index 78015d8..5992253 100644
--- 
a/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
+++ 
b/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
@@ -28,7 +28,8 @@ import org.apache.spark.tags.DockerTest
 /**
  * To run this test suite for a specific version (e.g., 
2019-CU13-ubuntu-20.04):
  * {{{
- *   ENABLE_DOCKER_INTEGRATION_TESTS=1 
MSSQLSERVER_DOCKER_IMAGE_NAME=2019-CU13-ubuntu-20.04
+ *   ENABLE_DOCKER_INTEGRATION_TESTS=1
+ *   
MSSQLSERVER_DOCKER_IMAGE_NAME=mcr.microsoft.com/mssql/server:2019-CU13-ubuntu-20.04
  * ./build/sbt -Pdocker-integration-tests
  * "testOnly org.apache.spark.sql.jdbc.MsSqlServerIntegrationSuite"
  * }}}
diff --git 
a/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala
 
b/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala
index 77b7742..75446fb 100644
--- 
a/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala
+++ 
b/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MsSqlServerIntegrationSuite.scala
@@ -31,7 +31,8 @@ import org.apache.spark.tags.DockerTest
 /**
  * To run this test suite for a specific version (e.g., 
2019-CU13-ubuntu-20.04):
  * {{{
- *   ENABLE_DOCKER_INTEGRATION_TESTS=1 
MSSQLSERVER_DOCKER_IMAGE_NAME=2019-CU13-ubuntu-20.04
+ *   ENABLE_DOCKER_INTEGRATION_TESTS=1
+ *   
MSSQLSERVER_DOCKER_IMAGE_NAME=mcr.microsoft.com/mssql/server:2019-CU13-ubuntu-20.04
  * ./build/sbt -Pdocker-integration-tests "testOnly 
*v2*MsSqlServerIntegrationSuite"
  * }}}
  */
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 2f21561..579822a 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -599,8 +599,7 @@ object DockerIntegrationTests {
   // This serves to override the override specified in DependencyOverrides:
   lazy val settings = Seq(
 dependencyOverrides += "com.google.guava" % "guava" % "18.0",
-resolvers += "DB2" at 
"https://app.camunda.com/nexus/content/repositories/public/";,
-libraryDependencies += "com.oracle" % "ojdbc6" % "11.2.0.1.0" from 
"https://app.camunda.com/nexus/content/repositories/public/com/oracle/ojdbc6/11.2.0.1.0/ojdbc6-11.2.0.1.0.jar";
 // scalastyle:ignore
+resolvers += "DB2" at 
"https://app.camunda.com/nexus/content/repositories/public/";
   )
 }
 

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-37763][BUILD] Upgrade jackson to 2.13.1

2021-12-27 Thread dongjoon

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new e653b8c  [SPARK-37763][BUILD] Upgrade jackson to 2.13.1
e653b8c is described below

commit e653b8cd89ea13c22823885c71cfb26514c72a03
Author: William Hyun 
AuthorDate: Mon Dec 27 23:28:39 2021 -0800

[SPARK-37763][BUILD] Upgrade jackson to 2.13.1

### What changes were proposed in this pull request?
This PR aims to upgrade jackson to 2.13.1.

### Why are the changes needed?
This release has the following fix.
- https://github.com/FasterXML/jackson-core/issues/721

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass the CIs.

Closes #35037 from williamhyun/jackson2131.

Authored-by: William Hyun 
Signed-off-by: Dongjoon Hyun 
---
 dev/deps/spark-deps-hadoop-2-hive-2.3 | 12 ++--
 dev/deps/spark-deps-hadoop-3-hive-2.3 | 12 ++--
 pom.xml   |  2 +-
 3 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 
b/dev/deps/spark-deps-hadoop-2-hive-2.3
index e38213f..9ad5f18 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
@@ -111,16 +111,16 @@ httpclient/4.5.13//httpclient-4.5.13.jar
 httpcore/4.4.14//httpcore-4.4.14.jar
 istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
 ivy/2.5.0//ivy-2.5.0.jar
-jackson-annotations/2.13.0//jackson-annotations-2.13.0.jar
+jackson-annotations/2.13.1//jackson-annotations-2.13.1.jar
 jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
-jackson-core/2.13.0//jackson-core-2.13.0.jar
-jackson-databind/2.13.0//jackson-databind-2.13.0.jar
-jackson-dataformat-cbor/2.13.0//jackson-dataformat-cbor-2.13.0.jar
-jackson-dataformat-yaml/2.13.0//jackson-dataformat-yaml-2.13.0.jar
+jackson-core/2.13.1//jackson-core-2.13.1.jar
+jackson-databind/2.13.1//jackson-databind-2.13.1.jar
+jackson-dataformat-cbor/2.13.1//jackson-dataformat-cbor-2.13.1.jar
+jackson-dataformat-yaml/2.13.1//jackson-dataformat-yaml-2.13.1.jar
 jackson-datatype-jsr310/2.13.0//jackson-datatype-jsr310-2.13.0.jar
 jackson-jaxrs/1.9.13//jackson-jaxrs-1.9.13.jar
 jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
-jackson-module-scala_2.12/2.13.0//jackson-module-scala_2.12-2.13.0.jar
+jackson-module-scala_2.12/2.13.1//jackson-module-scala_2.12-2.13.1.jar
 jackson-xc/1.9.13//jackson-xc-1.9.13.jar
 jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
 jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index e0286e6..a482b33 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -103,15 +103,15 @@ httpclient/4.5.13//httpclient-4.5.13.jar
 httpcore/4.4.14//httpcore-4.4.14.jar
 istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
 ivy/2.5.0//ivy-2.5.0.jar
-jackson-annotations/2.13.0//jackson-annotations-2.13.0.jar
+jackson-annotations/2.13.1//jackson-annotations-2.13.1.jar
 jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
-jackson-core/2.13.0//jackson-core-2.13.0.jar
-jackson-databind/2.13.0//jackson-databind-2.13.0.jar
-jackson-dataformat-cbor/2.13.0//jackson-dataformat-cbor-2.13.0.jar
-jackson-dataformat-yaml/2.13.0//jackson-dataformat-yaml-2.13.0.jar
+jackson-core/2.13.1//jackson-core-2.13.1.jar
+jackson-databind/2.13.1//jackson-databind-2.13.1.jar
+jackson-dataformat-cbor/2.13.1//jackson-dataformat-cbor-2.13.1.jar
+jackson-dataformat-yaml/2.13.1//jackson-dataformat-yaml-2.13.1.jar
 jackson-datatype-jsr310/2.13.0//jackson-datatype-jsr310-2.13.0.jar
 jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
-jackson-module-scala_2.12/2.13.0//jackson-module-scala_2.12-2.13.0.jar
+jackson-module-scala_2.12/2.13.1//jackson-module-scala_2.12-2.13.1.jar
 jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
 jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
 jakarta.servlet-api/4.0.3//jakarta.servlet-api-4.0.3.jar
diff --git a/pom.xml b/pom.xml
index 0facb65..7314eb1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -167,7 +167,7 @@
 
 true
 1.9.13
-2.13.0
+2.13.1
 1.1.8.4
 1.1.2
 2.2.1

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (780cf3c -> 67c39a0)

[spark] branch master updated (67c39a0 -> 75f919e)

[spark] branch master updated (d8920ae -> 6c69262)

[spark] branch master updated: [SPARK-37676][SQL] Support ANSI Aggregation Function: percentile_cont

[spark] branch master updated (73789da -> f4d6a0f)

[spark] branch master updated: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations

[spark] branch master updated (285d8dd -> 371e307)

[spark] branch master updated (371e307 -> 80dca37)

[spark] branch master updated (80dca37 -> 78899e3)

[spark] branch master updated: [SPARK-37728][SQL] Reading nested columns with ORC vectorized reader can cause ArrayIndexOutOfBoundsException

[spark] branch master updated: [SPARK-37715][SQL][BUILD][TESTS] Remove `ojdbc6` dependency and update docker-integration test docs

[spark] branch master updated: [SPARK-37763][BUILD] Upgrade jackson to 2.13.1

12 matches

Site Navigation

Mail list logo

Footer information