[jira] [Created] (SPARK-47749) Dataframe.collect should accept duplicated column names
Ruifeng Zheng created SPARK-47749:
-
Summary: Dataframe.collect should accept duplicated column names
Key: SPARK-47749
URL: https://issues.apache.org/jira/browse/SPARK-47749
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 4.0.0
Reporter: Ruifeng Zheng
{code:java}
+---+---+---+---+
| i| j| i| j|
+---+---+---+---+
| 1| a| 1| a|
+---+---+---+---+ {code}
collect fails with
{code:java}
[info] org.apache.spark.sql.AnalysisException: [AMBIGUOUS_COLUMN_OR_FIELD]
Column or field `i` is ambiguous and has 2 matches. SQLSTATE: 42702
[info] at
org.apache.spark.sql.errors.CompilationErrors.ambiguousColumnOrFieldError(CompilationErrors.scala:28)
[info] at
org.apache.spark.sql.errors.CompilationErrors.ambiguousColumnOrFieldError$(CompilationErrors.scala:23)
[info] at
org.apache.spark.sql.errors.CompilationErrors$.ambiguousColumnOrFieldError(CompilationErrors.scala:54)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.$anonfun$createFieldLookup$1(ArrowDeserializer.scala:460)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.$anonfun$createFieldLookup$1$adapted(ArrowDeserializer.scala:454)
[info] at scala.collection.immutable.List.foreach(List.scala:334)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.createFieldLookup(ArrowDeserializer.scala:454)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:328)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:86)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializingIterator.(ArrowDeserializer.scala:542)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47748) Upgrade `zstd-jni` to 1.5.6-2
BingKun Pan created SPARK-47748: --- Summary: Upgrade `zstd-jni` to 1.5.6-2 Key: SPARK-47748 URL: https://issues.apache.org/jira/browse/SPARK-47748 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: BingKun Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47750) Postgres: Document Mapping Spark SQL Data Types to PostgreSQL
Kent Yao created SPARK-47750: Summary: Postgres: Document Mapping Spark SQL Data Types to PostgreSQL Key: SPARK-47750 URL: https://issues.apache.org/jira/browse/SPARK-47750 Project: Spark Issue Type: Sub-task Components: Documentation, SQL Affects Versions: 4.0.0 Reporter: Kent Yao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47750) Postgres: Document Mapping Spark SQL Data Types to PostgreSQL
[ https://issues.apache.org/jira/browse/SPARK-47750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47750: --- Labels: pull-request-available (was: ) > Postgres: Document Mapping Spark SQL Data Types to PostgreSQL > - > > Key: SPARK-47750 > URL: https://issues.apache.org/jira/browse/SPARK-47750 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 4.0.0 >Reporter: Kent Yao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47595) Streaming: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47595: --- Labels: pull-request-available (was: ) > Streaming: Migrate logError with variables to structured logging framework > -- > > Key: SPARK-47595 > URL: https://issues.apache.org/jira/browse/SPARK-47595 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47748) Upgrade `zstd-jni` to 1.5.6-2
[ https://issues.apache.org/jira/browse/SPARK-47748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47748: --- Labels: pull-request-available (was: ) > Upgrade `zstd-jni` to 1.5.6-2 > - > > Key: SPARK-47748 > URL: https://issues.apache.org/jira/browse/SPARK-47748 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47592) Connector module: Migrate logError with variables to structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-47592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47592. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45877 [https://github.com/apache/spark/pull/45877] > Connector module: Migrate logError with variables to structured logging > framework > - > > Key: SPARK-47592 > URL: https://issues.apache.org/jira/browse/SPARK-47592 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: BingKun Pan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47745) Add License to Spark Operator
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47745: --- Labels: pull-request-available (was: ) > Add License to Spark Operator > - > > Key: SPARK-47745 > URL: https://issues.apache.org/jira/browse/SPARK-47745 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 4.0.0 >Reporter: Zhou JIANG >Priority: Major > Labels: pull-request-available > > Add license to the recently established operator repository. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47719) Change default of spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED
[ https://issues.apache.org/jira/browse/SPARK-47719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47719: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Change default of spark.sql.legacy.timeParserPolicy from EXCEPTION to > CORRECTED > --- > > Key: SPARK-47719 > URL: https://issues.apache.org/jira/browse/SPARK-47719 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Serge Rielau >Assignee: Serge Rielau >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > spark.sql.legacy.timeParserPolicy was introduced in Spark 3.0 and has been > set to EXCEPTION. > Changing it from EXCEPTION for SPark 4.0 to CORRECTED will reduce errors and > reflects a prudent timeframe. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47727) Make SparkConf to root level to for both SparkSession and SparkContext
[ https://issues.apache.org/jira/browse/SPARK-47727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47727. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45873 [https://github.com/apache/spark/pull/45873] > Make SparkConf to root level to for both SparkSession and SparkContext > -- > > Key: SPARK-47727 > URL: https://issues.apache.org/jira/browse/SPARK-47727 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47727) Make SparkConf to root level to for both SparkSession and SparkContext
[ https://issues.apache.org/jira/browse/SPARK-47727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47727: - Assignee: Hyukjin Kwon > Make SparkConf to root level to for both SparkSession and SparkContext > -- > > Key: SPARK-47727 > URL: https://issues.apache.org/jira/browse/SPARK-47727 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47709) Upgrade tink to 1.13.0
[ https://issues.apache.org/jira/browse/SPARK-47709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47709: - Assignee: Yang Jie > Upgrade tink to 1.13.0 > -- > > Key: SPARK-47709 > URL: https://issues.apache.org/jira/browse/SPARK-47709 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > > [https://github.com/tink-crypto/tink-java/releases/tag/v1.13.0] > > * AES-GCM is now about 20% faster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47709) Upgrade tink to 1.13.0
[ https://issues.apache.org/jira/browse/SPARK-47709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47709. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45843 [https://github.com/apache/spark/pull/45843] > Upgrade tink to 1.13.0 > -- > > Key: SPARK-47709 > URL: https://issues.apache.org/jira/browse/SPARK-47709 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > [https://github.com/tink-crypto/tink-java/releases/tag/v1.13.0] > > * AES-GCM is now about 20% faster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
