[jira] [Resolved] (SPARK-40547) Fix dead links in sparkr-vignettes.Rmd
[ https://issues.apache.org/jira/browse/SPARK-40547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40547. --- Fix Version/s: 3.3.1 3.4.0 Resolution: Fixed Issue resolved by pull request 37983 [https://github.com/apache/spark/pull/37983] > Fix dead links in sparkr-vignettes.Rmd > -- > > Key: SPARK-40547 > URL: https://issues.apache.org/jira/browse/SPARK-40547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Fix For: 3.3.1, 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40547) Fix dead links in sparkr-vignettes.Rmd
[ https://issues.apache.org/jira/browse/SPARK-40547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40547: - Assignee: Yuming Wang > Fix dead links in sparkr-vignettes.Rmd > -- > > Key: SPARK-40547 > URL: https://issues.apache.org/jira/browse/SPARK-40547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40547) Fix dead links in sparkr-vignettes.Rmd
[ https://issues.apache.org/jira/browse/SPARK-40547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608794#comment-17608794 ] Apache Spark commented on SPARK-40547: -- User 'wangyum' has created a pull request for this issue: https://github.com/apache/spark/pull/37983 > Fix dead links in sparkr-vignettes.Rmd > -- > > Key: SPARK-40547 > URL: https://issues.apache.org/jira/browse/SPARK-40547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40547) Fix dead links in sparkr-vignettes.Rmd
[ https://issues.apache.org/jira/browse/SPARK-40547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40547: Assignee: Apache Spark > Fix dead links in sparkr-vignettes.Rmd > -- > > Key: SPARK-40547 > URL: https://issues.apache.org/jira/browse/SPARK-40547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40547) Fix dead links in sparkr-vignettes.Rmd
[ https://issues.apache.org/jira/browse/SPARK-40547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608793#comment-17608793 ] Apache Spark commented on SPARK-40547: -- User 'wangyum' has created a pull request for this issue: https://github.com/apache/spark/pull/37983 > Fix dead links in sparkr-vignettes.Rmd > -- > > Key: SPARK-40547 > URL: https://issues.apache.org/jira/browse/SPARK-40547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40547) Fix dead links in sparkr-vignettes.Rmd
[ https://issues.apache.org/jira/browse/SPARK-40547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40547: Assignee: (was: Apache Spark) > Fix dead links in sparkr-vignettes.Rmd > -- > > Key: SPARK-40547 > URL: https://issues.apache.org/jira/browse/SPARK-40547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-40547) Fix dead links in sparkr-vignettes.Rmd
Yuming Wang created SPARK-40547: --- Summary: Fix dead links in sparkr-vignettes.Rmd Key: SPARK-40547 URL: https://issues.apache.org/jira/browse/SPARK-40547 Project: Spark Issue Type: Improvement Components: Documentation Affects Versions: 3.3.0 Reporter: Yuming Wang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38717) Handle Hive's bucket spec case preserving behaviour
[
https://issues.apache.org/jira/browse/SPARK-38717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608735#comment-17608735
]
Apache Spark commented on SPARK-38717:
--
User 'peter-toth' has created a pull request for this issue:
https://github.com/apache/spark/pull/37982
> Handle Hive's bucket spec case preserving behaviour
> ---
>
> Key: SPARK-38717
> URL: https://issues.apache.org/jira/browse/SPARK-38717
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: Peter Toth
>Priority: Major
>
> {code}
> CREATE TABLE t(
> c STRING,
> B_C STRING
> )
> PARTITIONED BY (p_c STRING)
> CLUSTERED BY (B_C) INTO 4 BUCKETS
> STORED AS PARQUET
> {code}
> then
> {code}
> SELECT * FROM t
> {code}
> fails with:
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Bucket columns
> B_C is not part of the table columns ([FieldSchema(name:c, type:string,
> comment:null), FieldSchema(name:b_c, type:string, comment:null)]
> at
> org.apache.hadoop.hive.ql.metadata.Table.setBucketCols(Table.java:552)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$.toHiveTable(HiveClientImpl.scala:1098)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getPartitionsByFilter$1(HiveClientImpl.scala:764)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:274)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.getPartitionsByFilter(HiveClientImpl.scala:763)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$listPartitionsByFilter$1(HiveExternalCatalog.scala:1287)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
> ... 110 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38717) Handle Hive's bucket spec case preserving behaviour
[
https://issues.apache.org/jira/browse/SPARK-38717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608734#comment-17608734
]
Apache Spark commented on SPARK-38717:
--
User 'peter-toth' has created a pull request for this issue:
https://github.com/apache/spark/pull/37982
> Handle Hive's bucket spec case preserving behaviour
> ---
>
> Key: SPARK-38717
> URL: https://issues.apache.org/jira/browse/SPARK-38717
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: Peter Toth
>Priority: Major
>
> {code}
> CREATE TABLE t(
> c STRING,
> B_C STRING
> )
> PARTITIONED BY (p_c STRING)
> CLUSTERED BY (B_C) INTO 4 BUCKETS
> STORED AS PARQUET
> {code}
> then
> {code}
> SELECT * FROM t
> {code}
> fails with:
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Bucket columns
> B_C is not part of the table columns ([FieldSchema(name:c, type:string,
> comment:null), FieldSchema(name:b_c, type:string, comment:null)]
> at
> org.apache.hadoop.hive.ql.metadata.Table.setBucketCols(Table.java:552)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$.toHiveTable(HiveClientImpl.scala:1098)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getPartitionsByFilter$1(HiveClientImpl.scala:764)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:274)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.getPartitionsByFilter(HiveClientImpl.scala:763)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$listPartitionsByFilter$1(HiveExternalCatalog.scala:1287)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
> ... 110 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39200) Stream is corrupted Exception while fetching the blocks from fallback storage system
[ https://issues.apache.org/jira/browse/SPARK-39200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39200. --- Fix Version/s: 3.3.1 3.2.3 3.4.0 Resolution: Fixed Issue resolved by pull request 37960 [https://github.com/apache/spark/pull/37960] > Stream is corrupted Exception while fetching the blocks from fallback storage > system > > > Key: SPARK-39200 > URL: https://issues.apache.org/jira/browse/SPARK-39200 > Project: Spark > Issue Type: Sub-task > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Rajendra Gujja >Assignee: Frank Yin >Priority: Major > Fix For: 3.3.1, 3.2.3, 3.4.0 > > > When executor decommissioning and fallback storage is enabled - the shuffle > reads are failing with `FetchFailedException: Stream is corrupted` > ref: https://issues.apache.org/jira/browse/SPARK-18105 (search for > decommission) > > This is happening when the shuffle block is bigger than `inputstream.read` > can read in one attempt. The code path is not reading the block fully > (`readFully`) and the partial read is causing the exception. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39200) Stream is corrupted Exception while fetching the blocks from fallback storage system
[ https://issues.apache.org/jira/browse/SPARK-39200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39200: - Assignee: Frank Yin (was: Frank Yin) > Stream is corrupted Exception while fetching the blocks from fallback storage > system > > > Key: SPARK-39200 > URL: https://issues.apache.org/jira/browse/SPARK-39200 > Project: Spark > Issue Type: Sub-task > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Rajendra Gujja >Assignee: Frank Yin >Priority: Major > > When executor decommissioning and fallback storage is enabled - the shuffle > reads are failing with `FetchFailedException: Stream is corrupted` > ref: https://issues.apache.org/jira/browse/SPARK-18105 (search for > decommission) > > This is happening when the shuffle block is bigger than `inputstream.read` > can read in one attempt. The code path is not reading the block fully > (`readFully`) and the partial read is causing the exception. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39200) Stream is corrupted Exception while fetching the blocks from fallback storage system
[ https://issues.apache.org/jira/browse/SPARK-39200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39200: - Assignee: Frank Yin > Stream is corrupted Exception while fetching the blocks from fallback storage > system > > > Key: SPARK-39200 > URL: https://issues.apache.org/jira/browse/SPARK-39200 > Project: Spark > Issue Type: Sub-task > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Rajendra Gujja >Assignee: Frank Yin >Priority: Major > > When executor decommissioning and fallback storage is enabled - the shuffle > reads are failing with `FetchFailedException: Stream is corrupted` > ref: https://issues.apache.org/jira/browse/SPARK-18105 (search for > decommission) > > This is happening when the shuffle block is bigger than `inputstream.read` > can read in one attempt. The code path is not reading the block fully > (`readFully`) and the partial read is causing the exception. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-40535) NPE from observe of collect_list
[
https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-40535.
--
Fix Version/s: 3.3.1
3.4.0
Resolution: Fixed
Issue resolved by pull request 37977
[https://github.com/apache/spark/pull/37977]
> NPE from observe of collect_list
>
>
> Key: SPARK-40535
> URL: https://issues.apache.org/jira/browse/SPARK-40535
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.0
>Reporter: Max Gekk
>Assignee: jiaan.geng
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> The code below reproduces the issue:
> {code:scala}
> import org.apache.spark.sql.functions._
> val df = spark.range(1,10,1,11)
> df.observe("collectedList", collect_list("id")).collect()
> {code}
> instead of
> {code}
> Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
> {code}
> it fails with the NPE:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.getBufferObject(interfaces.scala:641)
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.getBufferObject(interfaces.scala:602)
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.serializeAggregateBufferInPlace(interfaces.scala:624)
> at
> org.apache.spark.sql.execution.AggregatingAccumulator.withBufferSerialized(AggregatingAccumulator.scala:205)
> at
> org.apache.spark.sql.execution.AggregatingAccumulator.withBufferSerialized(AggregatingAccumulator.scala:33)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40535) NPE from observe of collect_list
[
https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-40535:
Assignee: jiaan.geng
> NPE from observe of collect_list
>
>
> Key: SPARK-40535
> URL: https://issues.apache.org/jira/browse/SPARK-40535
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.0
>Reporter: Max Gekk
>Assignee: jiaan.geng
>Priority: Major
>
> The code below reproduces the issue:
> {code:scala}
> import org.apache.spark.sql.functions._
> val df = spark.range(1,10,1,11)
> df.observe("collectedList", collect_list("id")).collect()
> {code}
> instead of
> {code}
> Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
> {code}
> it fails with the NPE:
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.getBufferObject(interfaces.scala:641)
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.getBufferObject(interfaces.scala:602)
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.serializeAggregateBufferInPlace(interfaces.scala:624)
> at
> org.apache.spark.sql.execution.AggregatingAccumulator.withBufferSerialized(AggregatingAccumulator.scala:205)
> at
> org.apache.spark.sql.execution.AggregatingAccumulator.withBufferSerialized(AggregatingAccumulator.scala:33)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40322) Fix all dead links
[ https://issues.apache.org/jira/browse/SPARK-40322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608617#comment-17608617 ] Apache Spark commented on SPARK-40322: -- User 'wangyum' has created a pull request for this issue: https://github.com/apache/spark/pull/37980 > Fix all dead links > -- > > Key: SPARK-40322 > URL: https://issues.apache.org/jira/browse/SPARK-40322 > Project: Spark > Issue Type: Bug > Components: Documentation >Affects Versions: 3.4.0 >Reporter: Yuming Wang >Priority: Major > > > [https://www.deadlinkchecker.com/website-dead-link-checker.asp] > > > ||Status||URL||Source link text|| > |-1 Not found: The server name or address could not be > resolved|[http://engineering.ooyala.com/blog/using-parquet-and-scrooge-spark]|[Using > Parquet and Scrooge with Spark|https://spark.apache.org/documentation.html]| > |-1 Not found: The server name or address could not be > resolved|[http://blinkdb.org/]|[BlinkDB|https://spark.apache.org/third-party-projects.html]| > |404 Not > Found|[https://github.com/AyasdiOpenSource/df]|[DF|https://spark.apache.org/third-party-projects.html]| > |-1 Timeout|[https://atp.io/]|[atp|https://spark.apache.org/powered-by.html]| > |-1 Not found: The server name or address could not be > resolved|[http://www.sehir.edu.tr/en/]|[Istanbul Sehir > University|https://spark.apache.org/powered-by.html]| > |404 Not Found|[http://nsn.com/]|[Nokia Solutions and > Networks|https://spark.apache.org/powered-by.html]| > |-1 Not found: The server name or address could not be > resolved|[http://www.nubetech.co/]|[Nube > Technologies|https://spark.apache.org/powered-by.html]| > |-1 Timeout|[http://ooyala.com/]|[Ooyala, > Inc.|https://spark.apache.org/powered-by.html]| > |-1 Not found: The server name or address could not be > resolved|[http://engineering.ooyala.com/blog/fast-spark-queries-memory-datasets]|[Spark > for Fast Queries|https://spark.apache.org/powered-by.html]| > |-1 Not found: The server name or address could not be > resolved|[http://www.sisa.samsung.com/]|[Samsung Research > America|https://spark.apache.org/powered-by.html]| > |-1 > Timeout|[https://checker.apache.org/projs/spark.html]|[https://checker.apache.org/projs/spark.html|https://spark.apache.org/release-process.html]| > |404 Not Found|[https://ampcamp.berkeley.edu/amp-camp-two-strata-2013/]|[AMP > Camp 2 [302 from > http://ampcamp.berkeley.edu/amp-camp-two-strata-2013/]|https://spark.apache.org/documentation.html]| > |404 Not Found|[https://ampcamp.berkeley.edu/agenda-2012/]|[AMP Camp 1 [302 > from > http://ampcamp.berkeley.edu/agenda-2012/]|https://spark.apache.org/documentation.html]| > |404 Not Found|[https://ampcamp.berkeley.edu/4/]|[AMP Camp 4 [302 from > http://ampcamp.berkeley.edu/4/]|https://spark.apache.org/documentation.html]| > |404 Not Found|[https://ampcamp.berkeley.edu/3/]|[AMP Camp 3 [302 from > http://ampcamp.berkeley.edu/3/]|https://spark.apache.org/documentation.html]| > |-500 Internal Server > Error-|-[https://www.packtpub.com/product/spark-cookbook/9781783987061]-|-[Spark > Cookbook [301 from > https://www.packtpub.com/big-data-and-business-intelligence/spark-cookbook]|https://spark.apache.org/documentation.html]-| > |-500 Internal Server > Error-|-[https://www.packtpub.com/product/apache-spark-graph-processing/9781784391805]-|-[Apache > Spark Graph Processing [301 from > https://www.packtpub.com/big-data-and-business-intelligence/apache-spark-graph-processing]|https://spark.apache.org/documentation.html]-| > |500 Internal Server > Error|[https://prevalentdesignevents.com/sparksummit/eu17/]|[register|https://spark.apache.org/news/]| > |500 Internal Server > Error|[https://prevalentdesignevents.com/sparksummit/ss17/?_ga=1.211902866.780052874.1433437196]|[register|https://spark.apache.org/news/]| > |500 Internal Server > Error|[https://www.prevalentdesignevents.com/sparksummit2015/europe/registration.aspx?source=header]|[register|https://spark.apache.org/news/]| > |500 Internal Server > Error|[https://www.prevalentdesignevents.com/sparksummit2015/europe/speaker/]|[Spark > Summit Europe|https://spark.apache.org/news/]| > |-1 > Timeout|[http://strataconf.com/strata2013]|[Strata|https://spark.apache.org/news/]| > |-1 Not found: The server name or address could not be > resolved|[http://blog.quantifind.com/posts/spark-unit-test/]|[Unit testing > with Spark|https://spark.apache.org/news/]| > |-1 Not found: The server name or address could not be > resolved|[http://blog.quantifind.com/posts/logging-post/]|[Configuring > Spark's logs|https://spark.apache.org/news/]| > |-1 > Timeout|[http://strata.oreilly.com/2012/08/seven-reasons-why-i-like-spark.html]|[Spark|https://spark.apache.org/news/]| > |-1 >
[jira] [Assigned] (SPARK-40322) Fix all dead links
[ https://issues.apache.org/jira/browse/SPARK-40322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40322: Assignee: Apache Spark > Fix all dead links > -- > > Key: SPARK-40322 > URL: https://issues.apache.org/jira/browse/SPARK-40322 > Project: Spark > Issue Type: Bug > Components: Documentation >Affects Versions: 3.4.0 >Reporter: Yuming Wang >Assignee: Apache Spark >Priority: Major > > > [https://www.deadlinkchecker.com/website-dead-link-checker.asp] > > > ||Status||URL||Source link text|| > |-1 Not found: The server name or address could not be > resolved|[http://engineering.ooyala.com/blog/using-parquet-and-scrooge-spark]|[Using > Parquet and Scrooge with Spark|https://spark.apache.org/documentation.html]| > |-1 Not found: The server name or address could not be > resolved|[http://blinkdb.org/]|[BlinkDB|https://spark.apache.org/third-party-projects.html]| > |404 Not > Found|[https://github.com/AyasdiOpenSource/df]|[DF|https://spark.apache.org/third-party-projects.html]| > |-1 Timeout|[https://atp.io/]|[atp|https://spark.apache.org/powered-by.html]| > |-1 Not found: The server name or address could not be > resolved|[http://www.sehir.edu.tr/en/]|[Istanbul Sehir > University|https://spark.apache.org/powered-by.html]| > |404 Not Found|[http://nsn.com/]|[Nokia Solutions and > Networks|https://spark.apache.org/powered-by.html]| > |-1 Not found: The server name or address could not be > resolved|[http://www.nubetech.co/]|[Nube > Technologies|https://spark.apache.org/powered-by.html]| > |-1 Timeout|[http://ooyala.com/]|[Ooyala, > Inc.|https://spark.apache.org/powered-by.html]| > |-1 Not found: The server name or address could not be > resolved|[http://engineering.ooyala.com/blog/fast-spark-queries-memory-datasets]|[Spark > for Fast Queries|https://spark.apache.org/powered-by.html]| > |-1 Not found: The server name or address could not be > resolved|[http://www.sisa.samsung.com/]|[Samsung Research > America|https://spark.apache.org/powered-by.html]| > |-1 > Timeout|[https://checker.apache.org/projs/spark.html]|[https://checker.apache.org/projs/spark.html|https://spark.apache.org/release-process.html]| > |404 Not Found|[https://ampcamp.berkeley.edu/amp-camp-two-strata-2013/]|[AMP > Camp 2 [302 from > http://ampcamp.berkeley.edu/amp-camp-two-strata-2013/]|https://spark.apache.org/documentation.html]| > |404 Not Found|[https://ampcamp.berkeley.edu/agenda-2012/]|[AMP Camp 1 [302 > from > http://ampcamp.berkeley.edu/agenda-2012/]|https://spark.apache.org/documentation.html]| > |404 Not Found|[https://ampcamp.berkeley.edu/4/]|[AMP Camp 4 [302 from > http://ampcamp.berkeley.edu/4/]|https://spark.apache.org/documentation.html]| > |404 Not Found|[https://ampcamp.berkeley.edu/3/]|[AMP Camp 3 [302 from > http://ampcamp.berkeley.edu/3/]|https://spark.apache.org/documentation.html]| > |-500 Internal Server > Error-|-[https://www.packtpub.com/product/spark-cookbook/9781783987061]-|-[Spark > Cookbook [301 from > https://www.packtpub.com/big-data-and-business-intelligence/spark-cookbook]|https://spark.apache.org/documentation.html]-| > |-500 Internal Server > Error-|-[https://www.packtpub.com/product/apache-spark-graph-processing/9781784391805]-|-[Apache > Spark Graph Processing [301 from > https://www.packtpub.com/big-data-and-business-intelligence/apache-spark-graph-processing]|https://spark.apache.org/documentation.html]-| > |500 Internal Server > Error|[https://prevalentdesignevents.com/sparksummit/eu17/]|[register|https://spark.apache.org/news/]| > |500 Internal Server > Error|[https://prevalentdesignevents.com/sparksummit/ss17/?_ga=1.211902866.780052874.1433437196]|[register|https://spark.apache.org/news/]| > |500 Internal Server > Error|[https://www.prevalentdesignevents.com/sparksummit2015/europe/registration.aspx?source=header]|[register|https://spark.apache.org/news/]| > |500 Internal Server > Error|[https://www.prevalentdesignevents.com/sparksummit2015/europe/speaker/]|[Spark > Summit Europe|https://spark.apache.org/news/]| > |-1 > Timeout|[http://strataconf.com/strata2013]|[Strata|https://spark.apache.org/news/]| > |-1 Not found: The server name or address could not be > resolved|[http://blog.quantifind.com/posts/spark-unit-test/]|[Unit testing > with Spark|https://spark.apache.org/news/]| > |-1 Not found: The server name or address could not be > resolved|[http://blog.quantifind.com/posts/logging-post/]|[Configuring > Spark's logs|https://spark.apache.org/news/]| > |-1 > Timeout|[http://strata.oreilly.com/2012/08/seven-reasons-why-i-like-spark.html]|[Spark|https://spark.apache.org/news/]| > |-1 > Timeout|[http://strata.oreilly.com/2012/11/shark-real-time-queries-and-analytics-for-big-data.html]|[Shark|https://spark.apache.org/news/]| > |-1 >
[jira] [Assigned] (SPARK-40474) Correct CSV schema inference and data parsing behavior on columns with mixed dates and timestamps
[ https://issues.apache.org/jira/browse/SPARK-40474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40474: --- Assignee: Xiaonan Yang > Correct CSV schema inference and data parsing behavior on columns with mixed > dates and timestamps > - > > Key: SPARK-40474 > URL: https://issues.apache.org/jira/browse/SPARK-40474 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Xiaonan Yang >Assignee: Xiaonan Yang >Priority: Major > Fix For: 3.4.0 > > > In this ticket https://issues.apache.org/jira/browse/SPARK-39469, we > introduced the support of date type in CSV schema inference. The schema > inference behavior on date time columns now is: > * For a column only containing dates, we will infer it as Date type > * For a column only containing timestamps, we will infer it as Timestamp type > * For a column containing a mixture of dates and timestamps, we will infer > it as Timestamp type > However, we found that we are too ambitious on the last scenario, to support > which we have introduced much complexity in code and caused a lot of > performance concerns. Thus, we want to simplify and correct the behavior of > the last scenario as: > * For a column containing a mixture of dates and timestamps > ** If user specifies timestamp format, it will always be inferred as > `StringType` > ** If no timestamp format specified by user, we will try inferring it as > `TimestampType` if possible, otherwise it will be inferred as `StringType` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-40474) Correct CSV schema inference and data parsing behavior on columns with mixed dates and timestamps
[ https://issues.apache.org/jira/browse/SPARK-40474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40474. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37933 [https://github.com/apache/spark/pull/37933] > Correct CSV schema inference and data parsing behavior on columns with mixed > dates and timestamps > - > > Key: SPARK-40474 > URL: https://issues.apache.org/jira/browse/SPARK-40474 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Xiaonan Yang >Priority: Major > Fix For: 3.4.0 > > > In this ticket https://issues.apache.org/jira/browse/SPARK-39469, we > introduced the support of date type in CSV schema inference. The schema > inference behavior on date time columns now is: > * For a column only containing dates, we will infer it as Date type > * For a column only containing timestamps, we will infer it as Timestamp type > * For a column containing a mixture of dates and timestamps, we will infer > it as Timestamp type > However, we found that we are too ambitious on the last scenario, to support > which we have introduced much complexity in code and caused a lot of > performance concerns. Thus, we want to simplify and correct the behavior of > the last scenario as: > * For a column containing a mixture of dates and timestamps > ** If user specifies timestamp format, it will always be inferred as > `StringType` > ** If no timestamp format specified by user, we will try inferring it as > `TimestampType` if possible, otherwise it will be inferred as `StringType` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-40532) Python version for UDF should follow the servers version
[ https://issues.apache.org/jira/browse/SPARK-40532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Grund updated SPARK-40532: - Description: Currently, we artificially pin the Python version to 3.9 in the UDF generation code, but this should actually be the correct server vs client version. In addition the version should be configured as part of the function definition proto message. was:Currently, we artificially pin the Python version to 3.9 in the UDF generation code, but this should actually be the correct server vs client version. > Python version for UDF should follow the servers version > > > Key: SPARK-40532 > URL: https://issues.apache.org/jira/browse/SPARK-40532 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Priority: Minor > > Currently, we artificially pin the Python version to 3.9 in the UDF > generation code, but this should actually be the correct server vs client > version. > > In addition the version should be configured as part of the function > definition proto message. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40545) SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution
[
https://issues.apache.org/jira/browse/SPARK-40545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608577#comment-17608577
]
Apache Spark commented on SPARK-40545:
--
User 'LuciferYang' has created a pull request for this issue:
https://github.com/apache/spark/pull/37979
> SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution
>
>
> Key: SPARK-40545
> URL: https://issues.apache.org/jira/browse/SPARK-40545
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Tests
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Priority: Minor
>
> run
> {code:java}
> mvn clean install -Phive-thriftserver -pl sql/hive-thriftserver -Dtest=none
> -DwildcardSuites=org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite
> git status {code}
> The ` sql/hive-thriftserver/spark_derby/` directory will be found
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40545) SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution
[
https://issues.apache.org/jira/browse/SPARK-40545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-40545:
Assignee: Apache Spark
> SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution
>
>
> Key: SPARK-40545
> URL: https://issues.apache.org/jira/browse/SPARK-40545
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Tests
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Assignee: Apache Spark
>Priority: Minor
>
> run
> {code:java}
> mvn clean install -Phive-thriftserver -pl sql/hive-thriftserver -Dtest=none
> -DwildcardSuites=org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite
> git status {code}
> The ` sql/hive-thriftserver/spark_derby/` directory will be found
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40545) SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution
[
https://issues.apache.org/jira/browse/SPARK-40545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-40545:
Assignee: (was: Apache Spark)
> SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution
>
>
> Key: SPARK-40545
> URL: https://issues.apache.org/jira/browse/SPARK-40545
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Tests
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Priority: Minor
>
> run
> {code:java}
> mvn clean install -Phive-thriftserver -pl sql/hive-thriftserver -Dtest=none
> -DwildcardSuites=org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite
> git status {code}
> The ` sql/hive-thriftserver/spark_derby/` directory will be found
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40543: - Assignee: Ruifeng Zheng > Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers > > > Key: SPARK-40543 > URL: https://issues.apache.org/jira/browse/SPARK-40543 > Project: Spark > Issue Type: Sub-task > Components: ps, SQL >Affects Versions: 3.4.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40543. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37975 [https://github.com/apache/spark/pull/37975] > Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers > > > Key: SPARK-40543 > URL: https://issues.apache.org/jira/browse/SPARK-40543 > Project: Spark > Issue Type: Sub-task > Components: ps, SQL >Affects Versions: 3.4.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40330) Implement `Series.searchsorted`.
[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608572#comment-17608572 ] Apache Spark commented on SPARK-40330: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/37978 > Implement `Series.searchsorted`. > > > Key: SPARK-40330 > URL: https://issues.apache.org/jira/browse/SPARK-40330 > Project: Spark > Issue Type: Sub-task > Components: Pandas API on Spark >Affects Versions: 3.4.0 >Reporter: Haejoon Lee >Assignee: Ruifeng Zheng >Priority: Major > > We should implement `Series.searchsorted` for increasing pandas API coverage. > pandas docs: > https://pandas.pydata.org/docs/reference/api/pandas.Series.searchsorted.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40330) Implement `Series.searchsorted`.
[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40330: Assignee: Apache Spark (was: Ruifeng Zheng) > Implement `Series.searchsorted`. > > > Key: SPARK-40330 > URL: https://issues.apache.org/jira/browse/SPARK-40330 > Project: Spark > Issue Type: Sub-task > Components: Pandas API on Spark >Affects Versions: 3.4.0 >Reporter: Haejoon Lee >Assignee: Apache Spark >Priority: Major > > We should implement `Series.searchsorted` for increasing pandas API coverage. > pandas docs: > https://pandas.pydata.org/docs/reference/api/pandas.Series.searchsorted.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40330) Implement `Series.searchsorted`.
[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608571#comment-17608571 ] Apache Spark commented on SPARK-40330: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/37978 > Implement `Series.searchsorted`. > > > Key: SPARK-40330 > URL: https://issues.apache.org/jira/browse/SPARK-40330 > Project: Spark > Issue Type: Sub-task > Components: Pandas API on Spark >Affects Versions: 3.4.0 >Reporter: Haejoon Lee >Assignee: Ruifeng Zheng >Priority: Major > > We should implement `Series.searchsorted` for increasing pandas API coverage. > pandas docs: > https://pandas.pydata.org/docs/reference/api/pandas.Series.searchsorted.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-40330) Implement `Series.searchsorted`.
[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40330: Assignee: Ruifeng Zheng (was: Apache Spark) > Implement `Series.searchsorted`. > > > Key: SPARK-40330 > URL: https://issues.apache.org/jira/browse/SPARK-40330 > Project: Spark > Issue Type: Sub-task > Components: Pandas API on Spark >Affects Versions: 3.4.0 >Reporter: Haejoon Lee >Assignee: Ruifeng Zheng >Priority: Major > > We should implement `Series.searchsorted` for increasing pandas API coverage. > pandas docs: > https://pandas.pydata.org/docs/reference/api/pandas.Series.searchsorted.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-40545) SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution
Yang Jie created SPARK-40545:
Summary: SparkSQLEnvSuite failed to clean the `spark_derby`
directory after execution
Key: SPARK-40545
URL: https://issues.apache.org/jira/browse/SPARK-40545
Project: Spark
Issue Type: Improvement
Components: SQL, Tests
Affects Versions: 3.4.0
Reporter: Yang Jie
run
{code:java}
mvn clean install -Phive-thriftserver -pl sql/hive-thriftserver -Dtest=none
-DwildcardSuites=org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite
git status {code}
The ` sql/hive-thriftserver/spark_derby/` directory will be found
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
