[jira] [Updated] (SPARK-28510) Implement Spark's own GetFunctionsOperation
[
https://issues.apache.org/jira/browse/SPARK-28510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-28510:
Description: We should implement Spark's own {{GetFunctionsOperation}}
because Spark SQL and Hive's UDF have many differences.
> Implement Spark's own GetFunctionsOperation
> ---
>
> Key: SPARK-28510
> URL: https://issues.apache.org/jira/browse/SPARK-28510
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> We should implement Spark's own {{GetFunctionsOperation}} because Spark SQL
> and Hive's UDF have many differences.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28511) Get REV from RELEASE_VERSION instead of VERSION
Dongjoon Hyun created SPARK-28511:
-
Summary: Get REV from RELEASE_VERSION instead of VERSION
Key: SPARK-28511
URL: https://issues.apache.org/jira/browse/SPARK-28511
Project: Spark
Issue Type: Bug
Components: Project Infra
Affects Versions: 3.0.0
Reporter: Dongjoon Hyun
Unlike the other versions, `x.x.0-SNAPSHOT` causes `x.x.-1`. Although this will
not happen in the tags (there is no `SNAPSHOT` postfix), we had better fix this.
{code}
$ dev/create-release/do-release-docker.sh -d /tmp/spark-3.0.0 -n
Output directory already exists. Overwrite and continue? [y/n] y
Branch [branch-2.4]: master
Current branch version is 3.0.0-SNAPSHOT.
Release [3.0.-1]:
{code}
The following is the expected behavior.
{code}
$ dev/create-release/do-release-docker.sh -d /tmp/spark-3.0.0 -n
Branch [branch-2.4]: master
Current branch version is 3.0.0-SNAPSHOT.
Release [3.0.0]:
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures
Gengliang Wang created SPARK-28512:
--
Summary: New optional mode: throw runtime exceptions on casting
failures
Key: SPARK-28512
URL: https://issues.apache.org/jira/browse/SPARK-28512
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Gengliang Wang
In popular DBMS like MySQL/PostgreSQL/Oracle, runtime exceptions are thrown on
casting, e.g. cast('abc' as int)
While in Spark, the result is converted as null silently. It is by design since
we don't want a long-running job aborted by some casting failure. But there are
scenarios that users want to make sure all the data conversion are correct,
like the way they use MySQL/PostgreSQL/Oracle.
If the changes touch too much code, we can limit the new optional mode to table
insertion first. By default the new behavior is disabled.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26995) Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when using snappy
[
https://issues.apache.org/jira/browse/SPARK-26995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-26995:
--
Fix Version/s: 2.4.4
> Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when
> using snappy
> -
>
> Key: SPARK-26995
> URL: https://issues.apache.org/jira/browse/SPARK-26995
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Luca Canali
>Assignee: Luca Canali
>Priority: Minor
> Fix For: 2.4.4, 3.0.0
>
>
> Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when
> using snappy.
> The issue can be reproduced for example as follows:
> `Seq(1,2).toDF("id").write.format("parquet").save("DELETEME1")`
> The key part of the error stack is as follows `Caused by:
> java.lang.UnsatisfiedLinkError:
> /tmp/snappy-1.1.7-2b4872f1-7c41-4b84-bda1-dbcb8dd0ce4c-libsnappyjava.so:
> Error loading shared library ld-linux-x86-64.so.2: Noded by
> /tmp/snappy-1.1.7-2b4872f1-7c41-4b84-bda1-dbcb8dd0ce4c-libsnappyjava.so)`
> The source of the error appears to be due to the fact that libsnappyjava.so
> needs ld-linux-x86-64.so.2 and looks for it in /lib, while in Alpine Linux
> 3.9.0 with libc6-compat version 1.1.20-r3 ld-linux-x86-64.so.2 is located in
> /lib64.
> Note: this issue is not present with Alpine Linux 3.8 and libc6-compat
> version 1.1.19-r10
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26995) Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when using snappy
[
https://issues.apache.org/jira/browse/SPARK-26995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892559#comment-16892559
]
Dongjoon Hyun commented on SPARK-26995:
---
This is backported to `branch-2.4` via
https://github.com/apache/spark/pull/25255 .
> Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when
> using snappy
> -
>
> Key: SPARK-26995
> URL: https://issues.apache.org/jira/browse/SPARK-26995
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Luca Canali
>Assignee: Luca Canali
>Priority: Minor
> Fix For: 2.4.4, 3.0.0
>
>
> Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when
> using snappy.
> The issue can be reproduced for example as follows:
> `Seq(1,2).toDF("id").write.format("parquet").save("DELETEME1")`
> The key part of the error stack is as follows `Caused by:
> java.lang.UnsatisfiedLinkError:
> /tmp/snappy-1.1.7-2b4872f1-7c41-4b84-bda1-dbcb8dd0ce4c-libsnappyjava.so:
> Error loading shared library ld-linux-x86-64.so.2: Noded by
> /tmp/snappy-1.1.7-2b4872f1-7c41-4b84-bda1-dbcb8dd0ce4c-libsnappyjava.so)`
> The source of the error appears to be due to the fact that libsnappyjava.so
> needs ld-linux-x86-64.so.2 and looks for it in /lib, while in Alpine Linux
> 3.9.0 with libc6-compat version 1.1.20-r3 ld-linux-x86-64.so.2 is located in
> /lib64.
> Note: this issue is not present with Alpine Linux 3.8 and libc6-compat
> version 1.1.19-r10
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28156) Join plan sometimes does not use cached query
[
https://issues.apache.org/jira/browse/SPARK-28156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-28156:
Fix Version/s: 2.4.4
> Join plan sometimes does not use cached query
> -
>
> Key: SPARK-28156
> URL: https://issues.apache.org/jira/browse/SPARK-28156
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.3, 3.0.0, 2.4.3
>Reporter: Bruce Robbins
>Assignee: Liang-Chi Hsieh
>Priority: Major
> Fix For: 2.4.4, 3.0.0
>
>
> I came across a case where a cached query is referenced on both sides of a
> join, but the InMemoryRelation is inserted on only one side. This case occurs
> only when the cached query uses a (Hive-style) view.
> Consider this example:
> {noformat}
> // create the data
> val df1 = Seq.tabulate(10) { x => (x, x + 1, x + 2, x + 3) }.toDF("a", "b",
> "c", "d")
> df1.write.mode("overwrite").format("orc").saveAsTable("table1")
> sql("drop view if exists table1_vw")
> sql("create view table1_vw as select * from table1")
> // create the cached query
> val cacheddataDf = sql("""
> select a, b, c, d
> from table1_vw
> """)
> import org.apache.spark.storage.StorageLevel.DISK_ONLY
> cacheddataDf.createOrReplaceTempView("cacheddata")
> cacheddataDf.persist(DISK_ONLY)
> // main query
> val queryDf = sql(s"""
> select leftside.a, leftside.b
> from cacheddata leftside
> join cacheddata rightside
> on leftside.a = rightside.a
> """)
> queryDf.explain(true)
> {noformat}
> Note that the optimized plan does not use an InMemoryRelation for the right
> side, but instead just uses a Relation:
> {noformat}
> Project [a#45, b#46]
> +- Join Inner, (a#45 = a#37)
>:- Project [a#45, b#46]
>: +- Filter isnotnull(a#45)
>: +- InMemoryRelation [a#45, b#46, c#47, d#48], StorageLevel(disk, 1
> replicas)
>: +- *(1) FileScan orc default.table1[a#37,b#38,c#39,d#40]
> Batched: true, DataFilters: [], Format: ORC, Location:
> InMemoryFileIndex[file:/Users/brobbins/github/spark_upstream/spark-warehouse/table1],
> PartitionFilters: [], PushedFilters: [], ReadSchema:
> struct
>+- Project [a#37]
> +- Filter isnotnull(a#37)
> +- Relation[a#37,b#38,c#39,d#40] orc
> {noformat}
> The fragment does not match the cached query because AliasViewChild adds an
> extra projection under the View on the right side (see #2 below).
> AliasViewChild adds the extra projection because the exprIds in the View's
> output appears to have been renamed by Analyzer$ResolveReferences (#1 below).
> I have not yet looked at why.
> {noformat}
> -
> -
> -
>+- SubqueryAlias `rightside`
> +- SubqueryAlias `cacheddata`
> +- Project [a#73, b#74, c#75, d#76]
> +- SubqueryAlias `default`.`table1_vw`
> (#1) ->+- View (`default`.`table1_vw`, [a#73,b#74,c#75,d#76])
> (#2) -> +- Project [cast(a#45 as int) AS a#73, cast(b#46 as int) AS
> b#74, cast(c#47 as int) AS c#75, cast(d#48 as int) AS d#76]
> +- Project [cast(a#37 as int) AS a#45, cast(b#38 as int)
> AS b#46, cast(c#39 as int) AS c#47, cast(d#40 as int) AS d#48]
> +- Project [a#37, b#38, c#39, d#40]
>+- SubqueryAlias `default`.`table1`
> +- Relation[a#37,b#38,c#39,d#40] orc
> {noformat}
> In a larger query (where cachedata may be referred on either side only
> indirectly), this phenomenon can create certain oddities, as the fragment is
> not replaced with InMemoryRelation, and the fragment is present when the plan
> is optimized as a whole.
> In Spark 2.1.3, Spark uses InMemoryRelation on both sides.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27688) Beeline should show database in the prompt
[
https://issues.apache.org/jira/browse/SPARK-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892580#comment-16892580
]
Yuming Wang commented on SPARK-27688:
-
{code:sh}
build/sbt clean package -Phive -Phive-thriftserver -Phadoop-3.2
export SPARK_PREPEND_CLASSES=true
sbin/stop-thriftserver.sh
{code}
{noformat}
[root@spark-3267648 apache-spark]# bin/beeline -u
jdbc:hive2://localhost:1/default --showDbInPrompt
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes
ahead of assembly.
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Connecting to jdbc:hive2://localhost:1/default
Connected to: Spark SQL (version 3.0.0-SNAPSHOT)
Driver: Hive JDBC (version 2.3.5)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.5 by Apache Hive
0: jdbc:hive2://localhost:1/default (default)> use db2;
+-+
| Result |
+-+
+-+
No rows selected (0.168 seconds)
0: jdbc:hive2://localhost:1/default (db2)>
0: jdbc:hive2://localhost:1/default (db2)> use db1;
+-+
| Result |
+-+
+-+
No rows selected (0.091 seconds)
0: jdbc:hive2://localhost:1/default (db1)>
0: jdbc:hive2://localhost:1/default (db1)>
{noformat}
> Beeline should show database in the prompt
> --
>
> Key: SPARK-27688
> URL: https://issues.apache.org/jira/browse/SPARK-27688
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 2.4.3
>Reporter: Sandeep Katta
>Priority: Minor
>
> Since [HIVE-14123|https://issues.apache.org/jira/browse/HIVE-14123] supports
> display of database in beeline. Spark should also support this
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27688) Beeline should show database in the prompt
[ https://issues.apache.org/jira/browse/SPARK-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27688: Affects Version/s: (was: 2.4.3) 3.0.0 > Beeline should show database in the prompt > -- > > Key: SPARK-27688 > URL: https://issues.apache.org/jira/browse/SPARK-27688 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Sandeep Katta >Priority: Minor > > Since [HIVE-14123|https://issues.apache.org/jira/browse/HIVE-14123] supports > display of database in beeline. Spark should also support this -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-27688) Beeline should show database in the prompt
[ https://issues.apache.org/jira/browse/SPARK-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-27688. - Resolution: Fixed We supported it after upgrading the built-in Hive to 2.3.5. > Beeline should show database in the prompt > -- > > Key: SPARK-27688 > URL: https://issues.apache.org/jira/browse/SPARK-27688 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Sandeep Katta >Priority: Minor > > Since [HIVE-14123|https://issues.apache.org/jira/browse/HIVE-14123] supports > display of database in beeline. Spark should also support this -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27688) Beeline should show database in the prompt
[
https://issues.apache.org/jira/browse/SPARK-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892580#comment-16892580
]
Yuming Wang edited comment on SPARK-27688 at 7/25/19 9:24 AM:
--
{code:sh}
build/sbt clean package -Phive -Phive-thriftserver -Phadoop-3.2
export SPARK_PREPEND_CLASSES=true
sbin/start-thriftserver.sh
{code}
{noformat}
[root@spark-3267648 apache-spark]# bin/beeline -u
jdbc:hive2://localhost:1/default --showDbInPrompt
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes
ahead of assembly.
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Connecting to jdbc:hive2://localhost:1/default
Connected to: Spark SQL (version 3.0.0-SNAPSHOT)
Driver: Hive JDBC (version 2.3.5)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.5 by Apache Hive
0: jdbc:hive2://localhost:1/default (default)> use db2;
+-+
| Result |
+-+
+-+
No rows selected (0.168 seconds)
0: jdbc:hive2://localhost:1/default (db2)>
0: jdbc:hive2://localhost:1/default (db2)> use db1;
+-+
| Result |
+-+
+-+
No rows selected (0.091 seconds)
0: jdbc:hive2://localhost:1/default (db1)>
0: jdbc:hive2://localhost:1/default (db1)>
{noformat}
was (Author: q79969786):
{code:sh}
build/sbt clean package -Phive -Phive-thriftserver -Phadoop-3.2
export SPARK_PREPEND_CLASSES=true
sbin/stop-thriftserver.sh
{code}
{noformat}
[root@spark-3267648 apache-spark]# bin/beeline -u
jdbc:hive2://localhost:1/default --showDbInPrompt
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes
ahead of assembly.
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Connecting to jdbc:hive2://localhost:1/default
Connected to: Spark SQL (version 3.0.0-SNAPSHOT)
Driver: Hive JDBC (version 2.3.5)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.5 by Apache Hive
0: jdbc:hive2://localhost:1/default (default)> use db2;
+-+
| Result |
+-+
+-+
No rows selected (0.168 seconds)
0: jdbc:hive2://localhost:1/default (db2)>
0: jdbc:hive2://localhost:1/default (db2)> use db1;
+-+
| Result |
+-+
+-+
No rows selected (0.091 seconds)
0: jdbc:hive2://localhost:1/default (db1)>
0: jdbc:hive2://localhost:1/default (db1)>
{noformat}
> Beeline should show database in the prompt
> --
>
> Key: SPARK-27688
> URL: https://issues.apache.org/jira/browse/SPARK-27688
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Sandeep Katta
>Priority: Minor
>
> Since [HIVE-14123|https://issues.apache.org/jira/browse/HIVE-14123] supports
> display of database in beeline. Spark should also support this
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22063) Upgrade lintr to latest commit sha1 ID
[
https://issues.apache.org/jira/browse/SPARK-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892595#comment-16892595
]
Manu Zhang commented on SPARK-22063:
Is there any update in this thread ? Which lint-r version is used in build now ?
I also find upgrading lint-r would also upgrade testthat to latest version
while [SparkR requires testthat 1.0.2
|https://github.com/apache/spark/blob/master/docs/building-spark.md#running-r-tests]
> Upgrade lintr to latest commit sha1 ID
> --
>
> Key: SPARK-22063
> URL: https://issues.apache.org/jira/browse/SPARK-22063
> Project: Spark
> Issue Type: Improvement
> Components: SparkR
>Affects Versions: 2.3.0
>Reporter: Hyukjin Kwon
>Priority: Minor
>
> Currently, we set lintr to {{jimhester/lintr@a769c0b}} (see [this
> pr|https://github.com/apache/spark/commit/7d1175011c976756efcd4e4e4f70a8fd6f287026])
> and SPARK-14074.
> Today, I tried to upgrade the latest,
> https://github.com/jimhester/lintr/commit/5431140ffea65071f1327625d4a8de9688fa7e72
> This fixes many bugs and now finds many instances that I have observed and
> thought should be caught time to time:
> {code}
> inst/worker/worker.R:71:10: style: Remove spaces before the left parenthesis
> in a function call.
> return (output)
> ^
> R/column.R:241:1: style: Lines should not be more than 100 characters.
> #'
> \href{https://spark.apache.org/docs/latest/sparkr.html#data-type-mapping-between-r-and-spark}{
> ^~~~
> R/context.R:332:1: style: Variable and function names should not be longer
> than 30 characters.
> spark.getSparkFilesRootDirectory <- function() {
> ^~~~
> R/DataFrame.R:1912:1: style: Lines should not be more than 100 characters.
> #' @param j,select expression for the single Column or a list of columns to
> select from the SparkDataFrame.
> ^~~
> R/DataFrame.R:1918:1: style: Lines should not be more than 100 characters.
> #' @return A new SparkDataFrame containing only the rows that meet the
> condition with selected columns.
> ^~~
> R/DataFrame.R:2597:22: style: Remove spaces before the left parenthesis in a
> function call.
> return (joinRes)
> ^
> R/DataFrame.R:2652:1: style: Variable and function names should not be longer
> than 30 characters.
> generateAliasesForIntersectedCols <- function (x, intersectedColNames,
> suffix) {
> ^
> R/DataFrame.R:2652:47: style: Remove spaces before the left parenthesis in a
> function call.
> generateAliasesForIntersectedCols <- function (x, intersectedColNames,
> suffix) {
> ^
> R/DataFrame.R:2660:14: style: Remove spaces before the left parenthesis in a
> function call.
> stop ("The following column name: ", newJoin, " occurs more than once
> in the 'DataFrame'.",
> ^
> R/DataFrame.R:3047:1: style: Lines should not be more than 100 characters.
> #' @note The statistics provided by \code{summary} were change in 2.3.0 use
> \link{describe} for previous defaults.
> ^~
> R/DataFrame.R:3754:1: style: Lines should not be more than 100 characters.
> #' If grouping expression is missing \code{cube} creates a single global
> aggregate and is equivalent to
> ^~~
> R/DataFrame.R:3789:1: style: Lines should not be more than 100 characters.
> #' If grouping expression is missing \code{rollup} creates a single global
> aggregate and is equivalent to
> ^
> R/deserialize.R:46:10: style: Remove spaces before the left parenthesis in a
> function call.
> switch (type,
> ^
> R/functions.R:41:1: style: Lines should not be more than 100 characters.
> #' @param x Column to compute on. In \code{window}, it must be a time Column
> of \code{TimestampType}.
> ^
> R/functions.R:93:1: style: Lines should not be more than 100 characters.
> #' @param x Column to compute on. In \code{shiftLeft}, \code{shiftRight} and
> \code{shiftRightUnsigned},
> ^~~
> R/functio
[jira] [Comment Edited] (SPARK-22063) Upgrade lintr to latest commit sha1 ID
[
https://issues.apache.org/jira/browse/SPARK-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892595#comment-16892595
]
Manu Zhang edited comment on SPARK-22063 at 7/25/19 9:41 AM:
-
Is there any update in this thread ? Which lint-r version is used in build now ?
I also find upgrading lint-r would upgrade testthat to latest version while
[SparkR requires testthat 1.0.2
|https://github.com/apache/spark/blob/master/docs/building-spark.md#running-r-tests]
was (Author: mauzhang):
Is there any update in this thread ? Which lint-r version is used in build now ?
I also find upgrading lint-r would also upgrade testthat to latest version
while [SparkR requires testthat 1.0.2
|https://github.com/apache/spark/blob/master/docs/building-spark.md#running-r-tests]
> Upgrade lintr to latest commit sha1 ID
> --
>
> Key: SPARK-22063
> URL: https://issues.apache.org/jira/browse/SPARK-22063
> Project: Spark
> Issue Type: Improvement
> Components: SparkR
>Affects Versions: 2.3.0
>Reporter: Hyukjin Kwon
>Priority: Minor
>
> Currently, we set lintr to {{jimhester/lintr@a769c0b}} (see [this
> pr|https://github.com/apache/spark/commit/7d1175011c976756efcd4e4e4f70a8fd6f287026])
> and SPARK-14074.
> Today, I tried to upgrade the latest,
> https://github.com/jimhester/lintr/commit/5431140ffea65071f1327625d4a8de9688fa7e72
> This fixes many bugs and now finds many instances that I have observed and
> thought should be caught time to time:
> {code}
> inst/worker/worker.R:71:10: style: Remove spaces before the left parenthesis
> in a function call.
> return (output)
> ^
> R/column.R:241:1: style: Lines should not be more than 100 characters.
> #'
> \href{https://spark.apache.org/docs/latest/sparkr.html#data-type-mapping-between-r-and-spark}{
> ^~~~
> R/context.R:332:1: style: Variable and function names should not be longer
> than 30 characters.
> spark.getSparkFilesRootDirectory <- function() {
> ^~~~
> R/DataFrame.R:1912:1: style: Lines should not be more than 100 characters.
> #' @param j,select expression for the single Column or a list of columns to
> select from the SparkDataFrame.
> ^~~
> R/DataFrame.R:1918:1: style: Lines should not be more than 100 characters.
> #' @return A new SparkDataFrame containing only the rows that meet the
> condition with selected columns.
> ^~~
> R/DataFrame.R:2597:22: style: Remove spaces before the left parenthesis in a
> function call.
> return (joinRes)
> ^
> R/DataFrame.R:2652:1: style: Variable and function names should not be longer
> than 30 characters.
> generateAliasesForIntersectedCols <- function (x, intersectedColNames,
> suffix) {
> ^
> R/DataFrame.R:2652:47: style: Remove spaces before the left parenthesis in a
> function call.
> generateAliasesForIntersectedCols <- function (x, intersectedColNames,
> suffix) {
> ^
> R/DataFrame.R:2660:14: style: Remove spaces before the left parenthesis in a
> function call.
> stop ("The following column name: ", newJoin, " occurs more than once
> in the 'DataFrame'.",
> ^
> R/DataFrame.R:3047:1: style: Lines should not be more than 100 characters.
> #' @note The statistics provided by \code{summary} were change in 2.3.0 use
> \link{describe} for previous defaults.
> ^~
> R/DataFrame.R:3754:1: style: Lines should not be more than 100 characters.
> #' If grouping expression is missing \code{cube} creates a single global
> aggregate and is equivalent to
> ^~~
> R/DataFrame.R:3789:1: style: Lines should not be more than 100 characters.
> #' If grouping expression is missing \code{rollup} creates a single global
> aggregate and is equivalent to
> ^
> R/deserialize.R:46:10: style: Remove spaces before the left parenthesis in a
> function call.
> switch (type,
> ^
> R/functions.R:41:1: style: Lines should not be more than 100 characters.
> #' @param x Column to compute on. In \code{window}, it must be a time Column
> of \code{TimestampType}.
> ^~~
[jira] [Created] (SPARK-28513) Compute distinct label sets instead of subsets
Martin Junghanns created SPARK-28513:
Summary: Compute distinct label sets instead of subsets
Key: SPARK-28513
URL: https://issues.apache.org/jira/browse/SPARK-28513
Project: Spark
Issue Type: Improvement
Components: Graph
Affects Versions: 3.0.0
Reporter: Martin Junghanns
{code:scala}
CypherSession::createDataFrame(nodes: DataFrame, rels: DataFrame)
{code}
currently computes NodeFrames by filtering label columns, computing all
possible subsets and creating one NodeFrame for each subset. This results in
2^n sets / NodeFrames.
Instead, we should compute the distinct label sets that actually occur in the
nodes DataFrame.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28514) Remove the redundant transformImpl method in RF & GBT
zhengruifeng created SPARK-28514: Summary: Remove the redundant transformImpl method in RF & GBT Key: SPARK-28514 URL: https://issues.apache.org/jira/browse/SPARK-28514 Project: Spark Issue Type: Improvement Components: ML Affects Versions: 3.0.0 Reporter: zhengruifeng 1, In GBTClassifier & RandomForestClassifier, the real transform methods inherit from ProbabilisticClassificationModel which can deal with multi output columns. The transformImpl method, which deals with only one column - predictionCol, completely does nothing. This is quite confusing. 2, In GBTRegressor & RandomForestRegressor, the transformImpl do exactly what the superclass PredictionModel does (except model broadcasting), so can be removed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28515) to_timestamp returns null for summer time switch dates
Andreas Költringer created SPARK-28515:
--
Summary: to_timestamp returns null for summer time switch dates
Key: SPARK-28515
URL: https://issues.apache.org/jira/browse/SPARK-28515
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.3
Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
Reporter: Andreas Költringer
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{++---+
}}
{{| date_str| timestamp|}}
{{++---+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{++---+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{ SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm"); Date
parsedDate = dateFormat.parse("201503290201"); Timestamp timestamp = new
java.sql.Timestamp(parsedDate.getTime());}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28515) to_timestamp returns null for summer time switch dates
[
https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andreas Költringer updated SPARK-28515:
---
Description:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+
}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm"); }}
{{Date parsedDate = dateFormat.parse("201503290201"); }}
{{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
was:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{++---+
}}
{{| date_str| timestamp|}}
{{++---+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{++---+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{ SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm"); Date
parsedDate = dateFormat.parse("201503290201"); Timestamp timestamp = new
java.sql.Timestamp(parsedDate.getTime());}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
> to_timestamp returns null for summer time switch dates
> --
>
> Key: SPARK-28515
> URL: https://issues.apache.org/jira/browse/SPARK-28515
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
>Reporter: Andreas Költringer
>Priority: Major
>
> I am not sure if this is a bug - but it was a very unexpected behavior, so
> I'd like some clarification.
> When parsing datetime-strings, when the date-time in question falls into the
> range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at
> 2am the clock was forwarded to 3am), the {{to_timestamp}} method returns
> {{NULL}}.
> Minimal Example (using Python):
> {{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
> ['date_str'])}}
> {{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
> 'MMddhhmm')).show()}}
> {{+-+--+
> }}
> {{| date_str| timestamp|}}
> {{+-+--+}}
> {{|201503290159|2015-03-29 01:59:00|}}
> {{|201503290200| null|}}
> {{+-+--+}}
> A solution (or workaround) is to set the time zone for Spark to UTC:
> {{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
> (see e.g. [https://stackoverflow.com/q/52594762)]
>
> Plain Java does not do this, e.g. this works as expected:
> {{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm"); }}
> {{Date parsedDate = dateFormat.parse("201503290201"); }}
> {{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
>
> So, is this really the intended behaviour? Is there documentation about this?
> THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (SPARK-28515) to_timestamp returns null for summer time switch dates
[
https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andreas Költringer updated SPARK-28515:
---
Description:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+
}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm"); }}
{{Date parsedDate = dateFormat.parse("201503290201"); }}
{{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime()); }}
So, is this really the intended behaviour? Is there documentation about this?
THX.
was:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+
}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm"); }}
{{Date parsedDate = dateFormat.parse("201503290201"); }}
{{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
> to_timestamp returns null for summer time switch dates
> --
>
> Key: SPARK-28515
> URL: https://issues.apache.org/jira/browse/SPARK-28515
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
>Reporter: Andreas Költringer
>Priority: Major
>
> I am not sure if this is a bug - but it was a very unexpected behavior, so
> I'd like some clarification.
> When parsing datetime-strings, when the date-time in question falls into the
> range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at
> 2am the clock was forwarded to 3am), the {{to_timestamp}} method returns
> {{NULL}}.
> Minimal Example (using Python):
> {{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
> ['date_str'])}}
> {{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
> 'MMddhhmm')).show()}}
> {{+-+--+
> }}
> {{| date_str| timestamp|}}
> {{+-+--+}}
> {{|201503290159|2015-03-29 01:59:00|}}
> {{|201503290200| null|}}
> {{+-+--+}}
> A solution (or workaround) is to set the time zone for Spark to UTC:
> {{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
> (see e.g. [https://stackoverflow.com/q/52594762)]
>
> Plain Java does not do this, e.g. this works as expected:
> {{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm"); }}
> {{Date parsedDate = dateFormat.parse("201503290201"); }}
> {{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime()); }}
>
> So, is this really the intended behaviour? Is there documentation about this?
> THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (SPARK-28515) to_timestamp returns null for summer time switch dates
[
https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andreas Költringer updated SPARK-28515:
---
Description:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+
}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{
SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");
Date parsedDate = dateFormat.parse("201503290201");
Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());
}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
was:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+
}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm"); }}
{{Date parsedDate = dateFormat.parse("201503290201"); }}
{{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime()); }}
So, is this really the intended behaviour? Is there documentation about this?
THX.
> to_timestamp returns null for summer time switch dates
> --
>
> Key: SPARK-28515
> URL: https://issues.apache.org/jira/browse/SPARK-28515
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
>Reporter: Andreas Költringer
>Priority: Major
>
> I am not sure if this is a bug - but it was a very unexpected behavior, so
> I'd like some clarification.
> When parsing datetime-strings, when the date-time in question falls into the
> range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at
> 2am the clock was forwarded to 3am), the {{to_timestamp}} method returns
> {{NULL}}.
> Minimal Example (using Python):
> {{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
> ['date_str'])}}
> {{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
> 'MMddhhmm')).show()}}
> {{+-+--+
> }}
> {{| date_str| timestamp|}}
> {{+-+--+}}
> {{|201503290159|2015-03-29 01:59:00|}}
> {{|201503290200| null|}}
> {{+-+--+}}
> A solution (or workaround) is to set the time zone for Spark to UTC:
> {{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
> (see e.g. [https://stackoverflow.com/q/52594762)]
>
> Plain Java does not do this, e.g. this works as expected:
> {{
> SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");
> Date parsedDate = dateFormat.parse("201503290201");
> Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());
> }}
>
> So, is this really the intended behaviour? Is there documentation about this?
> THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
[jira] [Updated] (SPARK-28515) to_timestamp returns null for summer time switch dates
[
https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andreas Költringer updated SPARK-28515:
---
Description:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+
}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");}}
{{Date parsedDate = dateFormat.parse("201503290201");}}
{{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
was:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+
}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{
SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");
Date parsedDate = dateFormat.parse("201503290201");
Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());
}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
> to_timestamp returns null for summer time switch dates
> --
>
> Key: SPARK-28515
> URL: https://issues.apache.org/jira/browse/SPARK-28515
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
>Reporter: Andreas Költringer
>Priority: Major
>
> I am not sure if this is a bug - but it was a very unexpected behavior, so
> I'd like some clarification.
> When parsing datetime-strings, when the date-time in question falls into the
> range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at
> 2am the clock was forwarded to 3am), the {{to_timestamp}} method returns
> {{NULL}}.
> Minimal Example (using Python):
> {{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
> ['date_str'])}}
> {{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
> 'MMddhhmm')).show()}}
> {{+-+--+
> }}
> {{| date_str| timestamp|}}
> {{+-+--+}}
> {{|201503290159|2015-03-29 01:59:00|}}
> {{|201503290200| null|}}
> {{+-+--+}}
> A solution (or workaround) is to set the time zone for Spark to UTC:
> {{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
> (see e.g. [https://stackoverflow.com/q/52594762)]
>
> Plain Java does not do this, e.g. this works as expected:
> {{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");}}
> {{Date parsedDate = dateFormat.parse("201503290201");}}
> {{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
>
> So, is this really the intended behaviour? Is there documentation about this?
> THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (SPARK-28515) to_timestamp returns null for summer time switch dates
[
https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andreas Költringer updated SPARK-28515:
---
Description:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");}}
{{Date parsedDate = dateFormat.parse("201503290201");}}
{{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
was:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+
}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");}}
{{Date parsedDate = dateFormat.parse("201503290201");}}
{{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
> to_timestamp returns null for summer time switch dates
> --
>
> Key: SPARK-28515
> URL: https://issues.apache.org/jira/browse/SPARK-28515
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
>Reporter: Andreas Költringer
>Priority: Major
>
> I am not sure if this is a bug - but it was a very unexpected behavior, so
> I'd like some clarification.
> When parsing datetime-strings, when the date-time in question falls into the
> range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at
> 2am the clock was forwarded to 3am), the {{to_timestamp}} method returns
> {{NULL}}.
> Minimal Example (using Python):
> {{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
> ['date_str'])}}
> {{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
> 'MMddhhmm')).show()}}
> {{+-+--+}}
> {{| date_str| timestamp|}}
> {{+-+--+}}
> {{|201503290159|2015-03-29 01:59:00|}}
> {{|201503290200| null|}}
> {{+-+--+}}
> A solution (or workaround) is to set the time zone for Spark to UTC:
> {{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
> (see e.g. [https://stackoverflow.com/q/52594762)]
>
> Plain Java does not do this, e.g. this works as expected:
> {{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");}}
> {{Date parsedDate = dateFormat.parse("201503290201");}}
> {{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
>
> So, is this really the intended behaviour? Is there documentation about this?
> THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional command
[jira] [Resolved] (SPARK-28497) Disallow upcasting complex data types to string type
[ https://issues.apache.org/jira/browse/SPARK-28497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28497. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25242 [https://github.com/apache/spark/pull/25242] > Disallow upcasting complex data types to string type > > > Key: SPARK-28497 > URL: https://issues.apache.org/jira/browse/SPARK-28497 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Minor > Fix For: 3.0.0 > > > In the current implementation. complex types like Array/Map/StructType are > allowed to upcast as StringType. > This is not safe casting. We should disallow it. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28497) Disallow upcasting complex data types to string type
[ https://issues.apache.org/jira/browse/SPARK-28497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28497: Assignee: Gengliang Wang > Disallow upcasting complex data types to string type > > > Key: SPARK-28497 > URL: https://issues.apache.org/jira/browse/SPARK-28497 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Minor > > In the current implementation. complex types like Array/Map/StructType are > allowed to upcast as StringType. > This is not safe casting. We should disallow it. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures
[
https://issues.apache.org/jira/browse/SPARK-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892683#comment-16892683
]
Takeshi Yamamuro commented on SPARK-28512:
--
Is this related to https://issues.apache.org/jira/browse/SPARK-28470?
cc: [~mgaido]
> New optional mode: throw runtime exceptions on casting failures
> ---
>
> Key: SPARK-28512
> URL: https://issues.apache.org/jira/browse/SPARK-28512
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Priority: Major
>
> In popular DBMS like MySQL/PostgreSQL/Oracle, runtime exceptions are thrown
> on casting, e.g. cast('abc' as int)
> While in Spark, the result is converted as null silently. It is by design
> since we don't want a long-running job aborted by some casting failure. But
> there are scenarios that users want to make sure all the data conversion are
> correct, like the way they use MySQL/PostgreSQL/Oracle.
> If the changes touch too much code, we can limit the new optional mode to
> table insertion first. By default the new behavior is disabled.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures
[
https://issues.apache.org/jira/browse/SPARK-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892692#comment-16892692
]
Marco Gaido commented on SPARK-28512:
-
Thanks for pinging me [~maropu]. It is not the same issue, I think, because in
SPARK-28470 we are dealing only with cases when there is an overflow. Here
there is no overflow. Simply the value is not valid for the casted type.
> New optional mode: throw runtime exceptions on casting failures
> ---
>
> Key: SPARK-28512
> URL: https://issues.apache.org/jira/browse/SPARK-28512
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Priority: Major
>
> In popular DBMS like MySQL/PostgreSQL/Oracle, runtime exceptions are thrown
> on casting, e.g. cast('abc' as int)
> While in Spark, the result is converted as null silently. It is by design
> since we don't want a long-running job aborted by some casting failure. But
> there are scenarios that users want to make sure all the data conversion are
> correct, like the way they use MySQL/PostgreSQL/Oracle.
> If the changes touch too much code, we can limit the new optional mode to
> table insertion first. By default the new behavior is disabled.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28421) SparseVector.apply performance optimization
[ https://issues.apache.org/jira/browse/SPARK-28421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28421: -- Affects Version/s: 2.4.3 Priority: Minor (was: Major) Fix Version/s: 2.4.4 > SparseVector.apply performance optimization > --- > > Key: SPARK-28421 > URL: https://issues.apache.org/jira/browse/SPARK-28421 > Project: Spark > Issue Type: Improvement > Components: ML >Affects Versions: 3.0.0, 2.4.3 >Reporter: zhengruifeng >Assignee: zhengruifeng >Priority: Minor > Fix For: 2.4.4, 3.0.0 > > > Current impl of SparseVector.apply is inefficient: > on each call, breeze.linalg.SparseVector & > breeze.collection.mutable.SparseArray are created internally, then > binary-search is used to search the input position. > > This place should be optimized like .ml.SparseMatrix, which directly use > binary search, without conversion to breeze.linalg.Matrix. > > I tested the performance and found that if we avoid the internal conversions, > then a 2.5~5X speed up can be obtained. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28288) Convert and port 'window.sql' into UDF test base
[ https://issues.apache.org/jira/browse/SPARK-28288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28288: Assignee: YoungGyu Chun > Convert and port 'window.sql' into UDF test base > > > Key: SPARK-28288 > URL: https://issues.apache.org/jira/browse/SPARK-28288 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL, Tests >Affects Versions: 3.0.0 >Reporter: Hyukjin Kwon >Assignee: YoungGyu Chun >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28288) Convert and port 'window.sql' into UDF test base
[ https://issues.apache.org/jira/browse/SPARK-28288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28288. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25195 [https://github.com/apache/spark/pull/25195] > Convert and port 'window.sql' into UDF test base > > > Key: SPARK-28288 > URL: https://issues.apache.org/jira/browse/SPARK-28288 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL, Tests >Affects Versions: 3.0.0 >Reporter: Hyukjin Kwon >Assignee: YoungGyu Chun >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files
[
https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892875#comment-16892875
]
Tomas Bartalos commented on SPARK-28025:
I'm also affected by performance issue caused by .crc files leak in checkpoint
directory. [~skonto] thank you for the workaround, it works.
Would it be possible to implement cleaning of crc files, when one needs the
checksum ?
> HDFSBackedStateStoreProvider should not leak .crc files
>
>
> Key: SPARK-28025
> URL: https://issues.apache.org/jira/browse/SPARK-28025
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3
> Kubernetes 1.11(?) (OpenShift)
> StateStore storage on a mounted PVC. Viewed as a local filesystem by the
> `FileContextBasedCheckpointFileManager` :
> {noformat}
> scala> glusterfm.isLocal
> res17: Boolean = true{noformat}
>Reporter: Gerard Maas
>Priority: Major
>
> The HDFSBackedStateStoreProvider when using the default CheckpointFileManager
> is leaving '.crc' files behind. There's a .crc file created for each
> `atomicFile` operation of the CheckpointFileManager.
> Over time, the number of files becomes very large. It makes the state store
> file system constantly increase in size and, in our case, deteriorates the
> file system performance.
> Here's a sample of one of our spark storage volumes after 2 days of execution
> (4 stateful streaming jobs, each on a different sub-dir):
> #
> {noformat}
> Total files in PVC (used for checkpoints and state store)
> $find . | wc -l
> 431796
> # .crc files
> $find . -name "*.crc" | wc -l
> 418053{noformat}
> With each .crc file taking one storage block, the used storage runs into the
> GBs of data.
> These jobs are running on Kubernetes. Our shared storage provider, GlusterFS,
> shows serious performance deterioration with this large number of files:
> {noformat}
> DEBUG HDFSBackedStateStoreProvider: fetchFiles() took 29164ms{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28415) Add messageHandler to Kafka 10 direct stream API
[ https://issues.apache.org/jira/browse/SPARK-28415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892888#comment-16892888 ] Michael Spector commented on SPARK-28415: - In this case I think it's a regression / broken API bug. > Add messageHandler to Kafka 10 direct stream API > > > Key: SPARK-28415 > URL: https://issues.apache.org/jira/browse/SPARK-28415 > Project: Spark > Issue Type: New Feature > Components: DStreams >Affects Versions: 2.4.3 >Reporter: Michael Spector >Priority: Major > > Lack of messageHandler parameter to KafkaUtils.createDirectStrem(...) in new > Kafka API is what prevents us from upgrading our processes to use it, and > here's why: > # messageHandler() allowed parsing / filtering / projecting huge JSON files > at an early stage (only a small subset of JSON fields is required for a > process), without this current cluster configuration doesn't keep up with the > traffic. > # Transforming Kafka events right after a stream is created prevents from > using HasOffsetRanges interface later. This means that whole message must be > propagated to the end of a pipeline, which is very ineffective. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28511) Get REV from RELEASE_VERSION instead of VERSION
[
https://issues.apache.org/jira/browse/SPARK-28511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-28511.
---
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 25254
[https://github.com/apache/spark/pull/25254]
> Get REV from RELEASE_VERSION instead of VERSION
> ---
>
> Key: SPARK-28511
> URL: https://issues.apache.org/jira/browse/SPARK-28511
> Project: Spark
> Issue Type: Bug
> Components: Project Infra
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Minor
> Fix For: 3.0.0
>
>
> Unlike the other versions, `x.x.0-SNAPSHOT` causes `x.x.-1`. Although this
> will not happen in the tags (there is no `SNAPSHOT` postfix), we had better
> fix this.
> {code}
> $ dev/create-release/do-release-docker.sh -d /tmp/spark-3.0.0 -n
> Output directory already exists. Overwrite and continue? [y/n] y
> Branch [branch-2.4]: master
> Current branch version is 3.0.0-SNAPSHOT.
> Release [3.0.-1]:
> {code}
> The following is the expected behavior.
> {code}
> $ dev/create-release/do-release-docker.sh -d /tmp/spark-3.0.0 -n
> Branch [branch-2.4]: master
> Current branch version is 3.0.0-SNAPSHOT.
> Release [3.0.0]:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28511) Get REV from RELEASE_VERSION instead of VERSION
[
https://issues.apache.org/jira/browse/SPARK-28511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-28511:
-
Assignee: Dongjoon Hyun
> Get REV from RELEASE_VERSION instead of VERSION
> ---
>
> Key: SPARK-28511
> URL: https://issues.apache.org/jira/browse/SPARK-28511
> Project: Spark
> Issue Type: Bug
> Components: Project Infra
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Minor
>
> Unlike the other versions, `x.x.0-SNAPSHOT` causes `x.x.-1`. Although this
> will not happen in the tags (there is no `SNAPSHOT` postfix), we had better
> fix this.
> {code}
> $ dev/create-release/do-release-docker.sh -d /tmp/spark-3.0.0 -n
> Output directory already exists. Overwrite and continue? [y/n] y
> Branch [branch-2.4]: master
> Current branch version is 3.0.0-SNAPSHOT.
> Release [3.0.-1]:
> {code}
> The following is the expected behavior.
> {code}
> $ dev/create-release/do-release-docker.sh -d /tmp/spark-3.0.0 -n
> Branch [branch-2.4]: master
> Current branch version is 3.0.0-SNAPSHOT.
> Release [3.0.0]:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27734) Add memory based thresholds for shuffle spill
[
https://issues.apache.org/jira/browse/SPARK-27734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893018#comment-16893018
]
Paweł Wiejacha commented on SPARK-27734:
We also encountered this problem. We've reduced the problem to shuffling 60 GiB
of data divided into 5 partitions using *repartitionAndSortWithinPartitions*()
and processing (*foreachPartition*()) all of them using a single executor that
has 2 GiB of memory assigned. Processing each partition takes ~70 minutes (52
min GC time) and CPU usage is very high (due to GC).
Setting *spark.shuffle.spill.numElementsForceSpillThreshold* is very
inconvenient, so it would be nice to accept Adrian's pull request.
> Add memory based thresholds for shuffle spill
> -
>
> Key: SPARK-27734
> URL: https://issues.apache.org/jira/browse/SPARK-27734
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
>Affects Versions: 3.0.0
>Reporter: Adrian Muraru
>Priority: Minor
>
> When running large shuffles (700TB input data, 200k map tasks, 50k reducers
> on a 300 nodes cluster) the job is regularly OOMing in map and reduce phase.
> IIUC ShuffleExternalSorter (map side) and ExternalAppendOnlyMap and
> ExternalSorter (reduce side) are trying to max out the available execution
> memory. This in turn doesn't play nice with the Garbage Collector and
> executors are failing with OutOfMemoryError when the memory allocation from
> these in-memory structure is maxing out the available heap size (in our case
> we are running with 9 cores/executor, 32G per executor)
> To mitigate this, I set
> {{spark.shuffle.spill.numElementsForceSpillThreshold}} to force the spill on
> disk. While this config works, it is not flexible enough as it's expressed in
> number of elements, and in our case we run multiple shuffles in a single job
> and element size is different from one stage to another.
> We have an internal patch to extend this behaviour and add two new parameters
> to control the spill based on memory usage:
> - spark.shuffle.spill.map.maxRecordsSizeForSpillThreshold
> - spark.shuffle.spill.reduce.maxRecordsSizeForSpillThreshold
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28509) K8S integration tests are failing
[
https://issues.apache.org/jira/browse/SPARK-28509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893031#comment-16893031
]
shane knapp commented on SPARK-28509:
-
checked that worker today and all k8s builds are running successfully:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13265/console
> K8S integration tests are failing
> -
>
> Key: SPARK-28509
> URL: https://issues.apache.org/jira/browse/SPARK-28509
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Tests
>Affects Versions: 3.0.0
>Reporter: Marcelo Vanzin
>Assignee: shane knapp
>Priority: Major
>
> I've been seeing lots of failures in master. e.g.
> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13180/console
> {noformat}
> - Start pod creation from template *** FAILED ***
> io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
> at
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:201)
> at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
> at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> ...
> - PVs with local storage *** FAILED ***
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at: https://192.168.39.112:8443/api/v1/persistentvolumes. Message:
> PersistentVolume "test-local-pv" is invalid: [spec.local: Forbidden: Local
> volumes are disabled by feature-gate, metadata.annotations: Required value:
> Local volume requires node affinity]. Received status: Status(apiVersion=v1,
> code=422, details=StatusDetails(causes=[StatusCause(field=spec.local,
> message=Forbidden: Local volumes are disabled by feature-gate,
> reason=FieldValueForbidden, additionalProperties={}),
> StatusCause(field=metadata.annotations, message=Required value: Local volume
> requires node affinity, reason=FieldValueRequired, additionalProperties={})],
> group=null, kind=PersistentVolume, name=test-local-pv,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=PersistentVolume "test-local-pv" is invalid: [spec.local: Forbidden:
> Local volumes are disabled by feature-gate, metadata.annotations: Required
> value: Local volume requires node affinity],
> metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=Invalid, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:478)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:417)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:787)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:357)
> at
> org.apache.spark.deploy.k8s.integrationtest.PVTestsSuite.setupLocalStorage(PVTestsSuite.scala:87)
> at
> org.apache.spark.deploy.k8s.integrationtest.PVTestsSuite.$anonfun$$init$$1(PVTestsSuite.scala:137)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> ...
> - Launcher client dependencies *** FAILED ***
> The code passed to eventually never returned normally. Attempted 1 times
> over 6.67390320003 minutes. Last failure message: assertion failed:
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-28509) K8S integration tests are failing
[
https://issues.apache.org/jira/browse/SPARK-28509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shane knapp closed SPARK-28509.
---
> K8S integration tests are failing
> -
>
> Key: SPARK-28509
> URL: https://issues.apache.org/jira/browse/SPARK-28509
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Tests
>Affects Versions: 3.0.0
>Reporter: Marcelo Vanzin
>Assignee: shane knapp
>Priority: Major
>
> I've been seeing lots of failures in master. e.g.
> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13180/console
> {noformat}
> - Start pod creation from template *** FAILED ***
> io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
> at
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:201)
> at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
> at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> ...
> - PVs with local storage *** FAILED ***
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at: https://192.168.39.112:8443/api/v1/persistentvolumes. Message:
> PersistentVolume "test-local-pv" is invalid: [spec.local: Forbidden: Local
> volumes are disabled by feature-gate, metadata.annotations: Required value:
> Local volume requires node affinity]. Received status: Status(apiVersion=v1,
> code=422, details=StatusDetails(causes=[StatusCause(field=spec.local,
> message=Forbidden: Local volumes are disabled by feature-gate,
> reason=FieldValueForbidden, additionalProperties={}),
> StatusCause(field=metadata.annotations, message=Required value: Local volume
> requires node affinity, reason=FieldValueRequired, additionalProperties={})],
> group=null, kind=PersistentVolume, name=test-local-pv,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=PersistentVolume "test-local-pv" is invalid: [spec.local: Forbidden:
> Local volumes are disabled by feature-gate, metadata.annotations: Required
> value: Local volume requires node affinity],
> metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=Invalid, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:478)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:417)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:787)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:357)
> at
> org.apache.spark.deploy.k8s.integrationtest.PVTestsSuite.setupLocalStorage(PVTestsSuite.scala:87)
> at
> org.apache.spark.deploy.k8s.integrationtest.PVTestsSuite.$anonfun$$init$$1(PVTestsSuite.scala:137)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> ...
> - Launcher client dependencies *** FAILED ***
> The code passed to eventually never returned normally. Attempted 1 times
> over 6.67390320003 minutes. Last failure message: assertion failed:
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-25914) Separate projection from grouping and aggregate in logical Aggregate
[
https://issues.apache.org/jira/browse/SPARK-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li reassigned SPARK-25914:
---
Assignee: (was: Dilip Biswal)
> Separate projection from grouping and aggregate in logical Aggregate
>
>
> Key: SPARK-25914
> URL: https://issues.apache.org/jira/browse/SPARK-25914
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Maryann Xue
>Priority: Major
>
> Currently the Spark SQL logical Aggregate has two expression fields:
> {{groupingExpressions}} and {{aggregateExpressions}}, in which
> {{aggregateExpressions}} is actually the result expressions, or in other
> words, the project list in the SELECT clause.
>
> This would cause an exception while processing the following query:
> {code:java}
> SELECT concat('x', concat(a, 's'))
> FROM testData2
> GROUP BY concat(a, 's'){code}
> After optimization, the query becomes:
> {code:java}
> SELECT concat('x', a, 's')
> FROM testData2
> GROUP BY concat(a, 's'){code}
> The optimization rule {{CombineConcats}} optimizes the expressions by
> flattening "concat" and causes the query to fail since the expression
> {{concat('x', a, 's')}} in the SELECT clause is neither referencing a
> grouping expression nor a aggregate expression.
>
> The problem is that we try to mix two operations in one operator, and worse,
> in one field: the group-and-aggregate operation and the project operation.
> There are two ways to solve this problem:
> 1. Break the two operations into two logical operators, which means a
> group-by query can usually be mapped into a Project-over-Aggregate pattern.
> 2. Break the two operations into multiple fields in the Aggregate operator,
> the same way we do for physical aggregate classes (e.g.,
> {{HashAggregateExec}}, or {{SortAggregateExec}}). Thus,
> {{groupingExpressions}} would still be the expressions from the GROUP BY
> clause (as before), but {{aggregateExpressions}} would contain aggregate
> functions only, and {{resultExpressions}} would be the project list in the
> SELECT clause holding references to either {{groupingExpressions}} or
> {{aggregateExpressions}}.
>
> I would say option 1 is even clearer, but it would be more likely to break
> the pattern matching in existing optimization rules and thus require more
> changes in the compiler. So we'd probably wanna go with option 2. That said,
> I suggest we achieve this goal through two iterative steps:
>
> Phase 1: Keep the current fields of logical Aggregate as
> {{groupingExpressions}} and {{aggregateExpressions}}, but change the
> semantics of {{aggregateExpressions}} by replacing the grouping expressions
> with corresponding references to expressions in {{groupingExpressions}}. The
> aggregate expressions in {{aggregateExpressions}} will remain the same.
>
> Phase 2: Add {{resultExpressions}} for the project list, and keep only
> aggregate expressions in {{aggregateExpressions}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25787) [K8S] Spark can't use data locality information
[ https://issues.apache.org/jira/browse/SPARK-25787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893043#comment-16893043 ] Paweł Wiejacha commented on SPARK-25787: I *can* reproduce this issue. In Spark UI, Locality Level is always ANY instead of NODE_LOCAL when reading data from HDFS. As Yinan Li said, it seems that: > Support for data locality on k8s has not been ported to the upstream Spark > repo yet. I think that at least the pull request below should be ported and merged to support HDFS data locality in Spark on Kubernetes. https://github.com/apache-spark-on-k8s/spark/pull/216 Could you please reopen this issue? > [K8S] Spark can't use data locality information > --- > > Key: SPARK-25787 > URL: https://issues.apache.org/jira/browse/SPARK-25787 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Maciej Bryński >Priority: Major > > I started experimenting with Spark based on this presentation: > https://www.slideshare.net/databricks/hdfs-on-kuberneteslessons-learned-with-kimoon-kim > I'm using excelent https://github.com/apache-spark-on-k8s/kubernetes-HDFS > charts to deploy HDFS. > Unfortunately reading from HDFS gives ANY locality for every task. > Is data locality working on Kubernetes cluster ? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28516) adds `to_char`
Dylan Guedes created SPARK-28516: Summary: adds `to_char` Key: SPARK-28516 URL: https://issues.apache.org/jira/browse/SPARK-28516 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.0.0 Reporter: Dylan Guedes Currently, Spark does not have support for `to_char`. PgSQL, however, [does|[https://www.postgresql.org/docs/9.6/functions-formatting.html]]: Query example: SELECT to_char(SUM(n) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING),'9D9') -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16754) NPE when defining case class and searching Encoder in the same line
[
https://issues.apache.org/jira/browse/SPARK-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893115#comment-16893115
]
Josh Rosen commented on SPARK-16754:
This is still a problem as of Spark 2.4.x; I just encountered this in a
Zeppelin notebook. I was able to work around the problem by putting my case
class definition and usage in separate Zeppelin paragraphs / cells, but it's a
confusing experience for new users.
> NPE when defining case class and searching Encoder in the same line
> ---
>
> Key: SPARK-16754
> URL: https://issues.apache.org/jira/browse/SPARK-16754
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0, 2.3.0
> Environment: Spark Shell for Scala 2.11
>Reporter: Shixiong Zhu
>Priority: Minor
>
> Reproducer:
> {code}
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
> case class TestCaseClass(value: Int)
> import spark.implicits._
> Seq(TestCaseClass(1)).toDS().collect()
> // Exiting paste mode, now interpreting.
> java.lang.RuntimeException: baseClassName: $line14.$read
> at
> org.apache.spark.sql.catalyst.encoders.OuterScopes$$anonfun$getOuterScope$1.apply(OuterScopes.scala:62)
> at
> org.apache.spark.sql.catalyst.expressions.objects.NewInstance$$anonfun$12.apply(objects.scala:251)
> at
> org.apache.spark.sql.catalyst.expressions.objects.NewInstance$$anonfun$12.apply(objects.scala:251)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.spark.sql.catalyst.expressions.objects.NewInstance.doGenCode(objects.scala:251)
> at
> org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:104)
> at
> org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:101)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:101)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$$anonfun$3.apply(GenerateSafeProjection.scala:145)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$$anonfun$3.apply(GenerateSafeProjection.scala:142)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.immutable.List.map(List.scala:285)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:142)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:36)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:821)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.constructProjection$lzycompute(ExpressionEncoder.scala:258)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.constructProjection(ExpressionEncoder.scala:258)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:289)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$15.apply(Dataset.scala:2218)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$15.apply(Dataset.scala:2218)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2218)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
> at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2568)
> at
> org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2217)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collect$1.apply(Dataset.scala:)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collect$1.apply(Dataset.sc
[jira] [Updated] (SPARK-28502) Error with struct conversion while using pandas_udf
[
https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nasir Ali updated SPARK-28502:
--
Issue Type: Bug (was: Question)
> Error with struct conversion while using pandas_udf
> ---
>
> Key: SPARK-28502
> URL: https://issues.apache.org/jira/browse/SPARK-28502
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 2.4.3
> Environment: OS: Ubuntu
> Python: 3.6
>Reporter: Nasir Ali
>Priority: Minor
>
> What I am trying to do: Group data based on time intervals (e.g., 15 days
> window) and perform some operations on dataframe using (pandas) UDFs. I don't
> know if there is a better/cleaner way to do it.
> Below is the sample code that I tried and error message I am getting.
>
> {code:java}
> df = sparkSession.createDataFrame([(17.00, "2018-03-10T15:27:18+00:00"),
> (13.00, "2018-03-11T12:27:18+00:00"),
> (25.00, "2018-03-12T11:27:18+00:00"),
> (20.00, "2018-03-13T15:27:18+00:00"),
> (17.00, "2018-03-14T12:27:18+00:00"),
> (99.00, "2018-03-15T11:27:18+00:00"),
> (156.00, "2018-03-22T11:27:18+00:00"),
> (17.00, "2018-03-31T11:27:18+00:00"),
> (25.00, "2018-03-15T11:27:18+00:00"),
> (25.00, "2018-03-16T11:27:18+00:00")
> ],
>["id", "ts"])
> df = df.withColumn('ts', df.ts.cast('timestamp'))
> schema = StructType([
> StructField("id", IntegerType()),
> StructField("ts", TimestampType())
> ])
> @pandas_udf(schema, PandasUDFType.GROUPED_MAP)
> def some_udf(df):
> # some computation
> return df
> df.groupby('id', F.window("ts", "15 days")).apply(some_udf).show()
> {code}
> This throws following exception:
> {code:java}
> TypeError: Unsupported type in conversion from Arrow: struct timestamp[us, tz=America/Chicago], end: timestamp[us, tz=America/Chicago]>
> {code}
>
> However, if I use builtin agg method then it works all fine. For example,
> {code:java}
> df.groupby('id', F.window("ts", "15 days")).mean().show(truncate=False)
> {code}
> Output
> {code:java}
> +-+--+---+
> |id |window|avg(id)|
> +-+--+---+
> |13.0 |[2018-03-05 00:00:00, 2018-03-20 00:00:00]|13.0 |
> |17.0 |[2018-03-20 00:00:00, 2018-04-04 00:00:00]|17.0 |
> |156.0|[2018-03-20 00:00:00, 2018-04-04 00:00:00]|156.0 |
> |99.0 |[2018-03-05 00:00:00, 2018-03-20 00:00:00]|99.0 |
> |20.0 |[2018-03-05 00:00:00, 2018-03-20 00:00:00]|20.0 |
> |17.0 |[2018-03-05 00:00:00, 2018-03-20 00:00:00]|17.0 |
> |25.0 |[2018-03-05 00:00:00, 2018-03-20 00:00:00]|25.0 |
> +-+--+---+
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16754) NPE when defining case class and searching Encoder in the same line
[
https://issues.apache.org/jira/browse/SPARK-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893125#comment-16893125
]
Shixiong Zhu commented on SPARK-16754:
--
I think prepending
`org.apache.spark.sql.catalyst.encoders.OuterScopes.addOuterScope(this)` to
every command should fix the problem. I already forget why we didn't do this
for Scala 2.11.
> NPE when defining case class and searching Encoder in the same line
> ---
>
> Key: SPARK-16754
> URL: https://issues.apache.org/jira/browse/SPARK-16754
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0, 2.3.0
> Environment: Spark Shell for Scala 2.11
>Reporter: Shixiong Zhu
>Priority: Minor
>
> Reproducer:
> {code}
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
> case class TestCaseClass(value: Int)
> import spark.implicits._
> Seq(TestCaseClass(1)).toDS().collect()
> // Exiting paste mode, now interpreting.
> java.lang.RuntimeException: baseClassName: $line14.$read
> at
> org.apache.spark.sql.catalyst.encoders.OuterScopes$$anonfun$getOuterScope$1.apply(OuterScopes.scala:62)
> at
> org.apache.spark.sql.catalyst.expressions.objects.NewInstance$$anonfun$12.apply(objects.scala:251)
> at
> org.apache.spark.sql.catalyst.expressions.objects.NewInstance$$anonfun$12.apply(objects.scala:251)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.spark.sql.catalyst.expressions.objects.NewInstance.doGenCode(objects.scala:251)
> at
> org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:104)
> at
> org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:101)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:101)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$$anonfun$3.apply(GenerateSafeProjection.scala:145)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$$anonfun$3.apply(GenerateSafeProjection.scala:142)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.immutable.List.map(List.scala:285)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:142)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:36)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:821)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.constructProjection$lzycompute(ExpressionEncoder.scala:258)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.constructProjection(ExpressionEncoder.scala:258)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:289)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$15.apply(Dataset.scala:2218)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$15.apply(Dataset.scala:2218)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2218)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
> at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2568)
> at
> org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2217)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collect$1.apply(Dataset.scala:)
> at
> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collect$1.apply(Dataset.scala:)
> at org.apache.spark.sql.Dataset.withCallback(Dataset.
[jira] [Assigned] (SPARK-27845) DataSourceV2: InsertTable
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-27845: --- Assignee: John Zhuge > DataSourceV2: InsertTable > - > > Key: SPARK-27845 > URL: https://issues.apache.org/jira/browse/SPARK-27845 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Major > > Support multiple catalogs in the following use cases: > * INSERT INTO [TABLE] catalog.db.tbl > * INSERT OVERWRITE TABLE catalog.db.tbl -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27845) DataSourceV2: InsertTable
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893162#comment-16893162 ] Burak Yavuz commented on SPARK-27845: - Resolved with [https://github.com/apache/spark/pull/24832] > DataSourceV2: InsertTable > - > > Key: SPARK-27845 > URL: https://issues.apache.org/jira/browse/SPARK-27845 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Major > > Support multiple catalogs in the following use cases: > * INSERT INTO [TABLE] catalog.db.tbl > * INSERT OVERWRITE TABLE catalog.db.tbl -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-27845) DataSourceV2: InsertTable
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-27845. - Resolution: Done Fix Version/s: 3.0.0 > DataSourceV2: InsertTable > - > > Key: SPARK-27845 > URL: https://issues.apache.org/jira/browse/SPARK-27845 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Major > Fix For: 3.0.0 > > > Support multiple catalogs in the following use cases: > * INSERT INTO [TABLE] catalog.db.tbl > * INSERT OVERWRITE TABLE catalog.db.tbl -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28264) Revisiting Python / pandas UDF
[ https://issues.apache.org/jira/browse/SPARK-28264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893209#comment-16893209 ] Bryan Cutler commented on SPARK-28264: -- It's great to be taking another look at this, I think some aspects are really confusing. I left some comments in the doc, but to sum it up I think anything we can do to reduce the number of arguments and options will make it more user friendly. I worry that while replacing the pandas udf types with other options would make things more flexible, I'm not sure it makes it any easier to understand. > Revisiting Python / pandas UDF > -- > > Key: SPARK-28264 > URL: https://issues.apache.org/jira/browse/SPARK-28264 > Project: Spark > Issue Type: Improvement > Components: PySpark, SQL >Affects Versions: 3.0.0 >Reporter: Reynold Xin >Assignee: Reynold Xin >Priority: Major > > In the past two years, the pandas UDFs are perhaps the most important changes > to Spark for Python data science. However, these functionalities have evolved > organically, leading to some inconsistencies and confusions among users. This > document revisits UDF definition and naming, as a result of discussions among > Xiangrui, Li Jin, Hyukjin, and Reynold. > > See document here: > [https://docs.google.com/document/d/10Pkl-rqygGao2xQf6sddt0b-4FYK4g8qr_bXLKTL65A/edit#|https://docs.google.com/document/d/10Pkl-rqygGao2xQf6sddt0b-4FYK4g8qr_bXLKTL65A/edit] > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28517) pyspark with --conf spark.jars.packages causes duplicate jars to be uploaded
Barry created SPARK-28517:
-
Summary: pyspark with --conf spark.jars.packages causes duplicate
jars to be uploaded
Key: SPARK-28517
URL: https://issues.apache.org/jira/browse/SPARK-28517
Project: Spark
Issue Type: Bug
Components: PySpark, YARN
Affects Versions: 2.4.3
Environment: spark 2.4.3_2.12 without hadoop
yarn 2.6
python 2.7.16
centos 7
Reporter: Barry
h2. Steps to reproduce:
{{spark-submit --master yarn --conf
"spark.jars.packages=org.apache.spark:spark-avro_2.12:2.4.3"
${SPARK_HOME}/examples/src/main/python/pi.py 100}}
h2. Undesirable behavior:
warnings are printed package jars have been added to the distributed cache
multiple times
{{19/07/25 23:25:07 WARN Client: Same path resource
file:///home/barryl/.ivy2/jars/org.apache.spark_spark-avro_2.12-2.4.3.jar added
multiple times to distributed cache.}}
{{19/07/25 23:25:07 WARN Client: Same path resource
file:///home/barryl/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar added
multiple times to distributed cache.}}
This does not happen for Scala jobs, only Pyspark
h2. Full output of example run.
{{[barryl@hostname ~]$ /opt/spark2/bin/spark-submit --master yarn --conf
"spark.jars.packages=org.apache.spark:spark-avro_2.12:2.4.3"
/opt/spark2/examples/src/main/python/pi.py 100}}
{{Ivy Default Cache set to: /home/barryl/.ivy2/cache}}
{{The jars for the packages stored in: /home/barryl/.ivy2/jars}}
{{:: loading settings :: url =
jar:file:/opt/spark-2.4.3-bin-without-hadoop-scala-2.12/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml}}
{{org.apache.spark#spark-avro_2.12 added as a dependency}}
{{:: resolving dependencies ::
org.apache.spark#spark-submit-parent-2c34ecff-b060-4af9-9b9f-83867672748c;1.0}}
{{ confs: [default]}}
{{ found org.apache.spark#spark-avro_2.12;2.4.3 in central}}
{{ found org.spark-project.spark#unused;1.0.0 in central}}
{{:: resolution report :: resolve 457ms :: artifacts dl 5ms}}
{{ :: modules in use:}}
{{ org.apache.spark#spark-avro_2.12;2.4.3 from central in [default]}}
{{ org.spark-project.spark#unused;1.0.0 from central in [default]}}
{{ -}}
{{ | | modules || artifacts |}}
{{ | conf | number| search|dwnlded|evicted|| number|dwnlded|}}
{{ -}}
{{ | default | 2 | 0 | 0 | 0 || 2 | 0 |}}
{{ -}}
{{:: retrieving ::
org.apache.spark#spark-submit-parent-2c34ecff-b060-4af9-9b9f-83867672748c}}
{{ confs: [default]}}
{{ 0 artifacts copied, 2 already retrieved (0kB/7ms)}}
{{19/07/25 23:25:03 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive
is set, falling back to uploading libraries under SPARK_HOME.}}
{{19/07/25 23:25:07 WARN Client: Same path resource
file:///home/barryl/.ivy2/jars/org.apache.spark_spark-avro_2.12-2.4.3.jar added
multiple times to distributed cache.}}
{{19/07/25 23:25:07 WARN Client: Same path resource
file:///home/barryl/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar added
multiple times to distributed cache.}}
{{19/07/25 23:25:28 WARN TaskSetManager: Stage 0 contains a task of very large
size (365 KB). The maximum recommended task size is 100 KB.}}
{{Pi is roughly 3.142308}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28516) Data Type Formatting Functions: `to_char`
[ https://issues.apache.org/jira/browse/SPARK-28516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28516: Summary: Data Type Formatting Functions: `to_char` (was: adds `to_char`) > Data Type Formatting Functions: `to_char` > - > > Key: SPARK-28516 > URL: https://issues.apache.org/jira/browse/SPARK-28516 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Dylan Guedes >Priority: Major > > Currently, Spark does not have support for `to_char`. PgSQL, however, > [does|[https://www.postgresql.org/docs/9.6/functions-formatting.html]]: > Query example: > SELECT to_char(SUM(n) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 > FOLLOWING),'9D9') -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28137) Data Type Formatting Functions: `to_number`
[
https://issues.apache.org/jira/browse/SPARK-28137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-28137:
Summary: Data Type Formatting Functions: `to_number` (was: Data Type
Formatting Functions: `to_char`)
> Data Type Formatting Functions: `to_number`
> ---
>
> Key: SPARK-28137
> URL: https://issues.apache.org/jira/browse/SPARK-28137
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> ||Function||Return Type||Description||Example||
> |{{to_char(}}{{timestamp}}{{, }}{{text}}{{)}}|{{text}}|convert time stamp to
> string|{{to_char(current_timestamp, 'HH12:MI:SS')}}|
> |{{to_char(}}{{interval}}{{, }}{{text}}{{)}}|{{text}}|convert interval to
> string|{{to_char(interval '15h 2m 12s', 'HH24:MI:SS')}}|
> |{{to_char(}}{{int}}{{, }}{{text}}{{)}}|{{text}}|convert integer to
> string|{{to_char(125, '999')}}|
> |{{to_char}}{{(}}{{double precision}}{{, }}{{text}}{{)}}|{{text}}|convert
> real/double precision to string|{{to_char(125.8::real, '999D9')}}|
> |{{to_char(}}{{numeric}}{{, }}{{text}}{{)}}|{{text}}|convert numeric to
> string|{{to_char(-125.8, '999D99S')}}|
> |{{to_number(}}{{text}}{{, }}{{text}}{{)}}|{{numeric}}|convert string to
> numeric|{{to_number('12,454.8-', '99G999D9S')}}|
> https://www.postgresql.org/docs/12/functions-formatting.html
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28137) Data Type Formatting Functions: `to_char`
[
https://issues.apache.org/jira/browse/SPARK-28137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-28137:
Summary: Data Type Formatting Functions: `to_char` (was: Data Type
Formatting Functions)
> Data Type Formatting Functions: `to_char`
> -
>
> Key: SPARK-28137
> URL: https://issues.apache.org/jira/browse/SPARK-28137
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> ||Function||Return Type||Description||Example||
> |{{to_char(}}{{timestamp}}{{, }}{{text}}{{)}}|{{text}}|convert time stamp to
> string|{{to_char(current_timestamp, 'HH12:MI:SS')}}|
> |{{to_char(}}{{interval}}{{, }}{{text}}{{)}}|{{text}}|convert interval to
> string|{{to_char(interval '15h 2m 12s', 'HH24:MI:SS')}}|
> |{{to_char(}}{{int}}{{, }}{{text}}{{)}}|{{text}}|convert integer to
> string|{{to_char(125, '999')}}|
> |{{to_char}}{{(}}{{double precision}}{{, }}{{text}}{{)}}|{{text}}|convert
> real/double precision to string|{{to_char(125.8::real, '999D9')}}|
> |{{to_char(}}{{numeric}}{{, }}{{text}}{{)}}|{{text}}|convert numeric to
> string|{{to_char(-125.8, '999D99S')}}|
> |{{to_number(}}{{text}}{{, }}{{text}}{{)}}|{{numeric}}|convert string to
> numeric|{{to_number('12,454.8-', '99G999D9S')}}|
> https://www.postgresql.org/docs/12/functions-formatting.html
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28516) Data Type Formatting Functions: `to_char`
[
https://issues.apache.org/jira/browse/SPARK-28516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-28516:
Description:
Currently, Spark does not have support for `to_char`. PgSQL, however,
[does|[https://www.postgresql.org/docs/12/functions-formatting.html]]:
Query example:
{code:sql}
SELECT to_char(SUM(n) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1
FOLLOWING),'9D9')
{code}
||Function||Return Type||Description||Example||
|{{to_char(}}{{timestamp}}{{, }}{{text}}{{)}}|{{text}}|convert time stamp to
string|{{to_char(current_timestamp, 'HH12:MI:SS')}}|
|{{to_char(}}{{interval}}{{, }}{{text}}{{)}}|{{text}}|convert interval to
string|{{to_char(interval '15h 2m 12s', 'HH24:MI:SS')}}|
|{{to_char(}}{{int}}{{, }}{{text}}{{)}}|{{text}}|convert integer to
string|{{to_char(125, '999')}}|
|{{to_char}}{{(}}{{double precision}}{{, }}{{text}}{{)}}|{{text}}|convert
real/double precision to string|{{to_char(125.8::real, '999D9')}}|
|{{to_char(}}{{numeric}}{{, }}{{text}}{{)}}|{{text}}|convert numeric to
string|{{to_char(-125.8, '999D99S')}}|
was:
Currently, Spark does not have support for `to_char`. PgSQL, however,
[does|[https://www.postgresql.org/docs/9.6/functions-formatting.html]]:
Query example:
SELECT to_char(SUM(n) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1
FOLLOWING),'9D9')
> Data Type Formatting Functions: `to_char`
> -
>
> Key: SPARK-28516
> URL: https://issues.apache.org/jira/browse/SPARK-28516
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Currently, Spark does not have support for `to_char`. PgSQL, however,
> [does|[https://www.postgresql.org/docs/12/functions-formatting.html]]:
> Query example:
> {code:sql}
> SELECT to_char(SUM(n) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1
> FOLLOWING),'9D9')
> {code}
> ||Function||Return Type||Description||Example||
> |{{to_char(}}{{timestamp}}{{, }}{{text}}{{)}}|{{text}}|convert time stamp to
> string|{{to_char(current_timestamp, 'HH12:MI:SS')}}|
> |{{to_char(}}{{interval}}{{, }}{{text}}{{)}}|{{text}}|convert interval to
> string|{{to_char(interval '15h 2m 12s', 'HH24:MI:SS')}}|
> |{{to_char(}}{{int}}{{, }}{{text}}{{)}}|{{text}}|convert integer to
> string|{{to_char(125, '999')}}|
> |{{to_char}}{{(}}{{double precision}}{{, }}{{text}}{{)}}|{{text}}|convert
> real/double precision to string|{{to_char(125.8::real, '999D9')}}|
> |{{to_char(}}{{numeric}}{{, }}{{text}}{{)}}|{{text}}|convert numeric to
> string|{{to_char(-125.8, '999D99S')}}|
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28137) Data Type Formatting Functions: `to_number`
[
https://issues.apache.org/jira/browse/SPARK-28137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-28137:
Description:
||Function||Return Type||Description||Example||
|{{to_number(}}{{text}}{{, }}{{text}}{{)}}|{{numeric}}|convert string to
numeric|{{to_number('12,454.8-', '99G999D9S')}}|
https://www.postgresql.org/docs/12/functions-formatting.html
was:
||Function||Return Type||Description||Example||
|{{to_char(}}{{timestamp}}{{, }}{{text}}{{)}}|{{text}}|convert time stamp to
string|{{to_char(current_timestamp, 'HH12:MI:SS')}}|
|{{to_char(}}{{interval}}{{, }}{{text}}{{)}}|{{text}}|convert interval to
string|{{to_char(interval '15h 2m 12s', 'HH24:MI:SS')}}|
|{{to_char(}}{{int}}{{, }}{{text}}{{)}}|{{text}}|convert integer to
string|{{to_char(125, '999')}}|
|{{to_char}}{{(}}{{double precision}}{{, }}{{text}}{{)}}|{{text}}|convert
real/double precision to string|{{to_char(125.8::real, '999D9')}}|
|{{to_char(}}{{numeric}}{{, }}{{text}}{{)}}|{{text}}|convert numeric to
string|{{to_char(-125.8, '999D99S')}}|
|{{to_number(}}{{text}}{{, }}{{text}}{{)}}|{{numeric}}|convert string to
numeric|{{to_number('12,454.8-', '99G999D9S')}}|
https://www.postgresql.org/docs/12/functions-formatting.html
> Data Type Formatting Functions: `to_number`
> ---
>
> Key: SPARK-28137
> URL: https://issues.apache.org/jira/browse/SPARK-28137
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> ||Function||Return Type||Description||Example||
> |{{to_number(}}{{text}}{{, }}{{text}}{{)}}|{{numeric}}|convert string to
> numeric|{{to_number('12,454.8-', '99G999D9S')}}|
> https://www.postgresql.org/docs/12/functions-formatting.html
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23469) HashingTF should use corrected MurmurHash3 implementation
[ https://issues.apache.org/jira/browse/SPARK-23469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893258#comment-16893258 ] Huaxin Gao commented on SPARK-23469: I will work on this jira once PR https://github.com/apache/spark/pull/25250 (migrate the implementation of HashingTF from MLlib to ML) is merged. > HashingTF should use corrected MurmurHash3 implementation > - > > Key: SPARK-23469 > URL: https://issues.apache.org/jira/browse/SPARK-23469 > Project: Spark > Issue Type: Bug > Components: ML >Affects Versions: 2.4.0 >Reporter: Joseph K. Bradley >Priority: Major > > [SPARK-23381] added a corrected MurmurHash3 implementation but left the old > implementation alone. In Spark 2.3 and earlier, HashingTF will use the old > implementation. (We should not backport a fix for HashingTF since it would > be a major change of behavior.) But we should correct HashingTF in Spark > 2.4; this JIRA is for tracking this fix. > * Update HashingTF to use new implementation of MurmurHash3 > * Ensure backwards compatibility for ML persistence by having HashingTF use > the old MurmurHash3 when a model from Spark 2.3 or earlier is loaded. We can > add a Param to allow this. > Also, HashingTF still calls into the old spark.mllib.feature.HashingTF, so I > recommend we first migrate the code to spark.ml: [SPARK-21748]. We can leave > spark.mllib alone and just fix MurmurHash3 in spark.ml. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28476) Support ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/SPARK-28476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893274#comment-16893274 ] Xiao Li commented on SPARK-28476: - [~ivy.weichan.xu] > Support ALTER DATABASE SET LOCATION > --- > > Key: SPARK-28476 > URL: https://issues.apache.org/jira/browse/SPARK-28476 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Xiao Li >Priority: Major > > We can support the syntax of ALTER (DATABASE|SCHEMA) database_name SET > LOCATION path > Ref: [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL] > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28476) Support ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/SPARK-28476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-28476: --- Assignee: Weichen Xu > Support ALTER DATABASE SET LOCATION > --- > > Key: SPARK-28476 > URL: https://issues.apache.org/jira/browse/SPARK-28476 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Xiao Li >Assignee: Weichen Xu >Priority: Major > > We can support the syntax of ALTER (DATABASE|SCHEMA) database_name SET > LOCATION path > Ref: [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL] > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28289) Convert and port 'union.sql' into UDF test base
[ https://issues.apache.org/jira/browse/SPARK-28289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28289. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25202 [https://github.com/apache/spark/pull/25202] > Convert and port 'union.sql' into UDF test base > --- > > Key: SPARK-28289 > URL: https://issues.apache.org/jira/browse/SPARK-28289 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL, Tests >Affects Versions: 3.0.0 >Reporter: Hyukjin Kwon >Assignee: Yiheng Wang >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28289) Convert and port 'union.sql' into UDF test base
[ https://issues.apache.org/jira/browse/SPARK-28289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28289: Assignee: Yiheng Wang > Convert and port 'union.sql' into UDF test base > --- > > Key: SPARK-28289 > URL: https://issues.apache.org/jira/browse/SPARK-28289 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL, Tests >Affects Versions: 3.0.0 >Reporter: Hyukjin Kwon >Assignee: Yiheng Wang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28365) Fallback locale to en_US in StopWordsRemover if system default locale isn't in available locales in JVM
[
https://issues.apache.org/jira/browse/SPARK-28365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-28365.
--
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 25133
[https://github.com/apache/spark/pull/25133]
> Fallback locale to en_US in StopWordsRemover if system default locale isn't
> in available locales in JVM
> ---
>
> Key: SPARK-28365
> URL: https://issues.apache.org/jira/browse/SPARK-28365
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 3.0.0
>Reporter: Liang-Chi Hsieh
>Assignee: Liang-Chi Hsieh
>Priority: Major
> Fix For: 3.0.0
>
>
> Because the local default locale isn't in available locales at {{Locale}},
> when I did some tests locally with python code, {{StopWordsRemover}} related
> python test hits some errors, like:
> {code}
> Traceback (most recent call last):
> File "/spark-1/python/pyspark/ml/tests/test_feature.py", line 87, in
> test_stopwordsremover
> stopWordRemover = StopWordsRemover(inputCol="input", outputCol="output")
> File "/spark-1/python/pyspark/__init__.py", line 111, in wrapper
> return func(self, **kwargs)
> File "/spark-1/python/pyspark/ml/feature.py", line 2646, in __init__
> self.uid)
> File "/spark-1/python/pyspark/ml/wrapper.py", line 67, in _new_java_obj
> return java_obj(*java_args)
> File /spark-1/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line
> 1554, in __call__
> answer, self._gateway_client, None, self._fqn)
> File "/spark-1/python/pyspark/sql/utils.py", line 93, in deco
> raise converted
> pyspark.sql.utils.IllegalArgumentException: 'StopWordsRemover_4598673ee802
> parameter locale given invalid value en_TW.'
> {code}
> As per [~hyukjin.kwon]'s advice, instead of setting up locale to pass test,
> it is better to have a workable locale if system default locale can't be
> found in available locales in JVM. Otherwise, users have to manually change
> system locale or accessing a private property _jvm in PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28365) Fallback locale to en_US in StopWordsRemover if system default locale isn't in available locales in JVM
[
https://issues.apache.org/jira/browse/SPARK-28365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-28365:
Assignee: Liang-Chi Hsieh
> Fallback locale to en_US in StopWordsRemover if system default locale isn't
> in available locales in JVM
> ---
>
> Key: SPARK-28365
> URL: https://issues.apache.org/jira/browse/SPARK-28365
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 3.0.0
>Reporter: Liang-Chi Hsieh
>Assignee: Liang-Chi Hsieh
>Priority: Major
>
> Because the local default locale isn't in available locales at {{Locale}},
> when I did some tests locally with python code, {{StopWordsRemover}} related
> python test hits some errors, like:
> {code}
> Traceback (most recent call last):
> File "/spark-1/python/pyspark/ml/tests/test_feature.py", line 87, in
> test_stopwordsremover
> stopWordRemover = StopWordsRemover(inputCol="input", outputCol="output")
> File "/spark-1/python/pyspark/__init__.py", line 111, in wrapper
> return func(self, **kwargs)
> File "/spark-1/python/pyspark/ml/feature.py", line 2646, in __init__
> self.uid)
> File "/spark-1/python/pyspark/ml/wrapper.py", line 67, in _new_java_obj
> return java_obj(*java_args)
> File /spark-1/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line
> 1554, in __call__
> answer, self._gateway_client, None, self._fqn)
> File "/spark-1/python/pyspark/sql/utils.py", line 93, in deco
> raise converted
> pyspark.sql.utils.IllegalArgumentException: 'StopWordsRemover_4598673ee802
> parameter locale given invalid value en_TW.'
> {code}
> As per [~hyukjin.kwon]'s advice, instead of setting up locale to pass test,
> it is better to have a workable locale if system default locale can't be
> found in available locales in JVM. Otherwise, users have to manually change
> system locale or accessing a private property _jvm in PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28515) to_timestamp returns null for summer time switch dates
[
https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-28515:
-
Description:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{code:java}
>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
>>> ['date_str'])
>>> df.withColumn('timestamp', F.to_timestamp('date_str',
>>> 'MMddhhmm')).show()
-+
| date_str| timestamp|
-+
|201503290159|2015-03-29 01:59:00|
|201503290200| null|
-+ {code}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{code:java}
SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");
Date parsedDate = dateFormat.parse("201503290201");
Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());{code}
So, is this really the intended behaviour? Is there documentation about this?
THX.
was:
I am not sure if this is a bug - but it was a very unexpected behavior, so I'd
like some clarification.
When parsing datetime-strings, when the date-time in question falls into the
range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at 2am
the clock was forwarded to 3am), the {{to_timestamp}} method returns {{NULL}}.
Minimal Example (using Python):
{{>>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
['date_str'])}}
{{>>> df.withColumn('timestamp', F.to_timestamp('date_str',
'MMddhhmm')).show()}}
{{+-+--+}}
{{| date_str| timestamp|}}
{{+-+--+}}
{{|201503290159|2015-03-29 01:59:00|}}
{{|201503290200| null|}}
{{+-+--+}}
A solution (or workaround) is to set the time zone for Spark to UTC:
{{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
(see e.g. [https://stackoverflow.com/q/52594762)]
Plain Java does not do this, e.g. this works as expected:
{{SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");}}
{{Date parsedDate = dateFormat.parse("201503290201");}}
{{Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());}}
So, is this really the intended behaviour? Is there documentation about this?
THX.
> to_timestamp returns null for summer time switch dates
> --
>
> Key: SPARK-28515
> URL: https://issues.apache.org/jira/browse/SPARK-28515
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
>Reporter: Andreas Költringer
>Priority: Major
>
> I am not sure if this is a bug - but it was a very unexpected behavior, so
> I'd like some clarification.
> When parsing datetime-strings, when the date-time in question falls into the
> range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at
> 2am the clock was forwarded to 3am), the {{to_timestamp}} method returns
> {{NULL}}.
> Minimal Example (using Python):
> {code:java}
> >>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
> >>> ['date_str'])
> >>> df.withColumn('timestamp', F.to_timestamp('date_str',
> >>> 'MMddhhmm')).show()
> -+
> | date_str| timestamp|
> -+
> |201503290159|2015-03-29 01:59:00|
> |201503290200| null|
> -+ {code}
> A solution (or workaround) is to set the time zone for Spark to UTC:
> {{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
> (see e.g. [https://stackoverflow.com/q/52594762)]
> Plain Java does not do this, e.g. this works as expected:
>
> {code:java}
> SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");
> Date parsedDate = dateFormat.parse("201503290201");
> Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());{code}
> So, is this really the intended behaviour? Is there documentation about this?
> THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28515) to_timestamp returns null for summer time switch dates
[
https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893318#comment-16893318
]
Hyukjin Kwon commented on SPARK-28515:
--
Which Python version do you use? IIRC, Python 3.4 has an issue for the folded
time in DST.
> to_timestamp returns null for summer time switch dates
> --
>
> Key: SPARK-28515
> URL: https://issues.apache.org/jira/browse/SPARK-28515
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
>Reporter: Andreas Költringer
>Priority: Major
>
> I am not sure if this is a bug - but it was a very unexpected behavior, so
> I'd like some clarification.
> When parsing datetime-strings, when the date-time in question falls into the
> range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at
> 2am the clock was forwarded to 3am), the {{to_timestamp}} method returns
> {{NULL}}.
> Minimal Example (using Python):
> {code:java}
> >>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
> >>> ['date_str'])
> >>> df.withColumn('timestamp', F.to_timestamp('date_str',
> >>> 'MMddhhmm')).show()
> -+
> | date_str| timestamp|
> -+
> |201503290159|2015-03-29 01:59:00|
> |201503290200| null|
> -+ {code}
> A solution (or workaround) is to set the time zone for Spark to UTC:
> {{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
> (see e.g. [https://stackoverflow.com/q/52594762)]
> Plain Java does not do this, e.g. this works as expected:
>
> {code:java}
> SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");
> Date parsedDate = dateFormat.parse("201503290201");
> Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());{code}
> So, is this really the intended behaviour? Is there documentation about this?
> THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-28515) to_timestamp returns null for summer time switch dates
[
https://issues.apache.org/jira/browse/SPARK-28515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893318#comment-16893318
]
Hyukjin Kwon edited comment on SPARK-28515 at 7/26/19 4:29 AM:
---
Which Python version do you use? IIRC, Python 3.4 and Python 3.5 has an issue
for the folded time in DST.
was (Author: hyukjin.kwon):
Which Python version do you use? IIRC, Python 3.4 has an issue for the folded
time in DST.
> to_timestamp returns null for summer time switch dates
> --
>
> Key: SPARK-28515
> URL: https://issues.apache.org/jira/browse/SPARK-28515
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
> Environment: Spark 2.4.3 on Linux 64bit, openjdk-8-jre-headless
>Reporter: Andreas Költringer
>Priority: Major
>
> I am not sure if this is a bug - but it was a very unexpected behavior, so
> I'd like some clarification.
> When parsing datetime-strings, when the date-time in question falls into the
> range of a "summer time switch" (e.g. in (most of) Europe, on 2015-03-29 at
> 2am the clock was forwarded to 3am), the {{to_timestamp}} method returns
> {{NULL}}.
> Minimal Example (using Python):
> {code:java}
> >>> df = spark.createDataFrame([('201503290159',), ('201503290200',)],
> >>> ['date_str'])
> >>> df.withColumn('timestamp', F.to_timestamp('date_str',
> >>> 'MMddhhmm')).show()
> -+
> | date_str| timestamp|
> -+
> |201503290159|2015-03-29 01:59:00|
> |201503290200| null|
> -+ {code}
> A solution (or workaround) is to set the time zone for Spark to UTC:
> {{spark.conf.set("spark.sql.session.timeZone", "UTC")}}
> (see e.g. [https://stackoverflow.com/q/52594762)]
> Plain Java does not do this, e.g. this works as expected:
>
> {code:java}
> SimpleDateFormat dateFormat = new SimpleDateFormat("MMddhhmm");
> Date parsedDate = dateFormat.parse("201503290201");
> Timestamp timestamp = new java.sql.Timestamp(parsedDate.getTime());{code}
> So, is this really the intended behaviour? Is there documentation about this?
> THX.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures
[
https://issues.apache.org/jira/browse/SPARK-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28512:
--
Issue Type: Improvement (was: Bug)
> New optional mode: throw runtime exceptions on casting failures
> ---
>
> Key: SPARK-28512
> URL: https://issues.apache.org/jira/browse/SPARK-28512
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Priority: Major
>
> In popular DBMS like MySQL/PostgreSQL/Oracle, runtime exceptions are thrown
> on casting, e.g. cast('abc' as int)
> While in Spark, the result is converted as null silently. It is by design
> since we don't want a long-running job aborted by some casting failure. But
> there are scenarios that users want to make sure all the data conversion are
> correct, like the way they use MySQL/PostgreSQL/Oracle.
> If the changes touch too much code, we can limit the new optional mode to
> table insertion first. By default the new behavior is disabled.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures
[
https://issues.apache.org/jira/browse/SPARK-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893331#comment-16893331
]
Dongjoon Hyun commented on SPARK-28512:
---
I changed the type from `BUG` to `Improvement` since this is a `New optional
mode`.
> New optional mode: throw runtime exceptions on casting failures
> ---
>
> Key: SPARK-28512
> URL: https://issues.apache.org/jira/browse/SPARK-28512
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Priority: Major
>
> In popular DBMS like MySQL/PostgreSQL/Oracle, runtime exceptions are thrown
> on casting, e.g. cast('abc' as int)
> While in Spark, the result is converted as null silently. It is by design
> since we don't want a long-running job aborted by some casting failure. But
> there are scenarios that users want to make sure all the data conversion are
> correct, like the way they use MySQL/PostgreSQL/Oracle.
> If the changes touch too much code, we can limit the new optional mode to
> table insertion first. By default the new behavior is disabled.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures
[
https://issues.apache.org/jira/browse/SPARK-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893334#comment-16893334
]
Gengliang Wang commented on SPARK-28512:
[~dongjoon]Thanks!
> New optional mode: throw runtime exceptions on casting failures
> ---
>
> Key: SPARK-28512
> URL: https://issues.apache.org/jira/browse/SPARK-28512
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Priority: Major
>
> In popular DBMS like MySQL/PostgreSQL/Oracle, runtime exceptions are thrown
> on casting, e.g. cast('abc' as int)
> While in Spark, the result is converted as null silently. It is by design
> since we don't want a long-running job aborted by some casting failure. But
> there are scenarios that users want to make sure all the data conversion are
> correct, like the way they use MySQL/PostgreSQL/Oracle.
> If the changes touch too much code, we can limit the new optional mode to
> table insertion first. By default the new behavior is disabled.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25653) Add tag ExtendedHiveTest for HiveSparkSubmitSuite
[ https://issues.apache.org/jira/browse/SPARK-25653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25653: -- Priority: Minor (was: Major) > Add tag ExtendedHiveTest for HiveSparkSubmitSuite > - > > Key: SPARK-25653 > URL: https://issues.apache.org/jira/browse/SPARK-25653 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Minor > Fix For: 3.0.0 > > > The total run time of HiveSparkSubmitSuite is about 10 minutes. > While the related code is stable, add tag ExtendedHiveTest for it. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28518) Only filter out the checksum file when enabling fall back to HDFS
Yuming Wang created SPARK-28518: --- Summary: Only filter out the checksum file when enabling fall back to HDFS Key: SPARK-28518 URL: https://issues.apache.org/jira/browse/SPARK-28518 Project: Spark Issue Type: Bug Components: SQL, Tests Affects Versions: 3.0.0 Reporter: Yuming Wang -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25653) Add tag ExtendedHiveTest for HiveSparkSubmitSuite
[ https://issues.apache.org/jira/browse/SPARK-25653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25653: -- Affects Version/s: (was: 2.4.0) 3.0.0 > Add tag ExtendedHiveTest for HiveSparkSubmitSuite > - > > Key: SPARK-25653 > URL: https://issues.apache.org/jira/browse/SPARK-25653 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Fix For: 3.0.0 > > > The total run time of HiveSparkSubmitSuite is about 10 minutes. > While the related code is stable, add tag ExtendedHiveTest for it. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28518) Only filter out the checksum file when enabling fall back to HDFS
[ https://issues.apache.org/jira/browse/SPARK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28518: Description: StatisticsCollectionTestBase.getDataSize is incorrect. We should filter out the checksum file with reference to ChecksumFileSystem.isChecksumFile. https://github.com/apache/spark/pull/25014#discussion_r307050435 > Only filter out the checksum file when enabling fall back to HDFS > - > > Key: SPARK-28518 > URL: https://issues.apache.org/jira/browse/SPARK-28518 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > StatisticsCollectionTestBase.getDataSize is incorrect. We should filter out > the checksum file with reference to ChecksumFileSystem.isChecksumFile. > https://github.com/apache/spark/pull/25014#discussion_r307050435 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28518) Fix StatisticsCollectionTestBase#getDataSize
[ https://issues.apache.org/jira/browse/SPARK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28518: Summary: Fix StatisticsCollectionTestBase#getDataSize (was: Only filter out the checksum file when enabling fall back to HDFS) > Fix StatisticsCollectionTestBase#getDataSize > > > Key: SPARK-28518 > URL: https://issues.apache.org/jira/browse/SPARK-28518 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > StatisticsCollectionTestBase.getDataSize is incorrect. We should filter out > the checksum file with reference to ChecksumFileSystem.isChecksumFile. > https://github.com/apache/spark/pull/25014#discussion_r307050435 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28518) Fix StatisticsCollectionTestBase#getDataSize refer to ChecksumFileSystem.isChecksumFile
[ https://issues.apache.org/jira/browse/SPARK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28518: Summary: Fix StatisticsCollectionTestBase#getDataSize refer to ChecksumFileSystem.isChecksumFile (was: Fix StatisticsCollectionTestBase#getDataSize) > Fix StatisticsCollectionTestBase#getDataSize refer to > ChecksumFileSystem.isChecksumFile > --- > > Key: SPARK-28518 > URL: https://issues.apache.org/jira/browse/SPARK-28518 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > StatisticsCollectionTestBase.getDataSize is incorrect. We should filter out > the checksum file with reference to ChecksumFileSystem.isChecksumFile. > https://github.com/apache/spark/pull/25014#discussion_r307050435 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28518) Fix StatisticsCollectionTestBase#getDataSize refer to ChecksumFileSystem.isChecksumFile
[ https://issues.apache.org/jira/browse/SPARK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28518: Description: StatisticsCollectionTestBase.getDataSize is incorrect. We should refer to ChecksumFileSystem.isChecksumFile. https://github.com/apache/spark/pull/25014#discussion_r307050435 was: StatisticsCollectionTestBase.getDataSize is incorrect. We should filter out the checksum file with reference to ChecksumFileSystem.isChecksumFile. https://github.com/apache/spark/pull/25014#discussion_r307050435 > Fix StatisticsCollectionTestBase#getDataSize refer to > ChecksumFileSystem.isChecksumFile > --- > > Key: SPARK-28518 > URL: https://issues.apache.org/jira/browse/SPARK-28518 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > StatisticsCollectionTestBase.getDataSize is incorrect. We should refer to > ChecksumFileSystem.isChecksumFile. > https://github.com/apache/spark/pull/25014#discussion_r307050435 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28518) Fix StatisticsCollectionTestBase#getDataSize refer to ChecksumFileSystem#isChecksumFile
[ https://issues.apache.org/jira/browse/SPARK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28518: Summary: Fix StatisticsCollectionTestBase#getDataSize refer to ChecksumFileSystem#isChecksumFile (was: Fix StatisticsCollectionTestBase#getDataSize refer to ChecksumFileSystem.isChecksumFile) > Fix StatisticsCollectionTestBase#getDataSize refer to > ChecksumFileSystem#isChecksumFile > --- > > Key: SPARK-28518 > URL: https://issues.apache.org/jira/browse/SPARK-28518 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > StatisticsCollectionTestBase.getDataSize is incorrect. We should refer to > ChecksumFileSystem.isChecksumFile. > https://github.com/apache/spark/pull/25014#discussion_r307050435 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28237) Idempotence checker for Idempotent batches in RuleExecutors
[
https://issues.apache.org/jira/browse/SPARK-28237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li resolved SPARK-28237.
-
Resolution: Fixed
Fix Version/s: 3.0.0
> Idempotence checker for Idempotent batches in RuleExecutors
> ---
>
> Key: SPARK-28237
> URL: https://issues.apache.org/jira/browse/SPARK-28237
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yesheng Ma
>Priority: Major
> Fix For: 3.0.0
>
>
> The current {{RuleExecutor}} system contains two kinds of strategies:
> {{Once}} and {{FixedPoint}}. The {{Once}} strategy is supposed to run once.
> However, for particular rules (e.g. PullOutNondeterministic), they are
> designed to be idempotent, but Spark currently lacks corresponding mechanism
> to prevent such kind of non-idempotent behavior from happening.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28237) Idempotence checker for Idempotent batches in RuleExecutors
[
https://issues.apache.org/jira/browse/SPARK-28237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li reassigned SPARK-28237:
---
Assignee: Yesheng Ma
> Idempotence checker for Idempotent batches in RuleExecutors
> ---
>
> Key: SPARK-28237
> URL: https://issues.apache.org/jira/browse/SPARK-28237
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yesheng Ma
>Assignee: Yesheng Ma
>Priority: Major
> Fix For: 3.0.0
>
>
> The current {{RuleExecutor}} system contains two kinds of strategies:
> {{Once}} and {{FixedPoint}}. The {{Once}} strategy is supposed to run once.
> However, for particular rules (e.g. PullOutNondeterministic), they are
> designed to be idempotent, but Spark currently lacks corresponding mechanism
> to prevent such kind of non-idempotent behavior from happening.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27714) Support Join Reorder based on Genetic Algorithm when the # of joined tables > 12
[ https://issues.apache.org/jira/browse/SPARK-27714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-27714: Target Version/s: 3.0.0 > Support Join Reorder based on Genetic Algorithm when the # of joined tables > > 12 > > > Key: SPARK-27714 > URL: https://issues.apache.org/jira/browse/SPARK-27714 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Xianyin Xin >Priority: Major > > Now the join reorder logic is based on dynamic planning which can find the > most optimized plan theoretically, but the searching cost grows rapidly with > the # of joined tables grows. It would be better to introduce Genetic > algorithm (GA) to overcome this problem. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-27714) Support Join Reorder based on Genetic Algorithm when the # of joined tables > 12
[ https://issues.apache.org/jira/browse/SPARK-27714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-27714: --- Assignee: Xianyin Xin > Support Join Reorder based on Genetic Algorithm when the # of joined tables > > 12 > > > Key: SPARK-27714 > URL: https://issues.apache.org/jira/browse/SPARK-27714 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Xianyin Xin >Assignee: Xianyin Xin >Priority: Major > > Now the join reorder logic is based on dynamic planning which can find the > most optimized plan theoretically, but the searching cost grows rapidly with > the # of joined tables grows. It would be better to introduce Genetic > algorithm (GA) to overcome this problem. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
