[jira] [Created] (SPARK-33557) spark.storage.blockManagerSlaveTimeoutMs default value does not follow spark.network.timeout value when the latter was changed
Ohad created SPARK-33557:
Summary: spark.storage.blockManagerSlaveTimeoutMs default value
does not follow spark.network.timeout value when the latter was changed
Key: SPARK-33557
URL: https://issues.apache.org/jira/browse/SPARK-33557
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.0.1, 3.0.0
Reporter: Ohad
According to the documentation "spark.network.timeout" is the default timeout
for "spark.storage.blockManagerSlaveTimeoutMs" which implies that when the user
sets "spark.network.timeout" the effective value of
"spark.storage.blockManagerSlaveTimeoutMs" should also be changed if it was not
specifically changed.
However this is not the case since the default value of
"spark.storage.blockManagerSlaveTimeoutMs" is always the default value of
"spark.network.timeout" (120s)
"spark.storage.blockManagerSlaveTimeoutMs" is defined in the package object of
"org.apache.spark.internal.config" as follows:
{code:java}
private[spark] val STORAGE_BLOCKMANAGER_SLAVE_TIMEOUT =
ConfigBuilder("spark.storage.blockManagerSlaveTimeoutMs")
.version("0.7.0")
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString(Network.NETWORK_TIMEOUT.defaultValueString)
{code}
So it seems like the its default value is indeed "fixed" to
"spark.network.timeout" default value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33556) Add array_to_vector function for dataframe column
[ https://issues.apache.org/jira/browse/SPARK-33556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238673#comment-17238673 ] Apache Spark commented on SPARK-33556: -- User 'WeichenXu123' has created a pull request for this issue: https://github.com/apache/spark/pull/30498 > Add array_to_vector function for dataframe column > - > > Key: SPARK-33556 > URL: https://issues.apache.org/jira/browse/SPARK-33556 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.1.0 >Reporter: Weichen Xu >Priority: Major > > Add array_to_vector function for dataframe column. > Input column type: Array > Output column type: apache.spark.ml.linalg.Vector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33556) Add array_to_vector function for dataframe column
[ https://issues.apache.org/jira/browse/SPARK-33556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33556: Assignee: Apache Spark > Add array_to_vector function for dataframe column > - > > Key: SPARK-33556 > URL: https://issues.apache.org/jira/browse/SPARK-33556 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.1.0 >Reporter: Weichen Xu >Assignee: Apache Spark >Priority: Major > > Add array_to_vector function for dataframe column. > Input column type: Array > Output column type: apache.spark.ml.linalg.Vector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33556) Add array_to_vector function for dataframe column
[ https://issues.apache.org/jira/browse/SPARK-33556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33556: Assignee: (was: Apache Spark) > Add array_to_vector function for dataframe column > - > > Key: SPARK-33556 > URL: https://issues.apache.org/jira/browse/SPARK-33556 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.1.0 >Reporter: Weichen Xu >Priority: Major > > Add array_to_vector function for dataframe column. > Input column type: Array > Output column type: apache.spark.ml.linalg.Vector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33556) Add array_to_vector function for dataframe column
[ https://issues.apache.org/jira/browse/SPARK-33556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238672#comment-17238672 ] Apache Spark commented on SPARK-33556: -- User 'WeichenXu123' has created a pull request for this issue: https://github.com/apache/spark/pull/30498 > Add array_to_vector function for dataframe column > - > > Key: SPARK-33556 > URL: https://issues.apache.org/jira/browse/SPARK-33556 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.1.0 >Reporter: Weichen Xu >Priority: Major > > Add array_to_vector function for dataframe column. > Input column type: Array > Output column type: apache.spark.ml.linalg.Vector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-33556) Add array_to_vector function for dataframe column
Weichen Xu created SPARK-33556: -- Summary: Add array_to_vector function for dataframe column Key: SPARK-33556 URL: https://issues.apache.org/jira/browse/SPARK-33556 Project: Spark Issue Type: New Feature Components: ML Affects Versions: 3.1.0 Reporter: Weichen Xu Add array_to_vector function for dataframe column. Input column type: Array Output column type: apache.spark.ml.linalg.Vector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28064) Order by does not accept a call to rank()
[
https://issues.apache.org/jira/browse/SPARK-28064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238662#comment-17238662
]
Dylan Guedes commented on SPARK-28064:
--
Sorry, my only intention was to help to map the differences between PostgreSQL
and SparkSQL APIs.
> Order by does not accept a call to rank()
> -
>
> Key: SPARK-28064
> URL: https://issues.apache.org/jira/browse/SPARK-28064
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Currently in Spark, we can't use a call to `rank()` in a order by; we need to
> first rename the rank column to, for instance, `r` and then, use `order by
> r`. For example:
> This does not work:
> {code:sql}
> SELECT depname, empno, salary, rank() OVER w FROM empsalary WINDOW w AS
> (PARTITION BY depname ORDER BY salary) ORDER BY rank() OVER w;
> {code}
> However, this one does:
> {code:sql}
> SELECT depname, empno, salary, rank() OVER w as r FROM empsalary WINDOW w AS
> (PARTITION BY depname ORDER BY salary) ORDER BY r;
> {code}
> By the way, I took this one from Postgres behavior: postgres accept both ways.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28648) Adds support to `groups` unit type in window clauses
[ https://issues.apache.org/jira/browse/SPARK-28648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238654#comment-17238654 ] jiaan.geng commented on SPARK-28648: it seems groups only exists in PgSQL > Adds support to `groups` unit type in window clauses > > > Key: SPARK-28648 > URL: https://issues.apache.org/jira/browse/SPARK-28648 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Dylan Guedes >Priority: Major > > Spark currently support the two most common window functions unit types: rows > and ranges. However, in PgSQL a new type was added: `groups`. > According to [this > source|https://blog.jooq.org/2018/07/05/postgresql-11s-support-for-sql-standard-groups-and-exclude-window-function-clauses/], > the difference is: > """ROWS counts the exact number of rows in the frame. > RANGE performs logical windowing where we don’t count the number of rows, but > look for a value offset. > GROUPS counts all groups of tied rows within the window.""" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28645) Throw an error on window redefinition
[
https://issues.apache.org/jira/browse/SPARK-28645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238648#comment-17238648
]
jiaan.geng commented on SPARK-28645:
I'm working on!
> Throw an error on window redefinition
> -
>
> Key: SPARK-28645
> URL: https://issues.apache.org/jira/browse/SPARK-28645
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Currently in Spark one could redefine a window. For instance:
> {code:sql}select count(*) OVER w FROM tenk1 WINDOW w AS (ORDER BY unique1), w
> AS (ORDER BY unique1);{code}
> The window `w` is defined two times. In PgSQL, on the other hand, a thrown
> will happen:
> {code:sql}ERROR: window "w" is already defined{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28508) Support for range frame+row frame in the same query
[
https://issues.apache.org/jira/browse/SPARK-28508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiaan.geng resolved SPARK-28508.
Resolution: Invalid
> Support for range frame+row frame in the same query
> ---
>
> Key: SPARK-28508
> URL: https://issues.apache.org/jira/browse/SPARK-28508
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Currently, looks like some queries does not works if both, a range frame and
> a row frame are given. However, PgSQL is able to handle them:
> {code:sql}
> select last(salary) over(order by enroll_date range between 1 preceding and 1
> following), lag(salary) over(order by enroll_date range between 1 preceding
> and 1 following),
> salary, enroll_date from empsalary;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28508) Support for range frame+row frame in the same query
[
https://issues.apache.org/jira/browse/SPARK-28508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238638#comment-17238638
]
jiaan.geng commented on SPARK-28508:
Spark has supported this feature.
> Support for range frame+row frame in the same query
> ---
>
> Key: SPARK-28508
> URL: https://issues.apache.org/jira/browse/SPARK-28508
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Currently, looks like some queries does not works if both, a range frame and
> a row frame are given. However, PgSQL is able to handle them:
> {code:sql}
> select last(salary) over(order by enroll_date range between 1 preceding and 1
> following), lag(salary) over(order by enroll_date range between 1 preceding
> and 1 following),
> salary, enroll_date from empsalary;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33540) Subexpression elimination for interpreted predicate
[ https://issues.apache.org/jira/browse/SPARK-33540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33540: Assignee: L. C. Hsieh (was: Apache Spark) > Subexpression elimination for interpreted predicate > --- > > Key: SPARK-33540 > URL: https://issues.apache.org/jira/browse/SPARK-33540 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 3.1.0 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > > We can support subexpression elimination for interpreted predicate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33540) Subexpression elimination for interpreted predicate
[ https://issues.apache.org/jira/browse/SPARK-33540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33540: Assignee: Apache Spark (was: L. C. Hsieh) > Subexpression elimination for interpreted predicate > --- > > Key: SPARK-33540 > URL: https://issues.apache.org/jira/browse/SPARK-33540 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 3.1.0 >Reporter: L. C. Hsieh >Assignee: Apache Spark >Priority: Major > > We can support subexpression elimination for interpreted predicate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33540) Subexpression elimination for interpreted predicate
[ https://issues.apache.org/jira/browse/SPARK-33540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238626#comment-17238626 ] Apache Spark commented on SPARK-33540: -- User 'viirya' has created a pull request for this issue: https://github.com/apache/spark/pull/30497 > Subexpression elimination for interpreted predicate > --- > > Key: SPARK-33540 > URL: https://issues.apache.org/jira/browse/SPARK-33540 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 3.1.0 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > > We can support subexpression elimination for interpreted predicate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28064) Order by does not accept a call to rank()
[
https://issues.apache.org/jira/browse/SPARK-28064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238625#comment-17238625
]
jiaan.geng commented on SPARK-28064:
Yeah! But it seems nothing useful.
> Order by does not accept a call to rank()
> -
>
> Key: SPARK-28064
> URL: https://issues.apache.org/jira/browse/SPARK-28064
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Currently in Spark, we can't use a call to `rank()` in a order by; we need to
> first rename the rank column to, for instance, `r` and then, use `order by
> r`. For example:
> This does not work:
> {code:sql}
> SELECT depname, empno, salary, rank() OVER w FROM empsalary WINDOW w AS
> (PARTITION BY depname ORDER BY salary) ORDER BY rank() OVER w;
> {code}
> However, this one does:
> {code:sql}
> SELECT depname, empno, salary, rank() OVER w as r FROM empsalary WINDOW w AS
> (PARTITION BY depname ORDER BY salary) ORDER BY r;
> {code}
> By the way, I took this one from Postgres behavior: postgres accept both ways.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32223) Support adding a user provided config map.
[ https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-32223: Description: One of the challenge with this is, spark.properties is not user provided and is calculated based on certain factors. So a user provided config map, cannot be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented with the correct spark.properties. Q, Do we support update to config map properties for an already running job? Ans: No, since the spark.properties is calculated at the time of job submission, it cannot be updated on the fly and it is not supported by Spark at the moment for all the configuration values. Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps? One of the use case, I can think of is programmatically submitting a `spark on k8s` job - e.g. spark as a service on a cloud deployment may find this feature useful. was: One of the challenge with this is, spark.properties is not user provided and is calculated based on certain factors. So a user provided config map, cannot be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented with the correct spark.properties. Q, Do we support update to config map properties for an already running job? Ans: No, since the spark.properties is calculated at the time of job submission, it cannot be updated on the fly and it is not supported by Spark at the moment for all the configuration values. Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps? One of the use case, I can think of is programmatically submitting a spark on k8s job - e.g. spark as a service on cloud deployment may find this feature useful. > Support adding a user provided config map. > -- > > Key: SPARK-32223 > URL: https://issues.apache.org/jira/browse/SPARK-32223 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Priority: Major > > One of the challenge with this is, spark.properties is not user provided and > is calculated based on certain factors. So a user provided config map, cannot > be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow > augmented with the correct spark.properties. > Q, Do we support update to config map properties for an already running job? > Ans: No, since the spark.properties is calculated at the time of job > submission, it cannot be updated on the fly and it is not supported by Spark > at the moment for all the configuration values. > Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map > helps? > One of the use case, I can think of is programmatically submitting a `spark > on k8s` job - e.g. spark as a service on a cloud deployment may find this > feature useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32223) Support adding a user provided config map.
[ https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238617#comment-17238617 ] Prashant Sharma commented on SPARK-32223: - Hi [~dongjoon], Do you think this is useful ? Any other thoughts on this? Thanks ! > Support adding a user provided config map. > -- > > Key: SPARK-32223 > URL: https://issues.apache.org/jira/browse/SPARK-32223 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Priority: Major > > One of the challenge with this is, spark.properties is not user provided and > is calculated based on certain factors. So a user provided config map, cannot > be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow > augmented with the correct spark.properties. > Q, Do we support update to config map properties for an already running job? > Ans: No, since the spark.properties is calculated at the time of job > submission, it cannot be updated on the fly and it is not supported by Spark > at the moment for all the configuration values. > Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map > helps? > One of the use case, I can think of is programmatically submitting a spark on > k8s job - e.g. spark as a service on cloud deployment may find this feature > useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32223) Support adding a user provided config map.
[ https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-32223: Description: One of the challenge with this is, spark.properties is not user provided and is calculated based on certain factors. So a user provided config map, cannot be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented with the correct spark.properties. Q, Do we support update to config map properties for an already running job? Ans: No, since the spark.properties is calculated at the time of job submission, it cannot be updated on the fly and it is not supported by Spark at the moment for all the configuration values. Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps? One of the use case, I can think of is programmatically submitting a spark on k8s job - e.g. spark as a service on cloud deployment may find this feature useful. was:The semantics of this will be discussed and added soon. > Support adding a user provided config map. > -- > > Key: SPARK-32223 > URL: https://issues.apache.org/jira/browse/SPARK-32223 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Priority: Major > > One of the challenge with this is, spark.properties is not user provided and > is calculated based on certain factors. So a user provided config map, cannot > be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow > augmented with the correct spark.properties. > Q, Do we support update to config map properties for an already running job? > Ans: No, since the spark.properties is calculated at the time of job > submission, it cannot be updated on the fly and it is not supported by Spark > at the moment for all the configuration values. > Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map > helps? > One of the use case, I can think of is programmatically submitting a spark on > k8s job - e.g. spark as a service on cloud deployment may find this feature > useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33429) Support drop column in spark also like in postgresql
[
https://issues.apache.org/jira/browse/SPARK-33429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238615#comment-17238615
]
jiaan.geng commented on SPARK-33429:
It seems Spark has supported
{code:java}
| ALTER TABLE multipartIdentifier
DROP (COLUMN | COLUMNS)
'(' columns=multipartIdentifierList ')'
#dropTableColumns
| ALTER TABLE multipartIdentifier
DROP (COLUMN | COLUMNS) columns=multipartIdentifierList
#dropTableColumns
{code}
> Support drop column in spark also like in postgresql
>
>
> Key: SPARK-33429
> URL: https://issues.apache.org/jira/browse/SPARK-33429
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 2.4.5
>Reporter: jobit mathew
>Priority: Minor
>
> Support alter table with drop column in spark also like in postgresql.
>
> [https://www.postgresql.org/docs/12/sql-altertable.html]
> alter table tablename DROP COLUMN/columns [ IF EXISTS ]
> columnname
>
> {code:java}
> spark-sql> drop database if exists hivemetastoretest cascade;
> Time taken: 1.067 seconds
> spark-sql> create database hivemetastoretest;
> Time taken: 0.326 seconds
> spark-sql> use hivemetastoretest;
> Time taken: 0.053 seconds
> spark-sql> create table jobit4 using parquet as select 2.5;
> Time taken: 5.058 seconds
> spark-sql> alter table jobit4 add columns(name string);
> Time taken: 1.194 seconds
> spark-sql> alter table jobit4 drop columns(name);
> Error in query:
> mismatched input 'columns' expecting \{'PARTITION', 'IF'}(line 1, pos 25)
> == SQL ==
> alter table jobit4 drop columns(name)
> -^^^
> spark-sql> alter table jobit4 drop columns name;
> Error in query:
> mismatched input 'columns' expecting \{'PARTITION', 'IF'}(line 1, pos 25)
> == SQL ==
> alter table jobit4 drop columns name
> -^^^
> spark-sql> [
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33527) Extend the function of decode so as consistent with mainstream databases
[
https://issues.apache.org/jira/browse/SPARK-33527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiaan.geng updated SPARK-33527:
---
Issue Type: New Feature (was: Bug)
> Extend the function of decode so as consistent with mainstream databases
>
>
> Key: SPARK-33527
> URL: https://issues.apache.org/jira/browse/SPARK-33527
> Project: Spark
> Issue Type: New Feature
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> In Spark, decode(bin, charset) - Decodes the first argument using the second
> argument character set.
> Unfortunately this is NOT what any other SQL vendor understands DECODE to do.
> DECODE generally is a short hand for a simple case expression:
> {code:java}
> SELECT DECODE(c1, 1, 'Hello', 2, 'World', '!') FROM (VALUES (1), (2), (3)) AS
> T(c1)
> =>
> (Hello),
> (World)
> (!)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-33549) Remove configuration spark.sql.legacy.allowCastNumericToTimestamp
[ https://issues.apache.org/jira/browse/SPARK-33549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33549. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30493 [https://github.com/apache/spark/pull/30493] > Remove configuration spark.sql.legacy.allowCastNumericToTimestamp > - > > Key: SPARK-33549 > URL: https://issues.apache.org/jira/browse/SPARK-33549 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Minor > Fix For: 3.1.0 > > > In the current master branch, there is a new configuration > `spark.sql.legacy.allowCastNumericToTimestamp` which controls whether to cast > Numeric types to Timestamp or not. The default value is true. > After https://github.com/apache/spark/pull/30260, the type conversion between > Timestamp type and Numeric type is disallowed in ANSI mode. So, we don't need > to a separate configuration `spark.sql.legacy.allowCastNumericToTimestamp` > for disallowing the conversion. > We should remove the configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-33555) Accessibility: spark history server page, the download button has insufficient 2.2:1 contrast .
danliu created SPARK-33555: -- Summary: Accessibility: spark history server page, the download button has insufficient 2.2:1 contrast . Key: SPARK-33555 URL: https://issues.apache.org/jira/browse/SPARK-33555 Project: Spark Issue Type: Bug Components: Graph Affects Versions: 2.4.5 Reporter: danliu In spark history server page, the download button has insufficient 2.2:1 contrast .Visual impaired user will find difficulty to see the control or element if that control or element have insufficient color contrast as user won't be able to perform functionality over such control. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-33554) Accessibility: insufficient color contrast
danliu created SPARK-33554: -- Summary: Accessibility: insufficient color contrast Key: SPARK-33554 URL: https://issues.apache.org/jira/browse/SPARK-33554 Project: Spark Issue Type: Bug Components: Graph Affects Versions: 2.4.5 Reporter: danliu Visual impaired user will find difficulty to see the control or element in spark history server page if that control or element have insufficient color contrast as user won't be able to perform functionality over such control. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-33553) Accessibility: "Spark 2.4.5" image link is not defined
danliu created SPARK-33553: -- Summary: Accessibility: "Spark 2.4.5" image link is not defined Key: SPARK-33553 URL: https://issues.apache.org/jira/browse/SPARK-33553 Project: Spark Issue Type: Bug Components: Graph Affects Versions: 2.4.5 Reporter: danliu In spark history server page, screen reader user will get confuse with the purpose of the image link of the "Spark 2.4.5" and can not access the image link efficiently if alt of the image is not defined. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-33552) Accessibility: in spark history server page, The element does not have a lang attribute, user depending on Assistive Technology get affected as lang attribute.
danliu created SPARK-33552: -- Summary: Accessibility: in spark history server page, The element does not have a lang attribute, user depending on Assistive Technology get affected as lang attribute. Key: SPARK-33552 URL: https://issues.apache.org/jira/browse/SPARK-33552 Project: Spark Issue Type: Bug Components: Graph Affects Versions: 3.0.1 Reporter: danliu in spark history server page, The elements do not have a lang attribute, user depending on Assistive Technology get affected as lang attribute. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33547) Doc Type Construct Literal usage
[ https://issues.apache.org/jira/browse/SPARK-33547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33547: Assignee: Apache Spark > Doc Type Construct Literal usage > > > Key: SPARK-33547 > URL: https://issues.apache.org/jira/browse/SPARK-33547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.1.0 >Reporter: angerszhu >Assignee: Apache Spark >Priority: Major > > Add Doc about type construct literal in > [https://spark.apache.org/docs/3.0.1/sql-ref-literals.html] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33547) Doc Type Construct Literal usage
[ https://issues.apache.org/jira/browse/SPARK-33547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33547: Assignee: (was: Apache Spark) > Doc Type Construct Literal usage > > > Key: SPARK-33547 > URL: https://issues.apache.org/jira/browse/SPARK-33547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.1.0 >Reporter: angerszhu >Priority: Major > > Add Doc about type construct literal in > [https://spark.apache.org/docs/3.0.1/sql-ref-literals.html] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33547) Doc Type Construct Literal usage
[ https://issues.apache.org/jira/browse/SPARK-33547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238588#comment-17238588 ] Apache Spark commented on SPARK-33547: -- User 'AngersZh' has created a pull request for this issue: https://github.com/apache/spark/pull/30496 > Doc Type Construct Literal usage > > > Key: SPARK-33547 > URL: https://issues.apache.org/jira/browse/SPARK-33547 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.1.0 >Reporter: angerszhu >Priority: Major > > Add Doc about type construct literal in > [https://spark.apache.org/docs/3.0.1/sql-ref-literals.html] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
