[jira] [Commented] (SPARK-23897) Guava version
[
https://issues.apache.org/jira/browse/SPARK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069352#comment-17069352
]
Jorge Machado commented on SPARK-23897:
---
I think that master is actually broken at least for commit
d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e. I builded with:
{code:java}
./build/mvn clean package -DskipTests -Phadoop-3.2 -Pkubernetes -Phadoop-cloud
{code}
I get:
{code:java}
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin
[10:43:24]jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin
[10:43:24]> $ java -version
[±master ✓]java version "1.8.0_211"Java(TM) SE
Runtime Environment (build 1.8.0_211-b12)Java HotSpot(TM) 64-Bit Server VM
(build 25.211-b12, mixed mode)
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin
[10:43:27]> $ ./run-example SparkPi 100
[±master ✓]Exception in thread "main"
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at
org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
at
org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:427)
at
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
at scala.Option.getOrElse(Option.scala:189) at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
> Guava version
> -
>
> Key: SPARK-23897
> URL: https://issues.apache.org/jira/browse/SPARK-23897
> Project: Spark
> Issue Type: Dependency upgrade
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Sercan Karaoglu
>Priority: Minor
>
> Guava dependency version 14 is pretty old, needs to be updated to at least
> 16, google cloud storage connector uses newer one which causes pretty popular
> error with guava; "java.lang.NoSuchMethodError:
> com.google.common.base.Splitter.splitToList(Ljava/lang/CharSequence;)Ljava/util/List;"
> and causes app to crash
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-23897) Guava version
[
https://issues.apache.org/jira/browse/SPARK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069352#comment-17069352
]
Jorge Machado edited comment on SPARK-23897 at 3/28/20, 9:50 AM:
-
I think that master is actually broken at least for commit
d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e. I builded with:
{code:java}
./build/mvn clean package -DskipTests -Phadoop-3.2 -Pkubernetes -Phadoop-cloud
{code}
I get:
{code:java}
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin
[10:43:24]jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin
[10:43:24]> $ java -version
[±master ✓]java version "1.8.0_211"Java(TM) SE
Runtime Environment (build 1.8.0_211-b12)Java HotSpot(TM) 64-Bit Server VM
(build 25.211-b12, mixed mode) jorge@Jorges-MacBook-Pro
~/Downloads/spark/dist/bin
[10:43:27]> $
./run-example SparkPi 100
[±master ✓]Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at
org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
at
org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:427)
at
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
at scala.Option.getOrElse(Option.scala:189) at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala){code}
If I delete guava 14 and add guava28 it works.
was (Author: jomach):
I think that master is actually broken at least for commit
d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e. I builded with:
{code:java}
./build/mvn clean package -DskipTests -Phadoop-3.2 -Pkubernetes -Phadoop-cloud
{code}
I get:
{code:java}
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin
[10:43:24]jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin
[10:43:24]> $ java -version
[±master ✓]java version "1.8.0_211"Java(TM) SE
Runtime Environment (build 1.8.0_211-b12)Java HotSpot(TM) 64-Bit Server VM
(build 25.211-b12, mixed mode)
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin
[10:43:27]> $ ./run-example SparkPi 100
[jira] [Created] (SPARK-31292) Replace toSet.toSeq with distinct for readability
Kengo Seki created SPARK-31292:
--
Summary: Replace toSet.toSeq with distinct for readability
Key: SPARK-31292
URL: https://issues.apache.org/jira/browse/SPARK-31292
Project: Spark
Issue Type: Improvement
Components: Spark Core, SQL
Affects Versions: 3.0.0
Reporter: Kengo Seki
Some classes use {{toSet.toSeq}} to make the elements of a sequence unique, but
it's a bit verbose. Using {{distinct}} instead is easier to understand.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-31293) Fix wrong examples and help messages for Kinesis integration
Kengo Seki created SPARK-31293:
--
Summary: Fix wrong examples and help messages for Kinesis
integration
Key: SPARK-31293
URL: https://issues.apache.org/jira/browse/SPARK-31293
Project: Spark
Issue Type: Improvement
Components: Documentation, DStreams
Affects Versions: 3.0.0
Reporter: Kengo Seki
There are some minor mistakes in the examples and the help messages for Kinesis
integration. For example, {{KinesisWordCountASL.scala}} takes three arguments
but its example is taking four, while {{kinesis_wordcount_asl.py}} takes four
but its example is taking three.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30272) Remove usage of Guava that breaks in Guava 27
[
https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069933#comment-17069933
]
Jorge Machado commented on SPARK-30272:
---
Hey Sean,
This seems still to make problems for example:
{code:java}
java.lang.NoClassDefFoundError:
com/google/common/util/concurrent/internal/InternalFutureFailureAccess
java.lang.NoClassDefFoundError:
com/google/common/util/concurrent/internal/InternalFutureFailureAccess at
java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:757) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at
java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at
java.net.URLClassLoader.access$100(URLClassLoader.java:74) at
java.net.URLClassLoader$1.run(URLClassLoader.java:369) at
java.net.URLClassLoader$1.run(URLClassLoader.java:363) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:362) at
java.lang.ClassLoader.loadClass(ClassLoader.java:419) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at
java.lang.ClassLoader.loadClass(ClassLoader.java:352) at
java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:757) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at
java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at
java.net.URLClassLoader.access$100(URLClassLoader.java:74) at
java.net.URLClassLoader$1.run(URLClassLoader.java:369) at
java.net.URLClassLoader$1.run(URLClassLoader.java:363) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:362) at
java.lang.ClassLoader.loadClass(ClassLoader.java:419) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at
java.lang.ClassLoader.loadClass(ClassLoader.java:352) at
java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:757) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at
java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at
java.net.URLClassLoader.access$100(URLClassLoader.java:74) at
java.net.URLClassLoader$1.run(URLClassLoader.java:369) at
java.net.URLClassLoader$1.run(URLClassLoader.java:363) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:362) at
java.lang.ClassLoader.loadClass(ClassLoader.java:419) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at
java.lang.ClassLoader.loadClass(ClassLoader.java:352) at
com.google.common.cache.LocalCache$LoadingValueReference.(LocalCache.java:3472)
at
com.google.common.cache.LocalCache$LoadingValueReference.(LocalCache.java:3476)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2134)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) at
com.google.common.cache.LocalCache.get(LocalCache.java:3951) at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:228) at
org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1588)
at
org.apache.hadoop.security.UserGroupInformation.getPrimaryGroupName(UserGroupInformation.java:1453)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:147)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:104)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at
org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:522)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:491)
at
org.apache.spark.SparkContext.$anonfun$newAPIHadoopFile$2(SparkContext.scala:1219)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:757) at
org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:1207) at
org.apache.spark.api.java.JavaSparkContext.newAPIHadoopFile(JavaSparkContext.scala:484)
{code}
I still see a lot of references to guava 14 on master is this normal ? Sorry
for the question...
> Remove usag
[jira] [Comment Edited] (SPARK-30272) Remove usage of Guava that breaks in Guava 27
[
https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069933#comment-17069933
]
Jorge Machado edited comment on SPARK-30272 at 3/28/20, 4:25 PM:
-
Hey Sean,
This seems still to make problems for example:
{code:java}
> $ ./bin/run-example SparkPi 100
>
>
> [±master ●]> $ ./bin/run-example SparkPi 100
>
>
> [±master ●]20/03/28 17:21:13 WARN Utils: Your hostname,
> Jorges-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using
> 192.168.1.2 instead (on interface en0)20/03/28 17:21:13 WARN Utils: Set
> SPARK_LOCAL_IP if you need to bind to another address20/03/28 17:21:14 WARN
> NativeCodeLoader: Unable to load native-hadoop library for your platform...
> using builtin-java classes where applicableUsing Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties20/03/28 17:21:14 INFO
> SparkContext: Running Spark version 3.1.0-SNAPSHOT20/03/28 17:21:14 INFO
> ResourceUtils:
> ==20/03/28
> 17:21:14 INFO ResourceUtils: No custom resources configured for
> spark.driver.20/03/28 17:21:14 INFO ResourceUtils:
> ==20/03/28
> 17:21:14 INFO SparkContext: Submitted application: Spark Pi20/03/28 17:21:14
> INFO ResourceProfile: Default ResourceProfile created, executor resources:
> Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name:
> memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name:
> cpus, amount: 1.0)20/03/28 17:21:14 INFO ResourceProfile: Limiting resource
> is cpu20/03/28 17:21:14 INFO ResourceProfileManager: Added ResourceProfile
> id: 020/03/28 17:21:14 INFO SecurityManager: Changing view acls to:
> jorge20/03/28 17:21:14 INFO SecurityManager: Changing modify acls to:
> jorge20/03/28 17:21:14 INFO SecurityManager: Changing view acls groups
> to:20/03/28 17:21:14 INFO SecurityManager: Changing modify acls groups
> to:20/03/28 17:21:14 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(jorge); groups
> with view permissions: Set(); users with modify permissions: Set(jorge);
> groups with modify permissions: Set()20/03/28 17:21:14 INFO Utils:
> Successfully started service 'sparkDriver' on port 58192.20/03/28 17:21:14
> INFO SparkEnv: Registering MapOutputTracker20/03/28 17:21:14 INFO SparkEnv:
> Registering BlockManagerMaster20/03/28 17:21:14 INFO
> BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information20/03/28 17:21:14 INFO BlockManagerMasterEndpoint:
> BlockManagerMasterEndpoint up20/03/28 17:21:14 INFO SparkEnv: Registering
> BlockManagerMasterHeartbeat20/03/28 17:21:14 INFO DiskBlockManager: Created
> local directory at
> /private/var/folders/0h/5b7dw9p11l58hyk0_s0d3cnhgn/T/blockmgr-d9e88815-075e-4c9b-9cc8-21c72e97c86920/03/28
> 17:21:14 INFO MemoryStore: MemoryStore started with capacity 366.3
> MiB20/03/28 17:21:14 INFO SparkEnv: Registering
> OutputCommitCoordinator20/03/28 17:21:15 INFO Utils: Successfully started
> service 'SparkUI' on port 4040.20/03/28 17:21:15 INFO SparkUI: Bound SparkUI
> to 0.0.0.0, and started at http://192.168.1.2:404020/03/28 17:21:15 INFO
> SparkContext: Added JAR
> file:///Users/jorge/Downloads/spark/dist/examples/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar
> at spark://192.168.1.2:58192/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar
> with timestamp 158541247516620/03/28 17:21:15 INFO SparkContext: Added JAR
> file:///Users/jorge/Downloads/spark/dist/examples/jars/scopt_2.12-3.7.1.jar
> at spark://192.168.1.2:58192/jars/scopt_2.12-3.7.1.jar with timestamp
> 158541247516620/03/28 17:21:15 INFO Executor: Starting executor ID driver on
> host 192.168.1.220/03/28 17:21:15 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port
> 58193.20/03/28 17:21:15 INFO NettyBlockTransferService: Server created on
> 192.168.1.2:5819320/03/28 17:21:15 INFO BlockManager: Using
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
> policyException in thread "main" java.lang.NoClassDefFoundError:
> org/sparkproject/guava/util/concurrent/internal/InternalFutureFailureAccess
> at java.lang.ClassLoader.defineClass1(Native Method) at
> java.lang.ClassLoader.defineClass(ClassLoad
[jira] [Commented] (SPARK-31268) TaskEnd event with zero Executor Metrics when task duration less then poll interval
[ https://issues.apache.org/jira/browse/SPARK-31268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069939#comment-17069939 ] angerszhu commented on SPARK-31268: --- [https://github.com/apache/spark/pull/28034] > TaskEnd event with zero Executor Metrics when task duration less then poll > interval > --- > > Key: SPARK-31268 > URL: https://issues.apache.org/jira/browse/SPARK-31268 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: angerszhu >Priority: Major > Attachments: screenshot-1.png > > > TaskEnd event with zero Executor Metrics when task duration less then poll > interval -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28965) Document workings of CBO
[ https://issues.apache.org/jira/browse/SPARK-28965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-28965: --- Affects Version/s: (was: 2.4.3) 3.1.0 > Document workings of CBO > > > Key: SPARK-28965 > URL: https://issues.apache.org/jira/browse/SPARK-28965 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 3.1.0 >Reporter: Dilip Biswal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-30095) create function syntax has to be enhance in Doc for multiple dependent jars
[ https://issues.apache.org/jira/browse/SPARK-30095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-30095. Resolution: Not A Problem > create function syntax has to be enhance in Doc for multiple dependent jars > > > Key: SPARK-30095 > URL: https://issues.apache.org/jira/browse/SPARK-30095 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > Create Function Example and Syntax has to be enhance as below > 1. Case 1: How to use multiple dependent jars in the path while creating > function is not clear. -- Syntax to be given > 2. Case 2: What are the different schema supported like file:/// is not > updated in doc - Supported Schema to be provided -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29458) Document scalar functions usage in APIs in SQL getting started.
[ https://issues.apache.org/jira/browse/SPARK-29458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-29458: --- Affects Version/s: (was: 2.4.3) 3.1.0 > Document scalar functions usage in APIs in SQL getting started. > --- > > Key: SPARK-29458 > URL: https://issues.apache.org/jira/browse/SPARK-29458 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 3.1.0 >Reporter: Dilip Biswal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30272) Remove usage of Guava that breaks in Guava 27
[ https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069996#comment-17069996 ] Sean R. Owen commented on SPARK-30272: -- Spark still uses Guava 14 in order to retain compatibility with Hadoop <= 3.2.0. However this patch tries to ensure it doesn't use things that are present in Guava 27. Spark doesn't use that class directly, nor do I remember seeing an error like this. However I see for example: https://github.com/google/guava/blob/master/futures/failureaccess/src/com/google/common/util/concurrent/internal/InternalFutureFailureAccess.java#L26 which suggests this is part of a different JAR that _should_ be included automatically as a transitive dependency. How are you building and running? are you using Hadoop 3.2.1? This doesn't seem to make our PR builders fail, but doesn't mean there isn't some subtle issue in there. > Remove usage of Guava that breaks in Guava 27 > - > > Key: SPARK-30272 > URL: https://issues.apache.org/jira/browse/SPARK-30272 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Sean R. Owen >Assignee: Sean R. Owen >Priority: Major > Fix For: 3.0.0 > > > Background: > https://issues.apache.org/jira/browse/SPARK-29250 > https://github.com/apache/spark/pull/25932 > Hadoop 3.2.1 will update Guava from 11 to 27. There are a number of methods > that changed between those releases, typically just a rename, but, means one > set of code can't work with both, while we want to work with Hadoop 2.x and > 3.x. Among them: > - Objects.toStringHelper was moved to MoreObjects; we can just use the > Commons Lang3 equivalent > - Objects.hashCode etc were renamed; use java.util.Objects equivalents > - MoreExecutors.sameThreadExecutor() became directExecutor(); for same-thread > execution we can use a dummy implementation of ExecutorService / Executor > - TypeToken.isAssignableFrom become isSupertypeOf; work around with reflection > There is probably more to the Guava issue than just this change, but it will > make Spark itself work with more versions and reduce our exposure to Guava > along the way anyway. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31292) Replace toSet.toSeq with distinct for readability
[
https://issues.apache.org/jira/browse/SPARK-31292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takeshi Yamamuro resolved SPARK-31292.
--
Fix Version/s: 3.0.0
Assignee: Kengo Seki
Resolution: Fixed
Resolved by [https://github.com/apache/spark/pull/28062]
> Replace toSet.toSeq with distinct for readability
> -
>
> Key: SPARK-31292
> URL: https://issues.apache.org/jira/browse/SPARK-31292
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
>Affects Versions: 3.0.0
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Trivial
> Fix For: 3.0.0
>
>
> Some classes use {{toSet.toSeq}} to make the elements of a sequence unique,
> but it's a bit verbose. Using {{distinct}} instead is easier to understand.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-31294) Benchmark the performance regression
Dongjoon Hyun created SPARK-31294: - Summary: Benchmark the performance regression Key: SPARK-31294 URL: https://issues.apache.org/jira/browse/SPARK-31294 Project: Spark Issue Type: Sub-task Components: SQL, Tests Affects Versions: 3.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31294) Benchmark the performance regression
[ https://issues.apache.org/jira/browse/SPARK-31294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31294: - Assignee: Maxim Gekk > Benchmark the performance regression > > > Key: SPARK-31294 > URL: https://issues.apache.org/jira/browse/SPARK-31294 > Project: Spark > Issue Type: Sub-task > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Maxim Gekk >Assignee: Maxim Gekk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31294) Benchmark the performance regression
[ https://issues.apache.org/jira/browse/SPARK-31294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31294: -- Reporter: Maxim Gekk (was: Dongjoon Hyun) > Benchmark the performance regression > > > Key: SPARK-31294 > URL: https://issues.apache.org/jira/browse/SPARK-31294 > Project: Spark > Issue Type: Sub-task > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Maxim Gekk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30951) Potential data loss for legacy applications after switch to proleptic Gregorian calendar
[
https://issues.apache.org/jira/browse/SPARK-30951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30951:
--
Labels: release-notes (was: )
> Potential data loss for legacy applications after switch to proleptic
> Gregorian calendar
>
>
> Key: SPARK-30951
> URL: https://issues.apache.org/jira/browse/SPARK-30951
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Bruce Robbins
>Assignee: Maxim Gekk
>Priority: Blocker
> Labels: release-notes
> Fix For: 3.0.0
>
>
> tl;dr: We recently discovered some Spark 2.x sites that have lots of data
> containing dates before October 15, 1582. This could be an issue when such
> sites try to upgrade to Spark 3.0.
> From SPARK-26651:
> {quote}"The changes might impact on the results for dates and timestamps
> before October 15, 1582 (Gregorian)
> {quote}
> We recently discovered that some large scale Spark 2.x applications rely on
> dates before October 15, 1582.
> Two cases came up recently:
> * An application that uses a commercial third-party library to encode
> sensitive dates. On insert, the library encodes the actual date as some other
> date. On select, the library decodes the date back to the original date. The
> encoded value could be any date, including one before October 15, 1582 (e.g.,
> "0602-04-04").
> * An application that uses a specific unlikely date (e.g., "1200-01-01") as
> a marker to indicate "unknown date" (in lieu of null)
> Both sites ran into problems after another component in their system was
> upgraded to use the proleptic Gregorian calendar. Spark applications that
> read files created by the upgraded component were interpreting encoded or
> marker dates incorrectly, and vice versa. Also, their data now had a mix of
> calendars (hybrid and proleptic Gregorian) with no metadata to indicate which
> file used which calendar.
> Both sites had enormous amounts of existing data, so re-encoding the dates
> using some other scheme was not a feasible solution.
> This is relevant to Spark 3:
> Any Spark 2 application that uses such date-encoding schemes may run into
> trouble when run on Spark 3. The application may not properly interpret the
> dates previously written by Spark 2. Also, once the Spark 3 version of the
> application writes data, the tables will have a mix of calendars (hybrid and
> proleptic gregorian) with no metadata to indicate which file uses which
> calendar.
> Similarly, sites might run with mixed Spark versions, resulting in data
> written by one version that cannot be interpreted by the other. And as above,
> the tables will now have a mix of calendars with no way to detect which file
> uses which calendar.
> As with the two real-life example cases, these applications may have enormous
> amounts of legacy data, so re-encoding the dates using some other scheme may
> not be feasible.
> We might want to consider a configuration setting to allow the user to
> specify the calendar for storing and retrieving date and timestamp values
> (not sure how such a flag would affect other date and timestamp-related
> functions). I realize the change is far bigger than just adding a
> configuration setting.
> Here's a quick example of where trouble may happen, using the real-life case
> of the marker date.
> In Spark 2.4:
> {noformat}
> scala> spark.read.orc(s"$home/data/datefile").filter("dt ==
> '1200-01-01'").count
> res0: Long = 1
> scala>
> {noformat}
> In Spark 3.0 (reading from the same legacy file):
> {noformat}
> scala> spark.read.orc(s"$home/data/datefile").filter("dt ==
> '1200-01-01'").count
> res0: Long = 0
> scala>
> {noformat}
> By the way, Hive had a similar problem. Hive switched from hybrid calendar to
> proleptic Gregorian calendar between 2.x and 3.x. After some upgrade
> headaches related to dates before 1582, the Hive community made the following
> changes:
> * When writing date or timestamp data to ORC, Parquet, and Avro files, Hive
> checks a configuration setting to determine which calendar to use.
> * When writing date or timestamp data to ORC, Parquet, and Avro files, Hive
> stores the calendar type in the metadata.
> * When reading date or timestamp data from ORC, Parquet, and Avro files,
> Hive checks the metadata for the calendar type.
> * When reading date or timestamp data from ORC, Parquet, and Avro files that
> lack calendar metadata, Hive's behavior is determined by a configuration
> setting. This allows Hive to read legacy data (note: if the data already
> consists of a mix of calendar types with no metadata, there is no good
> solution).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SPARK-30951) Potential data loss for legacy applications after switch to proleptic Gregorian calendar
[
https://issues.apache.org/jira/browse/SPARK-30951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070169#comment-17070169
]
Dongjoon Hyun commented on SPARK-30951:
---
I added `release-notes` label because
`spark.sql.legacy.parquet.rebaseDateTime.enabled` is `false` by default.
cc [~rxin]
> Potential data loss for legacy applications after switch to proleptic
> Gregorian calendar
>
>
> Key: SPARK-30951
> URL: https://issues.apache.org/jira/browse/SPARK-30951
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Bruce Robbins
>Assignee: Maxim Gekk
>Priority: Blocker
> Labels: release-notes
> Fix For: 3.0.0
>
>
> tl;dr: We recently discovered some Spark 2.x sites that have lots of data
> containing dates before October 15, 1582. This could be an issue when such
> sites try to upgrade to Spark 3.0.
> From SPARK-26651:
> {quote}"The changes might impact on the results for dates and timestamps
> before October 15, 1582 (Gregorian)
> {quote}
> We recently discovered that some large scale Spark 2.x applications rely on
> dates before October 15, 1582.
> Two cases came up recently:
> * An application that uses a commercial third-party library to encode
> sensitive dates. On insert, the library encodes the actual date as some other
> date. On select, the library decodes the date back to the original date. The
> encoded value could be any date, including one before October 15, 1582 (e.g.,
> "0602-04-04").
> * An application that uses a specific unlikely date (e.g., "1200-01-01") as
> a marker to indicate "unknown date" (in lieu of null)
> Both sites ran into problems after another component in their system was
> upgraded to use the proleptic Gregorian calendar. Spark applications that
> read files created by the upgraded component were interpreting encoded or
> marker dates incorrectly, and vice versa. Also, their data now had a mix of
> calendars (hybrid and proleptic Gregorian) with no metadata to indicate which
> file used which calendar.
> Both sites had enormous amounts of existing data, so re-encoding the dates
> using some other scheme was not a feasible solution.
> This is relevant to Spark 3:
> Any Spark 2 application that uses such date-encoding schemes may run into
> trouble when run on Spark 3. The application may not properly interpret the
> dates previously written by Spark 2. Also, once the Spark 3 version of the
> application writes data, the tables will have a mix of calendars (hybrid and
> proleptic gregorian) with no metadata to indicate which file uses which
> calendar.
> Similarly, sites might run with mixed Spark versions, resulting in data
> written by one version that cannot be interpreted by the other. And as above,
> the tables will now have a mix of calendars with no way to detect which file
> uses which calendar.
> As with the two real-life example cases, these applications may have enormous
> amounts of legacy data, so re-encoding the dates using some other scheme may
> not be feasible.
> We might want to consider a configuration setting to allow the user to
> specify the calendar for storing and retrieving date and timestamp values
> (not sure how such a flag would affect other date and timestamp-related
> functions). I realize the change is far bigger than just adding a
> configuration setting.
> Here's a quick example of where trouble may happen, using the real-life case
> of the marker date.
> In Spark 2.4:
> {noformat}
> scala> spark.read.orc(s"$home/data/datefile").filter("dt ==
> '1200-01-01'").count
> res0: Long = 1
> scala>
> {noformat}
> In Spark 3.0 (reading from the same legacy file):
> {noformat}
> scala> spark.read.orc(s"$home/data/datefile").filter("dt ==
> '1200-01-01'").count
> res0: Long = 0
> scala>
> {noformat}
> By the way, Hive had a similar problem. Hive switched from hybrid calendar to
> proleptic Gregorian calendar between 2.x and 3.x. After some upgrade
> headaches related to dates before 1582, the Hive community made the following
> changes:
> * When writing date or timestamp data to ORC, Parquet, and Avro files, Hive
> checks a configuration setting to determine which calendar to use.
> * When writing date or timestamp data to ORC, Parquet, and Avro files, Hive
> stores the calendar type in the metadata.
> * When reading date or timestamp data from ORC, Parquet, and Avro files,
> Hive checks the metadata for the calendar type.
> * When reading date or timestamp data from ORC, Parquet, and Avro files that
> lack calendar metadata, Hive's behavior is determined by a configuration
> setting. This allows Hive to read legacy data (note: if the data already
> consists of a mix of c
[jira] [Created] (SPARK-31295) Supplement version for configuration appear in doc
jiaan.geng created SPARK-31295: -- Summary: Supplement version for configuration appear in doc Key: SPARK-31295 URL: https://issues.apache.org/jira/browse/SPARK-31295 Project: Spark Issue Type: Sub-task Components: Documentation Affects Versions: 3.1.0 Reporter: jiaan.geng docs/spark-standalone.md docs/sql-data-sources-avro.md docs/sql-data-sources-orc.md docs/sql-data-sources-parquet.md -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23432) Expose executor memory metrics in the web UI for executors
[ https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23432: -- Affects Version/s: (was: 2.2.1) 3.1.0 > Expose executor memory metrics in the web UI for executors > -- > > Key: SPARK-23432 > URL: https://issues.apache.org/jira/browse/SPARK-23432 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Edward Lu >Priority: Major > > Add the new memory metrics (jvmUsedMemory, executionMemory, storageMemory, > and unifiedMemory, etc.) to the executors tab, in the summary and for each > executor. > This is a subtask for SPARK-23206. Please refer to the design doc for that > ticket for more details. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31270) Expose executor memory metrics at the task detal, in the Stages tab
[ https://issues.apache.org/jira/browse/SPARK-31270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31270: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Expose executor memory metrics at the task detal, in the Stages tab > --- > > Key: SPARK-31270 > URL: https://issues.apache.org/jira/browse/SPARK-31270 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: angerszhu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26341) Expose executor memory metrics at the stage level, in the Stages tab
[ https://issues.apache.org/jira/browse/SPARK-26341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26341: -- Affects Version/s: (was: 2.4.0) 3.1.0 > Expose executor memory metrics at the stage level, in the Stages tab > > > Key: SPARK-26341 > URL: https://issues.apache.org/jira/browse/SPARK-26341 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, Web UI >Affects Versions: 3.1.0 >Reporter: Edward Lu >Priority: Major > > Sub-task SPARK-23431 will add stage level executor memory metrics (peak > values for each stage, and peak values for each executor for the stage). This > information should also be exposed the the web UI, so that users can see > which stages are memory intensive. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-26399:
--
Affects Version/s: (was: 2.4.0)
3.1.0
> Add new stage-level REST APIs and parameters
>
>
> Key: SPARK-26399
> URL: https://issues.apache.org/jira/browse/SPARK-26399
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Edward Lu
>Priority: Major
>
> Add the peak values for the metrics to the stages REST API. Also add a new
> executorSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://:18080/api/v1/applications/ id>// attempt>/executorSummary{code}
> Add parameters to the stages REST API to specify:
> * filtering for task status, and returning tasks that match (for example,
> FAILED tasks).
> * task metric quantiles, add adding the task summary if specified
> * executor metric quantiles, and adding the executor summary if specified
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23431) Expose the new executor memory metrics at the stage level
[ https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23431: -- Affects Version/s: (was: 2.2.1) 3.1.0 > Expose the new executor memory metrics at the stage level > - > > Key: SPARK-23431 > URL: https://issues.apache.org/jira/browse/SPARK-23431 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Edward Lu >Priority: Major > > Collect and show the new executor memory metrics for each stage, to provide > more information on how memory is used per stage. > Modify the AppStatusListener to track the peak values for JVM used memory, > execution memory, storage memory, and unified memory for each executor for > each stage. > This is a subtask for SPARK-23206. Please refer to the design doc for that > ticket for more details. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31268) TaskEnd event with zero Executor Metrics when task duration less then poll interval
[ https://issues.apache.org/jira/browse/SPARK-31268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31268: -- Affects Version/s: (was: 3.0.0) 3.1.0 > TaskEnd event with zero Executor Metrics when task duration less then poll > interval > --- > > Key: SPARK-31268 > URL: https://issues.apache.org/jira/browse/SPARK-31268 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: angerszhu >Priority: Major > Attachments: screenshot-1.png > > > TaskEnd event with zero Executor Metrics when task duration less then poll > interval -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31021) Support MariaDB Kerberos login in JDBC connector
[ https://issues.apache.org/jira/browse/SPARK-31021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31021: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Support MariaDB Kerberos login in JDBC connector > > > Key: SPARK-31021 > URL: https://issues.apache.org/jira/browse/SPARK-31021 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31018) Deprecate support of multiple workers on the same host in Standalone
[ https://issues.apache.org/jira/browse/SPARK-31018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31018: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Deprecate support of multiple workers on the same host in Standalone > > > Key: SPARK-31018 > URL: https://issues.apache.org/jira/browse/SPARK-31018 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: wuyi >Priority: Major > > Deprecate support of multiple workers on the same host in Standalone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30978) Remove multiple workers on the same host support from Standalone backend
[ https://issues.apache.org/jira/browse/SPARK-30978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30978: -- Affects Version/s: (was: 3.0.0) > Remove multiple workers on the same host support from Standalone backend > > > Key: SPARK-30978 > URL: https://issues.apache.org/jira/browse/SPARK-30978 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Xingbo Jiang >Assignee: Xingbo Jiang >Priority: Major > > Based on our experience, there is no scenario that necessarily requires > deploying multiple Workers on the same node with Standalone backend. A worker > should book all the resources reserved to Spark on the host it is launched, > then it can allocate those resources to one or more executors launched by > this worker. Since each executor runs in a separated JVM, we can limit the > memory of each executor to avoid long GC pause. > The remaining concern is the local-cluster mode is implemented by launching > multiple workers on the local host, we might need to re-implement > LocalSparkCluster to launch only one Worker and multiple executors. It should > be fine because local-cluster mode is only used in running Spark unit test > cases, thus end users should not be affected by this change. > Removing multiple workers on the same host support could simplify the deploy > model of Standalone backend, and also reduce the burden to support legacy > deploy pattern in the future feature developments. > The proposal is to update the document to deprecate the support of system > environment `SPARK_WORKER_INSTANCES` in 3.0, and remove the support in the > next major version (3.1.0). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30835) Add support for YARN decommissioning & pre-emption
[ https://issues.apache.org/jira/browse/SPARK-30835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30835: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Add support for YARN decommissioning & pre-emption > -- > > Key: SPARK-30835 > URL: https://issues.apache.org/jira/browse/SPARK-30835 > Project: Spark > Issue Type: Sub-task > Components: YARN >Affects Versions: 3.1.0 >Reporter: Holden Karau >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30820) Add FMClassifier to SparkR
[
https://issues.apache.org/jira/browse/SPARK-30820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30820:
--
Affects Version/s: (was: 3.0.0)
> Add FMClassifier to SparkR
> --
>
> Key: SPARK-30820
> URL: https://issues.apache.org/jira/browse/SPARK-30820
> Project: Spark
> Issue Type: Sub-task
> Components: ML, SparkR
>Affects Versions: 3.1.0
>Reporter: Maciej Szymkiewicz
>Priority: Major
>
> Spark should provide a wrapper for {{o.a.s.ml.classification. FMClassifier}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30818) Add LinearRegression wrapper to SparkR
[
https://issues.apache.org/jira/browse/SPARK-30818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30818:
--
Affects Version/s: (was: 3.0.0)
> Add LinearRegression wrapper to SparkR
> --
>
> Key: SPARK-30818
> URL: https://issues.apache.org/jira/browse/SPARK-30818
> Project: Spark
> Issue Type: Sub-task
> Components: ML, SparkR
>Affects Versions: 3.1.0
>Reporter: Maciej Szymkiewicz
>Priority: Major
>
> Spark should provide a wrapper for {{o.a.s.ml.regression. LinearRegression}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30819) Add FMRegressor wrapper to SparkR
[
https://issues.apache.org/jira/browse/SPARK-30819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30819:
--
Affects Version/s: (was: 3.0.0)
> Add FMRegressor wrapper to SparkR
> -
>
> Key: SPARK-30819
> URL: https://issues.apache.org/jira/browse/SPARK-30819
> Project: Spark
> Issue Type: Sub-task
> Components: ML, SparkR
>Affects Versions: 3.1.0
>Reporter: Maciej Szymkiewicz
>Priority: Major
>
> Spark should provide a wrapper for {{o.a.s.ml.regression. FMRegressor}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30734) AnalysisException that window RangeFrame not match RowFrame
[
https://issues.apache.org/jira/browse/SPARK-30734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30734:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> AnalysisException that window RangeFrame not match RowFrame
> ---
>
> Key: SPARK-30734
> URL: https://issues.apache.org/jira/browse/SPARK-30734
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
>
>
> {code:java}
> select last(salary) over(order by salary range between 1000 preceding and
> 1000 following),
> lag(salary) over(order by salary range between 1000 preceding and 1000
> following),
> salary from empsalary
> org.apache.spark.sql.AnalysisException
> Window Frame specifiedwindowframe(RangeFrame, -1000, 1000) must match the
> required frame specifiedwindowframe(RowFrame, -1, -1);
> {code}
>
> Maybe we need to fix this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30824) Support submit sql content only contains comments.
[ https://issues.apache.org/jira/browse/SPARK-30824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30824: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Support submit sql content only contains comments. > -- > > Key: SPARK-30824 > URL: https://issues.apache.org/jira/browse/SPARK-30824 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jiaan.geng >Priority: Major > > Spark SQL cannot accept input comments as sql. > Postegresql could accept comments only. > We may need to resolve this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30707) Lead/Lag window function throws AnalysisException without ORDER BY clause
[
https://issues.apache.org/jira/browse/SPARK-30707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30707:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Lead/Lag window function throws AnalysisException without ORDER BY clause
> -
>
> Key: SPARK-30707
> URL: https://issues.apache.org/jira/browse/SPARK-30707
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> Lead/Lag window function throws AnalysisException without ORDER BY clause:
> {code:java}
> SELECT lead(ten, four + 1) OVER (PARTITION BY four), ten, four
> FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten)s
> org.apache.spark.sql.AnalysisException
> Window function lead(ten#x, (four#x + 1), null) requires window to be
> ordered, please add ORDER BY clause. For example SELECT lead(ten#x, (four#x +
> 1), null)(value_expr) OVER (PARTITION BY window_partition ORDER BY
> window_ordering) from table;
> {code}
>
> Maybe we need fix this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30175) Eliminate warnings: part 5
[
https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30175:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Eliminate warnings: part 5
> --
>
> Key: SPARK-30175
> URL: https://issues.apache.org/jira/browse/SPARK-30175
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jobit mathew
>Priority: Minor
>
> sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala
> {code:java}
> Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is
> deprecated (since 2.4.0): Use specific logical plans like AppendData instead
> def createPlan(batchId: Long): WriteToDataSourceV2 = {
> Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is
> deprecated (since 2.4.0): Use specific logical plans like AppendData instead
> WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query)
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
> {code:java}
> Warning:Warning:line (703)a pure expression does nothing in statement
> position; multiline expressions might require enclosing parentheses
> q1
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala
> {code:java}
> Warning:Warning:line (285)object typed in package scalalang is deprecated
> (since 3.0.0): please use untyped builtin aggregate functions.
> val aggregated =
> inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30172) Eliminate warnings: part 3
[
https://issues.apache.org/jira/browse/SPARK-30172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30172:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Eliminate warnings: part 3
> --
>
> Key: SPARK-30172
> URL: https://issues.apache.org/jira/browse/SPARK-30172
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> /sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
> Warning:Warning:line (422)method initialize in class AbstractSerDe is
> deprecated: see corresponding Javadoc for more information.
> serde.initialize(null, properties)
> /sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
> Warning:Warning:line (216)method initialize in class GenericUDTF is
> deprecated: see corresponding Javadoc for more information.
> protected lazy val outputInspector =
> function.initialize(inputInspectors.toArray)
> Warning:Warning:line (342)class UDAF in package exec is deprecated: see
> corresponding Javadoc for more information.
> new GenericUDAFBridge(funcWrapper.createFunction[UDAF]())
> Warning:Warning:line (503)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> def serialize(buffer: AggregationBuffer): Array[Byte] = {
> Warning:Warning:line (523)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> def deserialize(bytes: Array[Byte]): AggregationBuffer = {
> Warning:Warning:line (538)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> case class HiveUDAFBuffer(buf: AggregationBuffer, canDoMerge: Boolean)
> Warning:Warning:line (538)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> case class HiveUDAFBuffer(buf: AggregationBuffer, canDoMerge: Boolean)
> /sql/hive/src/main/java/org/apache/hadoop/hive/ql/io/orc/SparkOrcNewRecordReader.java
> Warning:Warning:line (44)java: getTypes() in org.apache.orc.Reader has
> been deprecated
> Warning:Warning:line (47)java: getTypes() in org.apache.orc.Reader has
> been deprecated
> /sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
> Warning:Warning:line (2,368)method readFooter in class ParquetFileReader
> is deprecated: see corresponding Javadoc for more information.
> val footer = ParquetFileReader.readFooter(
> /sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala
> Warning:Warning:line (202)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def getNewAggregationBuffer: AggregationBuffer = new
> MockUDAFBuffer(0L, 0L)
> Warning:Warning:line (204)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def reset(agg: AggregationBuffer): Unit = {
> Warning:Warning:line (212)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def iterate(agg: AggregationBuffer, parameters: Array[AnyRef]):
> Unit = {
> Warning:Warning:line (221)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def merge(agg: AggregationBuffer, partial: Object): Unit = {
> Warning:Warning:line (231)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def terminatePartial(agg: AggregationBuffer): AnyRef = {
> Warning:Warning:line (236)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def terminate(agg: AggregationBuffer): AnyRef =
> terminatePartial(agg)
> Warning:Warning:line (257)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def getNewAggregationBuffer: AggregationBuffer = {
> Warning:Warning:line (266)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def reset(agg: AggregationBuffer): Unit = {
> Warning:Warning:line (277)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def iterate(agg: AggregationBuffer, parameters: Array[AnyRef]):
> Unit = {
[jira] [Updated] (SPARK-30177) Eliminate warnings: part 7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30177: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Eliminate warnings: part 7 > -- > > Key: SPARK-30177 > URL: https://issues.apache.org/jira/browse/SPARK-30177 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > /mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala > Warning:Warning:line (108)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > assert(model.computeCost(dataset) < 0.1) > Warning:Warning:line (135)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > assert(model.computeCost(dataset) == summary.trainingCost) > Warning:Warning:line (195)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > model.computeCost(dataset) > > /sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala > Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java > enum Feature is deprecated: see corresponding Javadoc for more information. > jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS) > /sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java > Warning:Warning:line (28)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (37)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (46)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (55)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (64)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > /sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > Information:Information:java: > /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > uses unchecked or unsafe operations. > Information:Information:java: Recompile with -Xlint:unchecked for details. > /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java > Warning:Warning:line (478)java: > json(org.apache.spark.api.java.JavaRDD) in > org.apache.spark.sql.DataFrameReader has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30182) Support nested aggregates
[
https://issues.apache.org/jira/browse/SPARK-30182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30182:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support nested aggregates
> -
>
> Key: SPARK-30182
> URL: https://issues.apache.org/jira/browse/SPARK-30182
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> Spark SQL cannot supports a SQL with nested aggregate as below:
> {code:java}
> SELECT sum(salary), row_number() OVER (ORDER BY depname), sum(
> sum(salary) FILTER (WHERE enroll_date > '2007-01-01')
> ) FILTER (WHERE depname <> 'sales') OVER (ORDER BY depname DESC) AS
> "filtered_sum",
> depname
> FROM empsalary GROUP BY depname;{code}
> And Spark will throw exception as follows:
> {code:java}
> org.apache.spark.sql.AnalysisException
> It is not allowed to use an aggregate function in the argument of another
> aggregate function. Please use the inner aggregate function in a
> sub-query.{code}
> But PostgreSQL supports this syntax.
> {code:java}
> SELECT sum(salary), row_number() OVER (ORDER BY depname), sum(
> sum(salary) FILTER (WHERE enroll_date > '2007-01-01')
> ) FILTER (WHERE depname <> 'sales') OVER (ORDER BY depname DESC) AS
> "filtered_sum",
> depname
> FROM empsalary GROUP BY depname;
> sum | row_number | filtered_sum | depname
> ---++--+---
> 25100 | 1 | 22600 | develop
> 7400 | 2 | 3500 | personnel
> 14600 | 3 | | sales
> (3 rows){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29838) PostgreSQL dialect: cast to timestamp
[ https://issues.apache.org/jira/browse/SPARK-29838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29838: -- Affects Version/s: (was: 3.0.0) 3.1.0 > PostgreSQL dialect: cast to timestamp > - > > Key: SPARK-29838 > URL: https://issues.apache.org/jira/browse/SPARK-29838 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Major > > Make SparkSQL's cast to timestamp behavior be consistent with PostgreSQL when > spark.sql.dialect is configured as PostgreSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30169:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Eliminate warnings in Kafka connector
> -
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> *Eliminate compilation warnings (10) in module external/kafka-0-10*:
> 1.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> {code:java}
> Line:108
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line:162
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> {code:java}
> Line:206
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 3.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> {code:java}
> Line:258
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:314
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:476
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 4.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> {code:java}
> Line:60
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:88
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:100
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:178
> method createTopic in object AdminUtils is deprecated (since 1.1.0): This
> method is deprecated and will be replaced by kafka.zk.AdminZkClient.
> {code}
> *Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
> 1.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
> {code}
> Line:74
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
> {code}
> Line:119
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:382
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> {code}
> 3.
> spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
> {code}
> Line:84
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:118
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:246
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 458
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 472
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 581
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 582
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient in
[jira] [Updated] (SPARK-30168) Eliminate warnings in Parquet datasource
[
https://issues.apache.org/jira/browse/SPARK-30168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30168:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Eliminate warnings in Parquet datasource
>
>
> Key: SPARK-30168
> URL: https://issues.apache.org/jira/browse/SPARK-30168
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> #
> sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala
> {code}
> Warning:Warning:line (120)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> Option[TimeZone]) => RecordReader[Void, T]): RecordReader[Void, T]
> = {
> Warning:Warning:line (125)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> new org.apache.parquet.hadoop.ParquetInputSplit(
> Warning:Warning:line (134)method readFooter in class ParquetFileReader is
> deprecated: see corresponding Javadoc for more information.
> ParquetFileReader.readFooter(conf, filePath,
> SKIP_ROW_GROUPS).getFileMetaData
> Warning:Warning:line (183)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> split: ParquetInputSplit,
> Warning:Warning:line (212)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> split: ParquetInputSplit,
> {code}
> #
> sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
> {code}
> Warning:Warning:line (55)java: org.apache.parquet.hadoop.ParquetInputSplit in
> org.apache.parquet.hadoop has been deprecated
> Warning:Warning:line (95)java:
> org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has
> been deprecated
> Warning:Warning:line (95)java:
> org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has
> been deprecated
> Warning:Warning:line (97)java: getRowGroupOffsets() in
> org.apache.parquet.hadoop.ParquetInputSplit has been deprecated
> Warning:Warning:line (105)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (108)java:
> filterRowGroups(org.apache.parquet.filter2.compat.FilterCompat.Filter,java.util.List,org.apache.parquet.schema.MessageType)
> in org.apache.parquet.filter2.compat.RowGroupFilter has been deprecated
> Warning:Warning:line (111)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (147)java:
> ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List,java.util.List)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (203)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (226)java:
> ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List,java.util.List)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> {code}
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompatibilityTest.scala
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
> #
> sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29845) PostgreSQL dialect: cast to decimal
[ https://issues.apache.org/jira/browse/SPARK-29845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29845: -- Affects Version/s: (was: 3.0.0) 3.1.0 > PostgreSQL dialect: cast to decimal > --- > > Key: SPARK-29845 > URL: https://issues.apache.org/jira/browse/SPARK-29845 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > Make SparkSQL's cast to decimal behavior be consistent with PostgreSQL when > spark.sql.dialect is configured as PostgreSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29841) PostgreSQL dialect: cast to date
[ https://issues.apache.org/jira/browse/SPARK-29841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29841: -- Affects Version/s: (was: 3.0.0) 3.1.0 > PostgreSQL dialect: cast to date > > > Key: SPARK-29841 > URL: https://issues.apache.org/jira/browse/SPARK-29841 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > Make SparkSQL's cast to date behavior be consistent with PostgreSQL when > spark.sql.dialect is configured as PostgreSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29840) PostgreSQL dialect: cast to integer
[
https://issues.apache.org/jira/browse/SPARK-29840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29840:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> PostgreSQL dialect: cast to integer
> ---
>
> Key: SPARK-29840
> URL: https://issues.apache.org/jira/browse/SPARK-29840
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jobit mathew
>Priority: Major
>
> Make SparkSQL's cast to integer behavior be consistent with PostgreSQL when
> spark.sql.dialect is configured as PostgreSQL.
> Example:*currently spark sql*
> {code:java}
> spark-sql> select CAST ('10C' AS INTEGER);
> NULL
> Time taken: 0.051 seconds, Fetched 1 row(s)
> spark-sql>
> {code}
> *postgresql*
> {code:java}
> postgresql
> select CAST ('10C' AS INTEGER);
> Error(s), warning(s):
> 22P02: invalid input syntax for integer: "10C"
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29842) PostgreSQL dialect: cast to double
[
https://issues.apache.org/jira/browse/SPARK-29842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29842:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> PostgreSQL dialect: cast to double
> --
>
> Key: SPARK-29842
> URL: https://issues.apache.org/jira/browse/SPARK-29842
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jobit mathew
>Priority: Major
>
> Make SparkSQL's cast to double behavior be consistent with PostgreSQL when
> spark.sql.dialect is configured as PostgreSQL.
> some examples
> {code:java}
> spark-sql> select CAST ('10.2' AS DOUBLE PRECISION);
> Error in query:
> extraneous input 'PRECISION' expecting ')'(line 1, pos 30)
> == SQL ==
> select CAST ('10.2' AS DOUBLE PRECISION)
> --^^^
> spark-sql> select CAST ('10.2' AS DOUBLE PRECISION);
> Error in query:
> extraneous input 'PRECISION' expecting ')'(line 1, pos 30)
> == SQL ==
> select CAST ('10.2' AS DOUBLE PRECISION)
> --^^^
> spark-sql> select CAST ('10.2' AS DOUBLE);
> 10.2
> Time taken: 0.08 seconds, Fetched 1 row(s)
> spark-sql> select CAST ('10.' AS DOUBLE);
> 10.
> Time taken: 0.08 seconds, Fetched 1 row(s)
> spark-sql> select CAST ('ff' AS DOUBLE);
> NULL
> Time taken: 0.08 seconds, Fetched 1 row(s)
> spark-sql> select CAST ('1' AS DOUBLE);
> 1.1112E16
> Time taken: 0.067 seconds, Fetched 1 row(s)
> spark-sql>
> {code}
> Postgresql
> select CAST ('10.222' AS DOUBLE PRECISION);
> select CAST ('1' AS DOUBLE PRECISION);
> select CAST ('ff' AS DOUBLE PRECISION);
>
>
> || ||float8||
> |1|10,222|
>
> || ||float8||
> |1|1,11E+16|
> Error(s), warning(s):
> 22P02: invalid input syntax for type double precision: "ff"
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29843) PostgreSQL dialect: cast to float
[ https://issues.apache.org/jira/browse/SPARK-29843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29843: -- Affects Version/s: (was: 3.0.0) 3.1.0 > PostgreSQL dialect: cast to float > - > > Key: SPARK-29843 > URL: https://issues.apache.org/jira/browse/SPARK-29843 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Major > > Make SparkSQL's cast to float behavior be consistent with PostgreSQL when > spark.sql.dialect is configured as PostgreSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29847) PostgreSQL dialect: cast to varchar
[
https://issues.apache.org/jira/browse/SPARK-29847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29847:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> PostgreSQL dialect: cast to varchar
> ---
>
> Key: SPARK-29847
> URL: https://issues.apache.org/jira/browse/SPARK-29847
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> In Spark
> jdbc:hive2://10.18.19.208:23040/default> select cast('10.345bb' as
> varchar(10));
> +---+
> | CAST(10.345bb AS STRING) |
> +---+
> | 10.345*bb* |
> +---+
>
> In PostgreSQL
> select cast('10.345bb' as varchar(10));
> varchar varchar1 *10.345*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29848) PostgreSQL dialect: cast to bigint
[
https://issues.apache.org/jira/browse/SPARK-29848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29848:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> PostgreSQL dialect: cast to bigint
> --
>
> Key: SPARK-29848
> URL: https://issues.apache.org/jira/browse/SPARK-29848
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> Spark:
> 0: jdbc:hive2://10.18.19.208:23040/default> select CAST('0xcc' AS
> bigint);
> +---+
> | CAST(0xcc AS BIGINT) |
> +---+
> | NULL |
> +---+
> Postgre SQL
> 22P02: invalid input syntax for integer: "0xcc"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29887) PostgreSQL dialect: cast to smallint
[ https://issues.apache.org/jira/browse/SPARK-29887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29887: -- Affects Version/s: (was: 3.0.0) 3.1.0 > PostgreSQL dialect: cast to smallint > > > Key: SPARK-29887 > URL: https://issues.apache.org/jira/browse/SPARK-29887 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > Make SparkSQL's cast to smallint behavior be consistent with PostgreSQL when > spark.sql.dialect is configured as PostgreSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29846) PostgreSQL dialect: cast to char
[
https://issues.apache.org/jira/browse/SPARK-29846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-29846.
---
Resolution: Won't Do
> PostgreSQL dialect: cast to char
>
>
> Key: SPARK-29846
> URL: https://issues.apache.org/jira/browse/SPARK-29846
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: jobit mathew
>Priority: Minor
>
> Make SparkSQL's cast to char behavior be consistent with PostgreSQL when
> spark.sql.dialect is configured as PostgreSQL.
> {code:java}
> spark-sql> select cast ('10.22333' as
> char(5));
> 10.22333
> Time taken: 0.062 seconds, Fetched 1 row(s)
> spark-sql>
> {code}
> *postgresql*
> select cast ('10.22333' as char(5));
>
> || ||bpchar||
> |1|10.22|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-29846) PostgreSQL dialect: cast to char
[
https://issues.apache.org/jira/browse/SPARK-29846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-29846.
-
> PostgreSQL dialect: cast to char
>
>
> Key: SPARK-29846
> URL: https://issues.apache.org/jira/browse/SPARK-29846
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: jobit mathew
>Priority: Minor
>
> Make SparkSQL's cast to char behavior be consistent with PostgreSQL when
> spark.sql.dialect is configured as PostgreSQL.
> {code:java}
> spark-sql> select cast ('10.22333' as
> char(5));
> 10.22333
> Time taken: 0.062 seconds, Fetched 1 row(s)
> spark-sql>
> {code}
> *postgresql*
> select cast ('10.22333' as char(5));
>
> || ||bpchar||
> |1|10.22|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29846) PostgreSQL dialect: cast to char
[
https://issues.apache.org/jira/browse/SPARK-29846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070180#comment-17070180
]
Dongjoon Hyun commented on SPARK-29846:
---
Hi, All.
I close this issue as `Won't Do` because of SPARK-31147. CHAR is not support
mainly.
> PostgreSQL dialect: cast to char
>
>
> Key: SPARK-29846
> URL: https://issues.apache.org/jira/browse/SPARK-29846
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: jobit mathew
>Priority: Minor
>
> Make SparkSQL's cast to char behavior be consistent with PostgreSQL when
> spark.sql.dialect is configured as PostgreSQL.
> {code:java}
> spark-sql> select cast ('10.22333' as
> char(5));
> 10.22333
> Time taken: 0.062 seconds, Fetched 1 row(s)
> spark-sql>
> {code}
> *postgresql*
> select cast ('10.22333' as char(5));
>
> || ||bpchar||
> |1|10.22|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28329) SELECT INTO syntax
[
https://issues.apache.org/jira/browse/SPARK-28329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28329:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> SELECT INTO syntax
> --
>
> Key: SPARK-28329
> URL: https://issues.apache.org/jira/browse/SPARK-28329
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> h2. Synopsis
> {noformat}
> [ WITH [ RECURSIVE ] with_query [, ...] ]
> SELECT [ ALL | DISTINCT [ ON ( expression [, ...] ) ] ]
> * | expression [ [ AS ] output_name ] [, ...]
> INTO [ TEMPORARY | TEMP | UNLOGGED ] [ TABLE ] new_table
> [ FROM from_item [, ...] ]
> [ WHERE condition ]
> [ GROUP BY expression [, ...] ]
> [ HAVING condition [, ...] ]
> [ WINDOW window_name AS ( window_definition ) [, ...] ]
> [ { UNION | INTERSECT | EXCEPT } [ ALL | DISTINCT ] select ]
> [ ORDER BY expression [ ASC | DESC | USING operator ] [ NULLS { FIRST |
> LAST } ] [, ...] ]
> [ LIMIT { count | ALL } ]
> [ OFFSET start [ ROW | ROWS ] ]
> [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
> [ FOR { UPDATE | SHARE } [ OF table_name [, ...] ] [ NOWAIT ] [...] ]
> {noformat}
> h2. Description
> {{SELECT INTO}} creates a new table and fills it with data computed by a
> query. The data is not returned to the client, as it is with a normal
> {{SELECT}}. The new table's columns have the names and data types associated
> with the output columns of the {{SELECT}}.
>
> {{CREATE TABLE AS}} offers a superset of the functionality offered by
> {{SELECT INTO}}.
> [https://www.postgresql.org/docs/11/sql-selectinto.html]
> [https://www.postgresql.org/docs/11/sql-createtableas.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28377) Fully support correlation names in the FROM clause
[
https://issues.apache.org/jira/browse/SPARK-28377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28377:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Fully support correlation names in the FROM clause
> --
>
> Key: SPARK-28377
> URL: https://issues.apache.org/jira/browse/SPARK-28377
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> Specifying a list of column names is not fully support. Example:
> {code:sql}
> create or replace temporary view J1_TBL as select * from
> (values (1, 4, 'one'), (2, 3, 'two'))
> as v(i, j, t);
> create or replace temporary view J2_TBL as select * from
> (values (1, -1), (2, 2))
> as v(i, k);
> SELECT '' AS xxx, t1.a, t2.e
> FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e)
> WHERE t1.a = t2.d;
> {code}
> PostgreSQL:
> {noformat}
> postgres=# SELECT '' AS xxx, t1.a, t2.e
> postgres-# FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e)
> postgres-# WHERE t1.a = t2.d;
> xxx | a | e
> -+---+
> | 1 | -1
> | 2 | 2
> (2 rows)
> {noformat}
> Spark SQL:
> {noformat}
> spark-sql> SELECT '' AS xxx, t1.a, t2.e
> > FROM J1_TBL t1 (a, b, c), J2_TBL t2 (d, e)
> > WHERE t1.a = t2.d;
> Error in query: cannot resolve '`t1.a`' given input columns: [a, b, c, d, e];
> line 3 pos 8;
> 'Project [ AS xxx#21, 't1.a, 't2.e]
> +- 'Filter ('t1.a = 't2.d)
>+- Join Inner
> :- Project [i#14 AS a#22, j#15 AS b#23, t#16 AS c#24]
> : +- SubqueryAlias `t1`
> : +- SubqueryAlias `j1_tbl`
> :+- Project [i#14, j#15, t#16]
> : +- Project [col1#11 AS i#14, col2#12 AS j#15, col3#13 AS
> t#16]
> : +- SubqueryAlias `v`
> : +- LocalRelation [col1#11, col2#12, col3#13]
> +- Project [i#19 AS d#25, k#20 AS e#26]
> +- SubqueryAlias `t2`
> +- SubqueryAlias `j2_tbl`
>+- Project [i#19, k#20]
> +- Project [col1#17 AS i#19, col2#18 AS k#20]
> +- SubqueryAlias `v`
> +- LocalRelation [col1#17, col2#18]
> {noformat}
>
> *Feature ID*: E051-08
> [https://www.postgresql.org/docs/11/sql-expressions.html]
> [https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/sqlref/src/tpc/db2z_correlationnames.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30472) [SQL] ANSI SQL: Throw exception on format invalid and overflow when casting String to IntegerType.
[ https://issues.apache.org/jira/browse/SPARK-30472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30472: -- Affects Version/s: (was: 3.0.0) 3.1.0 > [SQL] ANSI SQL: Throw exception on format invalid and overflow when casting > String to IntegerType. > -- > > Key: SPARK-30472 > URL: https://issues.apache.org/jira/browse/SPARK-30472 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: feiwang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28328) Support ANSI SQL:member predicate
[
https://issues.apache.org/jira/browse/SPARK-28328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28328:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support ANSI SQL:member predicate
> -
>
> Key: SPARK-28328
> URL: https://issues.apache.org/jira/browse/SPARK-28328
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> {code:java}
> ::=
>
> ::=
> [ NOT ] MEMBER [ OF ] {code}
> ANSI E061
> Oracle:[https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/Multiset-Conditions.html#GUID-228D6708-37E1-4C54-8715-7EC2CF5B5998]
> Postgresql:[https://www.postgresql.org/docs/9.1/features-sql-standard.html]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30220) Support Filter expression uses IN/EXISTS predicate sub-queries
[
https://issues.apache.org/jira/browse/SPARK-30220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30220:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support Filter expression uses IN/EXISTS predicate sub-queries
> --
>
> Key: SPARK-30220
> URL: https://issues.apache.org/jira/browse/SPARK-30220
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> Spark SQL cannot supports a SQL with nested aggregate as below:
>
> {code:java}
> select sum(unique1) FILTER (WHERE
> unique1 IN (SELECT unique1 FROM onek where unique1 < 100)) FROM tenk1;{code}
>
> And Spark will throw exception as follows:
>
> {code:java}
> org.apache.spark.sql.AnalysisException
> IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few
> commands: Aggregate [sum(cast(unique1#x as bigint)) AS sum(unique1)#xL]
> : +- Project [unique1#x]
> : +- Filter (unique1#x < 100)
> : +- SubqueryAlias `onek`
> : +- RelationV2[unique1#x, unique2#x, two#x, four#x, ten#x, twenty#x,
> hundred#x, thousand#x, twothousand#x, fivethous#x, tenthous#x, odd#x, even#x,
> stringu1#x, stringu2#x, string4#x] csv
> file:/home/xitong/code/gengjiaan/spark/sql/core/target/scala-2.12/test-classes/test-data/postgresql/onek.data
> +- SubqueryAlias `tenk1`
> +- RelationV2[unique1#x, unique2#x, two#x, four#x, ten#x, twenty#x,
> hundred#x, thousand#x, twothousand#x, fivethous#x, tenthous#x, odd#x, even#x,
> stringu1#x, stringu2#x, string4#x] csv
> file:/home/xitong/code/gengjiaan/spark/sql/core/target/scala-2.12/test-classes/test-data/postgresql/tenk.data{code}
>
> But PostgreSQL supports this syntax.
> {code:java}
> select sum(unique1) FILTER (WHERE
> unique1 IN (SELECT unique1 FROM onek where unique1 < 100)) FROM tenk1;
> sum
> --
> 4950
> (1 row){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30219) Support Filter expression reference the outer query
[
https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30219:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support Filter expression reference the outer query
> ---
>
> Key: SPARK-30219
> URL: https://issues.apache.org/jira/browse/SPARK-30219
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> Spark SQL cannot supports a SQL with nested aggregate as below:
>
> {code:java}
> select (select count(*) filter (where outer_c <> 0)
> from (values (1)) t0(inner_c))
> from (values (2),(3)) t1(outer_c);{code}
>
> And Spark will throw exception as follows:
> {code:java}
> org.apache.spark.sql.AnalysisException
> Expressions referencing the outer query are not supported outside of
> WHERE/HAVING clauses:
> Aggregate [count(1) AS count(1)#xL]
> +- Project [col1#x AS inner_c#x]
> +- SubqueryAlias `t0`
> +- LocalRelation [col1#x]{code}
> But PostgreSQL supports this syntax.
>
> {code:java}
> select (select count(*) filter (where outer_c <> 0)
> from (values (1)) t0(inner_c))
> from (values (2),(3)) t1(outer_c); -- outer query is aggregation query
> count
> ---
> 2
> (1 row){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30276) Support Filter expression allows simultaneous use of DISTINCT
[
https://issues.apache.org/jira/browse/SPARK-30276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30276:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support Filter expression allows simultaneous use of DISTINCT
> -
>
> Key: SPARK-30276
> URL: https://issues.apache.org/jira/browse/SPARK-30276
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> SPARK-27986 only supports Filter expression without DISTINCT.
> We need to support Filter expression allow simultaneous use of DISTINCT
> PostgreSQL support:
> {code:java}
> select ten, sum(distinct four) filter (where four > 10) from onek group by
> ten;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28330) ANSI SQL: Top-level in
[
https://issues.apache.org/jira/browse/SPARK-28330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28330:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> ANSI SQL: Top-level in
>
>
> Key: SPARK-28330
> URL: https://issues.apache.org/jira/browse/SPARK-28330
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> h2. {{LIMIT}} and {{OFFSET}}
> LIMIT and OFFSET allow you to retrieve just a portion of the rows that are
> generated by the rest of the query:
> {noformat}
> SELECT select_list
> FROM table_expression
> [ ORDER BY ... ]
> [ LIMIT { number | ALL } ] [ OFFSET number ]
> {noformat}
> If a limit count is given, no more than that many rows will be returned (but
> possibly fewer, if the query itself yields fewer rows). LIMIT ALL is the same
> as omitting the LIMIT clause, as is LIMIT with a NULL argument.
> OFFSET says to skip that many rows before beginning to return rows. OFFSET 0
> is the same as omitting the OFFSET clause, as is OFFSET with a NULL argument.
> If both OFFSET and LIMIT appear, then OFFSET rows are skipped before starting
> to count the LIMIT rows that are returned.
> https://www.postgresql.org/docs/11/queries-limit.html
> *Feature ID*: F861
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28447) ANSI SQL: Unicode escapes in literals
[ https://issues.apache.org/jira/browse/SPARK-28447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28447: -- Affects Version/s: (was: 3.0.0) 3.1.0 > ANSI SQL: Unicode escapes in literals > - > > Key: SPARK-28447 > URL: https://issues.apache.org/jira/browse/SPARK-28447 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Yuming Wang >Priority: Major > > [https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/strings.sql#L19-L44] > > *Feature ID*: F393 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28580) ANSI SQL: unique predicate
[
https://issues.apache.org/jira/browse/SPARK-28580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28580:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> ANSI SQL: unique predicate
> --
>
> Key: SPARK-28580
> URL: https://issues.apache.org/jira/browse/SPARK-28580
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> Format
> {code:java}
> ::=
> UNIQUE {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24497) ANSI SQL: Recursive query
[
https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-24497:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> ANSI SQL: Recursive query
> -
>
> Key: SPARK-24497
> URL: https://issues.apache.org/jira/browse/SPARK-24497
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> h3. *Examples*
> Here is an example for {{WITH RECURSIVE}} clause usage. Table "department"
> represents the structure of an organization as an adjacency list.
> {code:sql}
> CREATE TABLE department (
> id INTEGER PRIMARY KEY, -- department ID
> parent_department INTEGER REFERENCES department, -- upper department ID
> name TEXT -- department name
> );
> INSERT INTO department (id, parent_department, "name")
> VALUES
> (0, NULL, 'ROOT'),
> (1, 0, 'A'),
> (2, 1, 'B'),
> (3, 2, 'C'),
> (4, 2, 'D'),
> (5, 0, 'E'),
> (6, 4, 'F'),
> (7, 5, 'G');
> -- department structure represented here is as follows:
> --
> -- ROOT-+->A-+->B-+->C
> -- | |
> -- | +->D-+->F
> -- +->E-+->G
> {code}
>
> To extract all departments under A, you can use the following recursive
> query:
> {code:sql}
> WITH RECURSIVE subdepartment AS
> (
> -- non-recursive term
> SELECT * FROM department WHERE name = 'A'
> UNION ALL
> -- recursive term
> SELECT d.*
> FROM
> department AS d
> JOIN
> subdepartment AS sd
> ON (d.parent_department = sd.id)
> )
> SELECT *
> FROM subdepartment
> ORDER BY name;
> {code}
> More details:
> [http://wiki.postgresql.org/wiki/CTEReadme]
> [https://info.teradata.com/htmlpubs/DB_TTU_16_00/index.html#page/SQL_Reference/B035-1141-160K/lqe1472241402390.html]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29098) Test both ANSI mode and Spark mode
[ https://issues.apache.org/jira/browse/SPARK-29098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29098: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Test both ANSI mode and Spark mode > -- > > Key: SPARK-29098 > URL: https://issues.apache.org/jira/browse/SPARK-29098 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Gengliang Wang >Priority: Major > > The PostgreSQL test case improves the test coverage of Spark SQL. > There are SQL files that have different results with/without ANSI > flags(spark.sql.failOnIntegralTypeOverflow, spark.sql.parser.ansi.enabled, > etc) enabled. > We should run tests against these SQL files with both ANSI mode and Spark > mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27148) Support CURRENT_TIME and LOCALTIME when ANSI mode enabled
[
https://issues.apache.org/jira/browse/SPARK-27148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-27148:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support CURRENT_TIME and LOCALTIME when ANSI mode enabled
> -
>
> Key: SPARK-27148
> URL: https://issues.apache.org/jira/browse/SPARK-27148
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Takeshi Yamamuro
>Priority: Major
>
> CURRENT_TIME and LOCALTIME should be supported in the ANSI standard;
> {code:java}
> postgres=# select CURRENT_TIME;
> timetz
>
> 16:45:43.398109+09
> (1 row)
> postgres=# select LOCALTIME;
> time
>
> 16:45:48.60969
> (1 row){code}
> Before this, we need to support TIME types (java.sql.Time).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28682) ANSI SQL: Collation Support
[ https://issues.apache.org/jira/browse/SPARK-28682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28682: -- Affects Version/s: (was: 3.0.0) 3.1.0 > ANSI SQL: Collation Support > --- > > Key: SPARK-28682 > URL: https://issues.apache.org/jira/browse/SPARK-28682 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Yuming Wang >Priority: Major > > The collation feature allows specifying the sort order and character > classification behavior of data per-column, or even per-operation. This > alleviates the restriction that the LC_COLLATE and LC_CTYPE settings of a > database cannot be changed after its creation. > https://www.postgresql.org/docs/11/collation.html > *Implied Feature ID*: F690 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28325) Support ANSI SQL:SIMILAR TO ... ESCAPE syntax
[
https://issues.apache.org/jira/browse/SPARK-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28325:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support ANSI SQL:SIMILAR TO ... ESCAPE syntax
> -
>
> Key: SPARK-28325
> URL: https://issues.apache.org/jira/browse/SPARK-28325
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: jiaan.geng
>Priority: Major
>
> {code:java}
> ::=
>
> ::=
> [ NOT ] SIMILAR TO [ ESCAPE ]
> ::=
>
> ::=
>
> |
> ::=
>
> |
> ::=
>
> |
> |
> |
> |
> ::=
> [ ]
> ::=
> [ ]
> ::=
>
> ::=
>
> ::=
>
> |
> |
> |
> ::=
>
> |
> ::=
> !! See the Syntax Rules.
> 494 Foundation (SQL/Foundation)
> CD 9075-2:201?(E)
> 8.6
> ::=
> !! See the Syntax Rules.
> ::=
>
> | ...
> | ...
> | ...
> ...
> ::=
>
> ::=
>
> ::=
>
> |
> | bracket>
> ::=
> {code}
>
> Examples:
> {code}
> SELECT 'abc' RLIKE '%(b|d)%'; // false
> SELECT 'abc' SIMILAR TO '%(b|d)%' // true
> SELECT 'abc' RLIKE '(b|c)%'; // false
> SELECT 'abc' SIMILAR TO '(b|c)%'; // false{code}
>
> Currently, the following DBMSs support the syntax:
> *
> PostgreSQL:[https://www.postgresql.org/docs/current/functions-matching.html#FUNCTIONS-SIMILARTO-REGEXP]
> * Redshift:
> [https://docs.aws.amazon.com/redshift/latest/dg/pattern-matching-conditions-similar-to.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29631) Support ANSI SQL CREATE SEQUENCE
[
https://issues.apache.org/jira/browse/SPARK-29631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29631:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support ANSI SQL CREATE SEQUENCE
> -
>
> Key: SPARK-29631
> URL: https://issues.apache.org/jira/browse/SPARK-29631
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Takeshi Yamamuro
>Priority: Major
>
> {code}
> CREATE SEQUENCE seq1;
> CREATE TEMPORARY SEQUENCE seq1_temp;
> CREATE VIEW v9 AS SELECT seq1.is_called FROM seq1;
> CREATE VIEW v13_temp AS SELECT seq1_temp.is_called FROM seq1_temp;
> {code}
> https://www.postgresql.org/docs/current/sql-createsequence.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27951) ANSI SQL: NTH_VALUE function
[
https://issues.apache.org/jira/browse/SPARK-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-27951:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> ANSI SQL: NTH_VALUE function
>
>
> Key: SPARK-27951
> URL: https://issues.apache.org/jira/browse/SPARK-27951
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Zhu, Lipeng
>Priority: Major
>
> |{{nth_value({{value}}{{any}}, {{nth}}{{integer}})}}|{{same type as
> }}{{value}}|returns {{value}} evaluated at the row that is the {{nth}} row of
> the window frame (counting from 1); null if no such row|
> [https://www.postgresql.org/docs/8.4/functions-window.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30070) ANSI-F053: Overlaps datetimes predicate support
[
https://issues.apache.org/jira/browse/SPARK-30070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30070:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> ANSI-F053: Overlaps datetimes predicate support
> ---
>
> Key: SPARK-30070
> URL: https://issues.apache.org/jira/browse/SPARK-30070
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Kent Yao
>Priority: Major
>
> {code:java}
> * The operator `OVERLAPS` determines whether or not two chronological
> periods overlap in time. A
> * chronological period is specified by a pair of datetimes (starting and
> ending).
> *
> * If the length of the period is greater than 0, then the period consists of
> all points of time
> * greater than or equal to the lower endpoint, and less than the upper
> endpoint,
> * a.k.a [lower, upper).
> *
> * If the length of the period is equal to 0, then the period consists of a
> single point in time,
> * the lower endpoint, a.k.a [lower, lower].
> *
> * Two periods overlap if they have at least one point in common.
> {code}
> PostgreSQL
> {code:sql}
> postgres=# select (cast(a as timestamp), cast(b as timestamp)) overlaps
> (cast(c as timestamp), cast(d as timestamp)) from (values
> ('2011-11-11', '2011-11-11', '2011-11-11', '2011-11-11'),
> ('2011-11-10', '2011-11-11', '2011-11-11', '2011-11-12'),
> ('2011-11-11', '2011-11-10', '2011-11-11', '2011-11-12'),
> ('2011-11-11', '2011-11-10', '2011-11-12', '2011-11-11'),
> ('2011-11-10', '2011-11-11', '2011-11-12', '2011-11-13'),
> ('2011-11-10', '2011-11-20', '2011-11-11', '2011-11-19'),
> ('2011-11-11', '2011-11-19', '2011-11-10', '2011-11-20')) t(a,b,c,d);
> overlaps
> --
> t
> f
> f
> f
> f
> t
> t
> (7 rows)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29395) Precision of the interval type
[
https://issues.apache.org/jira/browse/SPARK-29395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29395:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Precision of the interval type
> --
>
> Key: SPARK-29395
> URL: https://issues.apache.org/jira/browse/SPARK-29395
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> PostgreSQL allows to specify interval precision, see
> [https://www.postgresql.org/docs/12/datatype-datetime.html]
> |{{interval [ _{{fields}}_ ] [ (_{{p}}_) ]}}|16 bytes|time
> interval|-17800 years|17800 years|1 microsecond|
> For example:
> {code}
> maxim=# SELECT interval '1 2:03.4567' day to second(2);
> interval
> ---
> 1 day 00:02:03.46
> (1 row)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29407) Support syntax for zero interval
[
https://issues.apache.org/jira/browse/SPARK-29407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29407:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support syntax for zero interval
>
>
> Key: SPARK-29407
> URL: https://issues.apache.org/jira/browse/SPARK-29407
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> Support special syntax for zero interval like PostgreSQL does:
> {code}
> maxim=# SELECT interval '0';
> interval
> --
> 00:00:00
> (1 row)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29524) Unordered interval units
[
https://issues.apache.org/jira/browse/SPARK-29524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29524:
--
Affects Version/s: (was: 2.4.4)
3.1.0
> Unordered interval units
>
>
> Key: SPARK-29524
> URL: https://issues.apache.org/jira/browse/SPARK-29524
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> Currently, Spark requires particular order of interval units in casting from
> strings - `YEAR` .. `MICROSECOND`. PostgreSQL allows any order:
> {code}
> maxim=# select interval '1 second 2 hours';
> interval
> --
> 02:00:01
> (1 row)
> {code}
> but Spark fails on while parsing:
> {code}
> spark-sql> select interval '1 second 2 hours';
> NULL
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29383) Support the optional prefix `@` in interval strings
[
https://issues.apache.org/jira/browse/SPARK-29383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29383:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support the optional prefix `@` in interval strings
> ---
>
> Key: SPARK-29383
> URL: https://issues.apache.org/jira/browse/SPARK-29383
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> PostgreSQL allows `@` at the beginning and `ago` at the end of interval
> strings:
> {code}
> maxim=# select interval '@ 14 seconds';
> interval
> --
> 00:00:14
> {code}
> See
> https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-INTERVAL-INPUT
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29384) Support `ago` in interval strings
[
https://issues.apache.org/jira/browse/SPARK-29384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29384:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support `ago` in interval strings
> -
>
> Key: SPARK-29384
> URL: https://issues.apache.org/jira/browse/SPARK-29384
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> PostgreSQL allow to specify direction in interval string by the `ago` word:
> {code}
> maxim=# select interval '@ 1 year 2 months 3 days 14 seconds ago';
> interval
>
> -1 years -2 mons -3 days -00:00:14
> {code}
> See
> https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-INTERVAL-INPUT
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29389) Support synonyms for interval units
[
https://issues.apache.org/jira/browse/SPARK-29389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29389:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support synonyms for interval units
> ---
>
> Key: SPARK-29389
> URL: https://issues.apache.org/jira/browse/SPARK-29389
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> Should be supported the following synonyms:
> {code}
> ["MILLENNIUM", ("MILLENNIA", "MIL", "MILS"),
>"CENTURY", ("CENTURIES", "C", "CENT"),
>"DECADE", ("DECADES", "DEC", "DECS"),
>"YEAR", ("Y", "YEARS", "YR", "YRS"),
>"QUARTER", ("QTR"),
>"MONTH", ("MON", "MONS", "MONTHS"),
>"DAY", ("D", "DAYS"),
>"HOUR", ("H", "HOURS", "HR", "HRS"),
>"MINUTE", ("M", "MIN", "MINS", "MINUTES"),
>"SECOND", ("S", "SEC", "SECONDS", "SECS"),
>"MILLISECONDS", ("MSEC", "MSECS", "MILLISECON",
> "MSECONDS", "MS"),
>"MICROSECONDS", ("USEC", "USECS", "USECONDS",
> "MICROSECON", "US"),
>"EPOCH"]
> {code}
> For example:
> {code}
> maxim=# select '1y 10mon -10d -10h -10min -10.01s
> ago'::interval;
> interval
>
> -1 years -10 mons +10 days 10:10:10.01
> (1 row)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29024) Ignore case while resolving time zones
[
https://issues.apache.org/jira/browse/SPARK-29024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29024:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Ignore case while resolving time zones
> --
>
> Key: SPARK-29024
> URL: https://issues.apache.org/jira/browse/SPARK-29024
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> The `zulu` time zone is checked by
> https://github.com/apache/spark/blob/67b4329fb08fd606461aa1ac9274c4a84d15d70e/sql/core/src/test/resources/sql-tests/inputs/pgSQL/timestamp.sql#L31
> but `getZoneId` fails on resolving it:
> {code}
> scala> getZoneId("zulu")
> java.time.zone.ZoneRulesException: Unknown time-zone ID: zulu
> at java.time.zone.ZoneRulesProvider.getProvider(ZoneRulesProvider.java:272)
> at java.time.zone.ZoneRulesProvider.getRules(ZoneRulesProvider.java:227)
> at java.time.ZoneRegion.ofId(ZoneRegion.java:120)
> at java.time.ZoneId.of(ZoneId.java:411)
> at java.time.ZoneId.of(ZoneId.java:359)
> at java.time.ZoneId.of(ZoneId.java:315)
> at
> org.apache.spark.sql.catalyst.util.DateTimeUtils$.getZoneId(DateTimeUtils.scala:77)
> ... 49 elided
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29406) Interval output styles
[
https://issues.apache.org/jira/browse/SPARK-29406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29406:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Interval output styles
> --
>
> Key: SPARK-29406
> URL: https://issues.apache.org/jira/browse/SPARK-29406
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> The output format of the interval type can be set to one of the four styles
> sql_standard, postgres, postgres_verbose, or iso_8601, using the command SET
> intervalstyle, see
>
> [https://www.postgresql.org/docs/11/datatype-datetime.html#DATATYPE-INTERVAL-OUTPUT]
> ||Style Specification||Year-Month Interval||Day-Time Interval||Mixed
> Interval||
> |{{sql_standard}}|1-2|3 4:05:06|-1-2 +3 -4:05:06|
> |{{postgres}}|1 year 2 mons|3 days 04:05:06|-1 year -2 mons +3 days -04:05:06|
> |{{postgres_verbose}}|@ 1 year 2 mons|@ 3 days 4 hours 5 mins 6 secs|@ 1 year
> 2 mons -3 days 4 hours 5 mins 6 secs ago|
> |{{iso_8601}}|P1Y2M|P3DT4H5M6S|P-1Y-2M3DT-4H-5M-6S|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29394) Support ISO 8601 format for intervals
[ https://issues.apache.org/jira/browse/SPARK-29394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29394: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Support ISO 8601 format for intervals > - > > Key: SPARK-29394 > URL: https://issues.apache.org/jira/browse/SPARK-29394 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Maxim Gekk >Priority: Major > > Interval values can also be written as ISO 8601 time intervals, using either > the “format with designators” of the standard's section 4.4.3.2 or the > “alternative format” of section 4.4.3.3. > For example: > |P1Y2M3DT4H5M6S|ISO 8601 “format with designators”| > |P0001-02-03T04:05:06|ISO 8601 “alternative format”: same meaning as above| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29370) Interval strings without explicit unit markings
[
https://issues.apache.org/jira/browse/SPARK-29370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29370:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Interval strings without explicit unit markings
> ---
>
> Key: SPARK-29370
> URL: https://issues.apache.org/jira/browse/SPARK-29370
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> In PostgreSQL, Quantities of days, hours, minutes, and seconds can be
> specified without explicit unit markings. For example, '1 12:59:10' is read
> the same as '1 day 12 hours 59 min 10 sec'. For example:
> {code:java}
> maxim=# select interval '1 12:59:10';
> interval
>
> 1 day 12:59:10
> (1 row)
> {code}
> It should allow to specify the sign:
> {code}
> maxim=# SELECT interval '1 +2:03:04' minute to second;
> interval
>
> 1 day 02:03:04
> maxim=# SELECT interval '1 -2:03:04' minute to second;
> interval
> -
> 1 day -02:03:04
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29388) Construct intervals from the `millenniums`, `centuries` or `decades` units
[
https://issues.apache.org/jira/browse/SPARK-29388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29388:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Construct intervals from the `millenniums`, `centuries` or `decades` units
> --
>
> Key: SPARK-29388
> URL: https://issues.apache.org/jira/browse/SPARK-29388
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> PostgreSQL supports `millenniums`, `centuries` or `decades` interval units.
> See
> {code}
> maxim=# select '4 millenniums 5 centuries 4 decades 1 year 4 months 4 days 17
> minutes 31 seconds'::interval;
> interval
> ---
> 4541 years 4 mons 4 days 00:17:31
> (1 row)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29006) Support special date/timestamp values `infinity`/`-infinity`
[
https://issues.apache.org/jira/browse/SPARK-29006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29006:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support special date/timestamp values `infinity`/`-infinity`
>
>
> Key: SPARK-29006
> URL: https://issues.apache.org/jira/browse/SPARK-29006
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> ||Input String||Valid Types||Description||
> |{{infinity}}|{{date}}, {{timestamp}}|later than all other time stamps|
> |{{-infinity}}|{{date}}, {{timestamp}}|earlier than all other time stamps|
> https://www.postgresql.org/docs/12/datatype-datetime.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28973) Add TimeType to Catalyst
[ https://issues.apache.org/jira/browse/SPARK-28973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28973: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Add TimeType to Catalyst > > > Key: SPARK-28973 > URL: https://issues.apache.org/jira/browse/SPARK-28973 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Maxim Gekk >Priority: Major > > The time type should represent local time in microsecond precision with valid > range of values [00:00:00.00, 23:59:59.99]. Internally, time can be > stored as number of microseconds since 00:00:00.00. > Support `java.time.LocalTime` as the external type for Catalyst's TimeType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-28973) Add TimeType to Catalyst
[ https://issues.apache.org/jira/browse/SPARK-28973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-28973. - > Add TimeType to Catalyst > > > Key: SPARK-28973 > URL: https://issues.apache.org/jira/browse/SPARK-28973 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Maxim Gekk >Priority: Major > > The time type should represent local time in microsecond precision with valid > range of values [00:00:00.00, 23:59:59.99]. Internally, time can be > stored as number of microseconds since 00:00:00.00. > Support `java.time.LocalTime` as the external type for Catalyst's TimeType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28973) Add TimeType to Catalyst
[ https://issues.apache.org/jira/browse/SPARK-28973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28973. --- Resolution: Won't Do According to the discussion on the PR, this issue is closed as 'Won't Do'. > Add TimeType to Catalyst > > > Key: SPARK-28973 > URL: https://issues.apache.org/jira/browse/SPARK-28973 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Maxim Gekk >Priority: Major > > The time type should represent local time in microsecond precision with valid > range of values [00:00:00.00, 23:59:59.99]. Internally, time can be > stored as number of microseconds since 00:00:00.00. > Support `java.time.LocalTime` as the external type for Catalyst's TimeType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29034) String Constants with C-style Escapes
[
https://issues.apache.org/jira/browse/SPARK-29034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29034:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> String Constants with C-style Escapes
> -
>
> Key: SPARK-29034
> URL: https://issues.apache.org/jira/browse/SPARK-29034
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> PostgreSQL also accepts "escape" string constants, which are an extension to
> the SQL standard. An escape string constant is specified by writing the
> letter {{E}} (upper or lower case) just before the opening single quote,
> e.g., {{E'foo'}}. (When continuing an escape string constant across lines,
> write {{E}} only before the first opening quote.) Within an escape string, a
> backslash character ({{\}}) begins a C-like _backslash escape_ sequence, in
> which the combination of backslash and following character(s) represent a
> special byte value, as shown in [Table
> 4-1|https://www.postgresql.org/docs/9.3/sql-syntax-lexical.html#SQL-BACKSLASH-TABLE].
> *Table 4-1. Backslash Escape Sequences*
> ||Backslash Escape Sequence||Interpretation||
> |{{\b}}|backspace|
> |{{\f}}|form feed|
> |{{\n}}|newline|
> |{{\r}}|carriage return|
> |{{\t}}|tab|
> |{{\}}{{o}}, {{\}}{{oo}}, {{\}}{{ooo}} ({{o}} = 0 - 7)|octal byte value|
> |{{\x}}{{h}}, {{\x}}{{hh}} ({{h}} = 0 - 9, A - F)|hexadecimal byte value|
> |{{\u}}{{}}, {{\U}}{{}} ({{x}} = 0 - 9, A - F)|16 or 32-bit
> hexadecimal Unicode character value|
> Any other character following a backslash is taken literally. Thus, to
> include a backslash character, write two backslashes ({{\\}}). Also, a single
> quote can be included in an escape string by writing {{\'}}, in addition to
> the normal way of {{''}}.
> It is your responsibility that the byte sequences you create, especially when
> using the octal or hexadecimal escapes, compose valid characters in the
> server character set encoding. When the server encoding is UTF-8, then the
> Unicode escapes or the alternative Unicode escape syntax, explained in
> [Section
> 4.1.2.3|https://www.postgresql.org/docs/9.3/sql-syntax-lexical.html#SQL-SYNTAX-STRINGS-UESCAPE],
> should be used instead. (The alternative would be doing the UTF-8 encoding
> by hand and writing out the bytes, which would be very cumbersome.)
> The Unicode escape syntax works fully only when the server encoding is
> {{UTF8}}. When other server encodings are used, only code points in the ASCII
> range (up to {{\u007F}}) can be specified. Both the 4-digit and the 8-digit
> form can be used to specify UTF-16 surrogate pairs to compose characters with
> code points larger than U+, although the availability of the 8-digit form
> technically makes this unnecessary. (When surrogate pairs are used when the
> server encoding is {{UTF8}}, they are first combined into a single code point
> that is then encoded in UTF-8.)
>
>
> [https://www.postgresql.org/docs/11/sql-syntax-lexical.html#SQL-BACKSLASH-TABLE]
>
> Example:
> {code:sql}
> postgres=# SET bytea_output TO escape;
> SET
> postgres=# SELECT E'Th\\000omas'::bytea;
>bytea
>
> Th\000omas
> (1 row)
> postgres=# SELECT 'Th\\000omas'::bytea;
> bytea
> -
> Th\\000omas
> (1 row)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-29117) char data type should support in Spark
[ https://issues.apache.org/jira/browse/SPARK-29117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-29117. - > char data type should support in Spark > -- > > Key: SPARK-29117 > URL: https://issues.apache.org/jira/browse/SPARK-29117 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > > CREATE TABLE CHAR_TBL(f1 char); does not support in Spark but supports in > postgreSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28754) [UDF] Supports for alter, rename, owner change should be supported in Spark
[ https://issues.apache.org/jira/browse/SPARK-28754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28754: -- Affects Version/s: (was: 3.0.0) 3.1.0 > [UDF] Supports for alter, rename, owner change should be supported in Spark > --- > > Key: SPARK-28754 > URL: https://issues.apache.org/jira/browse/SPARK-28754 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > > Spark should support alter,rename or change owner of User defined function. > Currently Spark supports create or replace existing function but does not > allow to New Name or Owner change. > PstgreSQL supports these. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29117) char data type should support in Spark
[ https://issues.apache.org/jira/browse/SPARK-29117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29117. --- Resolution: Won't Do According to SPARK-31147, this issue is closed as 'Won't Do'. > char data type should support in Spark > -- > > Key: SPARK-29117 > URL: https://issues.apache.org/jira/browse/SPARK-29117 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > > CREATE TABLE CHAR_TBL(f1 char); does not support in Spark but supports in > postgreSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28664) ORDER BY in aggregate function
[
https://issues.apache.org/jira/browse/SPARK-28664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28664:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> ORDER BY in aggregate function
> --
>
> Key: SPARK-28664
> URL: https://issues.apache.org/jira/browse/SPARK-28664
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> {code:sql}
> SELECT min(x ORDER BY y) FROM (VALUES(1, NULL)) AS d(x,y);
> SELECT min(x ORDER BY y) FROM (VALUES(1, 2)) AS d(x,y);
> {code}
> https://github.com/postgres/postgres/blob/44e95b5728a4569c494fa4ea4317f8a2f50a206b/src/test/regress/sql/aggregates.sql#L978-L982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28646) Allow usage of `count` only for parameterless aggregate function
[
https://issues.apache.org/jira/browse/SPARK-28646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28646:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Allow usage of `count` only for parameterless aggregate function
>
>
> Key: SPARK-28646
> URL: https://issues.apache.org/jira/browse/SPARK-28646
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Currently, Spark allows calls to `count` even for non parameterless aggregate
> function. For example, the following query actually works:
> {code:sql}SELECT count() OVER () FROM tenk1;{code}
> In PgSQL, on the other hand, the following error is thrown:
> {code:sql}ERROR: count(*) must be used to call a parameterless aggregate
> function{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28508) Support for range frame+row frame in the same query
[
https://issues.apache.org/jira/browse/SPARK-28508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28508:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Support for range frame+row frame in the same query
> ---
>
> Key: SPARK-28508
> URL: https://issues.apache.org/jira/browse/SPARK-28508
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Currently, looks like some queries does not works if both, a range frame and
> a row frame are given. However, PgSQL is able to handle them:
> {code:sql}
> select last(salary) over(order by enroll_date range between 1 preceding and 1
> following), lag(salary) over(order by enroll_date range between 1 preceding
> and 1 following),
> salary, enroll_date from empsalary;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28557) Support empty select list
[
https://issues.apache.org/jira/browse/SPARK-28557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-28557.
---
Resolution: Duplicate
> Support empty select list
> -
>
> Key: SPARK-28557
> URL: https://issues.apache.org/jira/browse/SPARK-28557
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> Examples:
> {code:sql}
> select union select;
> select intersect select;
> select except select;
> {code}
> https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/union.sql#L193-L199
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28506) not handling usage of group function and window function at some conditions
[
https://issues.apache.org/jira/browse/SPARK-28506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28506:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> not handling usage of group function and window function at some conditions
> ---
>
> Key: SPARK-28506
> URL: https://issues.apache.org/jira/browse/SPARK-28506
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Hi,
> looks like SparkSQL is not able to handle this query:
> {code:sql}SELECT rank() OVER (ORDER BY 1), count(*) FROM empsalary GROUP BY
> 1;{code}
> PgSQL, on the other hand, does.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-28557) Support empty select list
[
https://issues.apache.org/jira/browse/SPARK-28557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-28557.
-
> Support empty select list
> -
>
> Key: SPARK-28557
> URL: https://issues.apache.org/jira/browse/SPARK-28557
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> Examples:
> {code:sql}
> select union select;
> select intersect select;
> select except select;
> {code}
> https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/union.sql#L193-L199
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28501) frame bound must be a literal
[
https://issues.apache.org/jira/browse/SPARK-28501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28501:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> frame bound must be a literal
> -
>
> Key: SPARK-28501
> URL: https://issues.apache.org/jira/browse/SPARK-28501
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> Spark frame bound currently only supports literals:
> {code:sql}
> SELECT sum(unique1) over (order by unique1 rows (SELECT unique1 FROM tenk1
> ORDER BY unique1 LIMIT 1) + 1 PRECEDING), unique1 FROM tenk1 WHERE unique1 <
> 10;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28437) Different format when casting interval type to string type
[
https://issues.apache.org/jira/browse/SPARK-28437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28437:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Different format when casting interval type to string type
> --
>
> Key: SPARK-28437
> URL: https://issues.apache.org/jira/browse/SPARK-28437
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> *Spark SQL*:
> {code:sql}
> spark-sql> select cast(INTERVAL '10' SECOND as string);
> interval 10 seconds
> {code}
> *PostgreSQL*:
> {code:sql}
> postgres=# select substr(version(), 0, 16), cast(INTERVAL '10' SECOND as
> text);
> substr | text
> -+--
> PostgreSQL 11.3 | 00:00:10
> (1 row)
> {code}
> *Vertica*:
> {code:sql}
> dbadmin=> select version(), cast(INTERVAL '10' SECOND as varchar(255));
> version | ?column?
> +--
> Vertica Analytic Database v9.1.1-0 | 10
> (1 row)
> {code}
> *Presto*:
> {code:sql}
> presto> select cast(INTERVAL '10' SECOND as varchar(255));
> _col0
>
> 0 00:00:10.000
> (1 row)
> {code}
> *Oracle*:
> {code:sql}
> SQL> select cast(INTERVAL '10' SECOND as varchar(255)) from dual;
> CAST(INTERVAL'10'SECONDASVARCHAR(255))
>
> INTERVAL'+00 00:00:10.00'DAY TO SECOND
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28405) Join with USING caluse doesn't hide original tables
[
https://issues.apache.org/jira/browse/SPARK-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28405:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Join with USING caluse doesn't hide original tables
> ---
>
> Key: SPARK-28405
> URL: https://issues.apache.org/jira/browse/SPARK-28405
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Peter Toth
>Priority: Major
>
> The following query doesn't work in SparkSQL as t1.* and t2.* can't be used
> after {{USING}} caluse.
> {noformat}
> select t1.*, t2.* from (values (1), (2)) t1(c) left join (values (1)) t2(c)
> using (c){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28409) SELECT FROM syntax
[
https://issues.apache.org/jira/browse/SPARK-28409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28409:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> SELECT FROM syntax
> --
>
> Key: SPARK-28409
> URL: https://issues.apache.org/jira/browse/SPARK-28409
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> Examples:
> {code:sql}
> create or replace temporary view INT4_TBL as select * from
> (values (0), (123456), (-123456), (2147483647), (-2147483647))
> as v(f1);
> SELECT FROM INT4_TBL;
> {code}
> https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/union.sql#L205-L231
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28428) Spark `exclude` always expecting `()`
[
https://issues.apache.org/jira/browse/SPARK-28428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-28428:
--
Affects Version/s: (was: 3.0.0)
3.1.0
> Spark `exclude` always expecting `()`
> --
>
> Key: SPARK-28428
> URL: https://issues.apache.org/jira/browse/SPARK-28428
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Tests
>Affects Versions: 3.1.0
>Reporter: Dylan Guedes
>Priority: Major
>
> SparkSQL `exclude` always expects a following call to `()`, however, PgSQL
> `exclude` does not. Examples:
> {code:sql}
> SELECT sum(unique1) over (rows between 2 preceding and 2 following exclude no
> others),
> unique1, four
> FROM tenk1 WHERE unique1 < 10;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
