[
https://issues.apache.org/jira/browse/SPARK-31517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ross Bowen updated SPARK-31517:
-------------------------------
Description:
When specifying two columns within an `orderBy()` function, to attempt to get
an ordering by two columns in descending order, an error is returned.
{code:java}
library(magrittr)
library(SparkR)
cars <- cbind(model = rownames(mtcars), mtcars)
carsDF <- createDataFrame(cars)
carsDF %>%
mutate(rank = over(rank(), orderBy(windowPartitionBy(column("cyl")),
desc(column("mpg")), desc(column("disp"))))) %>%
head() {code}
This returns an error:
{code:java}
Error in ns[[i]] : subscript out of bounds{code}
This seems to be related to the more general issue that the following code,
excluding the use of the `desc()` function also fails:
{code:java}
carsDF %>%
mutate(rank = over(rank(), orderBy(windowPartitionBy(column("cyl")),
column("mpg"), column("disp")))) %>%
head(){code}
was:
When specifying two columns within an `orderBy()` function, to attempt to get
an ordering by two columns in descending order, an error is returned.
{code:java}
library(magrittr) library(SparkR) cars <- cbind(model = rownames(mtcars),
mtcars) carsDF <- createDataFrame(cars) carsDF %>% mutate(rank = over(rank(),
orderBy(windowPartitionBy(column("cyl")), desc(column("mpg")),
desc(column("disp"))))) %>% head() {code}
This returns an error:
{code:java}
Error in ns[[i]] : subscript out of bounds{code}
This seems to be related to the more general issue that the following code,
excluding the use of the `desc()` function also fails:
{code:java}
carsDF %>% mutate(rank = over(rank(),
orderBy(windowPartitionBy(column("cyl")), column("mpg"), column("disp")))) %>%
head(){code}
> SparkR::orderBy with multiple columns descending produces error
> ---------------------------------------------------------------
>
> Key: SPARK-31517
> URL: https://issues.apache.org/jira/browse/SPARK-31517
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 2.4.5
> Environment: Databricks Runtime 6.5
> Reporter: Ross Bowen
> Priority: Major
>
> When specifying two columns within an `orderBy()` function, to attempt to get
> an ordering by two columns in descending order, an error is returned.
> {code:java}
> library(magrittr)
> library(SparkR)
> cars <- cbind(model = rownames(mtcars), mtcars)
> carsDF <- createDataFrame(cars)
> carsDF %>%
> mutate(rank = over(rank(), orderBy(windowPartitionBy(column("cyl")),
> desc(column("mpg")), desc(column("disp"))))) %>%
> head() {code}
> This returns an error:
> {code:java}
> Error in ns[[i]] : subscript out of bounds{code}
> This seems to be related to the more general issue that the following code,
> excluding the use of the `desc()` function also fails:
> {code:java}
> carsDF %>%
> mutate(rank = over(rank(), orderBy(windowPartitionBy(column("cyl")),
> column("mpg"), column("disp")))) %>%
> head(){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
