[
https://issues.apache.org/jira/browse/SPARK-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li updated SPARK-12374:
Description:
Creating an actual logical/physical operator for range for matching the
performance of RDD Range APIs.
Compared with the old Range API, the new version is 3 times faster than the old
version.
{code}
scala> val startTime = System.currentTimeMillis; sqlContext.oldRange(0,
10, 1, 15).count(); val endTime = System.currentTimeMillis; val start =
new Timestamp(startTime); val end = new Timestamp(endTime); val elapsed =
(endTime - startTime)/ 1000.0
startTime: Long = 1450416394240
endTime: Long = 1450416421199
start: java.sql.Timestamp = 2015-12-17 21:26:34.24
end: java.sql.Timestamp = 2015-12-17 21:27:01.199
elapsed: Double = 26.959
{code}
{code}
scala> val startTime = System.currentTimeMillis; sqlContext.range(0,
10, 1, 15).count(); val endTime = System.currentTimeMillis; val start =
new Timestamp(startTime); val end = new Timestamp(endTime); val elapsed =
(endTime - startTime)/ 1000.0
startTime: Long = 1450416360107
endTime: Long = 1450416368590
start: java.sql.Timestamp = 2015-12-17 21:26:00.107
end: java.sql.Timestamp = 2015-12-17 21:26:08.59
elapsed: Double = 8.483
{code}
was:Creating an actual logical/physical operator for range for matching the
performance of RDD Range APIs.
> Improve performance of Range APIs via adding logical/physical operators
> ---
>
> Key: SPARK-12374
> URL: https://issues.apache.org/jira/browse/SPARK-12374
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Xiao Li
>Assignee: Apache Spark
>Priority: Critical
>
> Creating an actual logical/physical operator for range for matching the
> performance of RDD Range APIs.
> Compared with the old Range API, the new version is 3 times faster than the
> old version.
> {code}
> scala> val startTime = System.currentTimeMillis; sqlContext.oldRange(0,
> 10, 1, 15).count(); val endTime = System.currentTimeMillis; val start
> = new Timestamp(startTime); val end = new Timestamp(endTime); val elapsed =
> (endTime - startTime)/ 1000.0
> startTime: Long = 1450416394240
>
> endTime: Long = 1450416421199
> start: java.sql.Timestamp = 2015-12-17 21:26:34.24
> end: java.sql.Timestamp = 2015-12-17 21:27:01.199
> elapsed: Double = 26.959
> {code}
> {code}
> scala> val startTime = System.currentTimeMillis; sqlContext.range(0,
> 10, 1, 15).count(); val endTime = System.currentTimeMillis; val start
> = new Timestamp(startTime); val end = new Timestamp(endTime); val elapsed =
> (endTime - startTime)/ 1000.0
> startTime: Long = 1450416360107
>
> endTime: Long = 1450416368590
> start: java.sql.Timestamp = 2015-12-17 21:26:00.107
> end: java.sql.Timestamp = 2015-12-17 21:26:08.59
> elapsed: Double = 8.483
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
