[spark] branch master updated (80f7989 -> 0ba1d38)
This is an automated email from the ASF dual-hosted git repository. viirya pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 80f7989 [SPARK-35734][SQL] Format day-time intervals using type fields add 0ba1d38 [SPARK-35701][SQL] Use copy-on-write semantics for SQLConf registered configurations No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/internal/SQLConf.scala| 98 ++ .../spark/sql/catalyst/plans/SQLHelper.scala | 2 +- .../scala/org/apache/spark/sql/RuntimeConfig.scala | 4 +- .../scala/org/apache/spark/sql/SparkSession.scala | 2 +- .../spark/sql/api/python/PythonSQLUtils.scala | 4 +- .../apache/spark/sql/internal/SharedState.scala| 2 +- 6 files changed, 72 insertions(+), 40 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-35734][SQL] Format day-time intervals using type fields
This is an automated email from the ASF dual-hosted git repository.
maxgekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 80f7989 [SPARK-35734][SQL] Format day-time intervals using type fields
80f7989 is described below
commit 80f7989d9a761eff1a0b3c64ec3aabb81506953d
Author: Kousuke Saruta
AuthorDate: Sat Jun 12 21:45:12 2021 +0300
[SPARK-35734][SQL] Format day-time intervals using type fields
### What changes were proposed in this pull request?
This PR add a feature which formats day-time interval to strings using the
start and end fields of `DayTimeIntervalType`.
### Why are the changes needed?
Currently, they are ignored, and any `DayTimeIntervalType` is formatted as
`INTERVAL DAY TO SECOND.`
### Does this PR introduce _any_ user-facing change?
Yes. The format of day-time intervals is determined the start and end
fields.
### How was this patch tested?
New test.
Closes #32891 from sarutak/interval-format.
Authored-by: Kousuke Saruta
Signed-off-by: Max Gekk
---
.../spark/sql/catalyst/util/IntervalUtils.scala| 58 +--
.../sql/catalyst/util/IntervalUtilsSuite.scala | 84 ++
2 files changed, 138 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
index c18cca9..dda5581 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
@@ -21,6 +21,7 @@ import java.time.{Duration, Period}
import java.time.temporal.ChronoUnit
import java.util.concurrent.TimeUnit
+import scala.collection.mutable
import scala.util.control.NonFatal
import org.apache.spark.sql.catalyst.util.DateTimeConstants._
@@ -28,7 +29,7 @@ import
org.apache.spark.sql.catalyst.util.DateTimeUtils.millisToMicros
import org.apache.spark.sql.catalyst.util.IntervalStringStyles.{ANSI_STYLE,
HIVE_STYLE, IntervalStyle}
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.Decimal
+import org.apache.spark.sql.types.{DayTimeIntervalType, Decimal}
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
// The style of textual representation of intervals
@@ -960,18 +961,34 @@ object IntervalUtils {
def toDayTimeIntervalString(
micros: Long,
style: IntervalStyle,
- // TODO(SPARK-35734): Format day-time intervals using type fields
startField: Byte,
endField: Byte): String = {
var sign = ""
var rest = micros
+val from = DayTimeIntervalType.fieldToString(startField).toUpperCase
+val to = DayTimeIntervalType.fieldToString(endField).toUpperCase
if (micros < 0) {
if (micros == Long.MinValue) {
// Especial handling of minimum `Long` value because negate op
overflows `Long`.
// seconds = 106751991 * (24 * 60 * 60) + 4 * 60 * 60 + 54 =
9223372036854
// microseconds = -922337203685400L-775808 == Long.MinValue
val minIntervalString = style match {
- case ANSI_STYLE => "INTERVAL '-106751991 04:00:54.775808' DAY TO
SECOND"
+ case ANSI_STYLE =>
+val baseStr = "-106751991 04:00:54.775808"
+val fromPos = startField match {
+ case DayTimeIntervalType.DAY => 0
+ case DayTimeIntervalType.HOUR => 11
+ case DayTimeIntervalType.MINUTE => 14
+ case DayTimeIntervalType.SECOND => 17
+}
+val toPos = endField match {
+ case DayTimeIntervalType.DAY => 10
+ case DayTimeIntervalType.HOUR => 13
+ case DayTimeIntervalType.MINUTE => 16
+ case DayTimeIntervalType.SECOND => baseStr.length
+}
+val postfix = if (startField == endField) from else s"$from TO $to"
+s"INTERVAL '${baseStr.substring(fromPos, toPos)}' $postfix"
case HIVE_STYLE => "-106751991 04:00:54.775808000"
}
return minIntervalString
@@ -992,7 +1009,40 @@ object IntervalUtils {
val secStr = java.math.BigDecimal.valueOf(secondsWithFraction, 6)
.stripTrailingZeros()
.toPlainString()
-f"INTERVAL '$sign$days $hours%02d:$minutes%02d:$leadSecZero$secStr'
DAY TO SECOND"
+val formatBuilder = new StringBuilder("INTERVAL '")
+if (startField == endField) {
+ startField match {
+case DayTimeIntervalType.DAY => formatBuilder.append(s"$sign$days'
")
+case DayTimeIntervalType.HOUR =>
formatBuilder.append(f"$hours%02d' ")
+case
[spark] branch branch-3.0 updated (6597c3b -> 3902af8)
This is an automated email from the ASF dual-hosted git repository. viirya pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 6597c3b [SPARK-35746][UI] Fix taskid in the stage page task event timeline add 3902af8 [SPARK-35689][SS][3.0] Add log warn when keyWithIndexToValue returns null value No new revisions were added by this update. Summary of changes: .../state/SymmetricHashJoinStateManager.scala | 12 ++- .../state/SymmetricHashJoinStateManagerSuite.scala | 24 ++ 2 files changed, 35 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-35746][UI] Fix taskid in the stage page task event timeline
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 6597c3b [SPARK-35746][UI] Fix taskid in the stage page task event
timeline
6597c3b is described below
commit 6597c3bd5e91040dc53576c912c85d84f630bb17
Author: shahid
AuthorDate: Sat Jun 12 15:38:41 2021 +0900
[SPARK-35746][UI] Fix taskid in the stage page task event timeline
### What changes were proposed in this pull request?
Task id is given incorrect in the timeline plot in Stage Page
### Why are the changes needed?
Map event timeline plots to correct task
**Before:**
**After**
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually tested
Closes #32888 from shahidki31/shahid/fixtaskid.
Authored-by: shahid
Signed-off-by: Kousuke Saruta
(cherry picked from commit 450b415028c3b00f3a002126cd11318d3932e28f)
Signed-off-by: Kousuke Saruta
---
core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
index ccaa70b..e9eb62e 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
@@ -352,7 +352,7 @@ private[ui] class StagePage(parent: StagesTab, store:
AppStatusStore) extends We
|'content': '
+ |data-title="${s"Task " + taskInfo.taskId + " (attempt " +
attempt + ")"}
|Status: ${taskInfo.status}
|Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
|${
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-35746][UI] Fix taskid in the stage page task event timeline
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 78d3d0f [SPARK-35746][UI] Fix taskid in the stage page task event
timeline
78d3d0f is described below
commit 78d3d0f0a562743bb9a36854c2302b242f4d9309
Author: shahid
AuthorDate: Sat Jun 12 15:38:41 2021 +0900
[SPARK-35746][UI] Fix taskid in the stage page task event timeline
### What changes were proposed in this pull request?
Task id is given incorrect in the timeline plot in Stage Page
### Why are the changes needed?
Map event timeline plots to correct task
**Before:**
**After**
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually tested
Closes #32888 from shahidki31/shahid/fixtaskid.
Authored-by: shahid
Signed-off-by: Kousuke Saruta
(cherry picked from commit 450b415028c3b00f3a002126cd11318d3932e28f)
Signed-off-by: Kousuke Saruta
---
core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
index 47ba951..459e09a 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
@@ -355,7 +355,7 @@ private[ui] class StagePage(parent: StagesTab, store:
AppStatusStore) extends We
|'content': '
+ |data-title="${s"Task " + taskInfo.taskId + " (attempt " +
attempt + ")"}
|Status: ${taskInfo.status}
|Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
|${
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-35746][UI] Fix taskid in the stage page task event timeline
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 450b415 [SPARK-35746][UI] Fix taskid in the stage page task event
timeline
450b415 is described below
commit 450b415028c3b00f3a002126cd11318d3932e28f
Author: shahid
AuthorDate: Sat Jun 12 15:38:41 2021 +0900
[SPARK-35746][UI] Fix taskid in the stage page task event timeline
### What changes were proposed in this pull request?
Task id is given incorrect in the timeline plot in Stage Page
### Why are the changes needed?
Map event timeline plots to correct task
**Before:**
**After**
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually tested
Closes #32888 from shahidki31/shahid/fixtaskid.
Authored-by: shahid
Signed-off-by: Kousuke Saruta
---
core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
index 777a6b0..81dfe83 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
@@ -355,7 +355,7 @@ private[ui] class StagePage(parent: StagesTab, store:
AppStatusStore) extends We
|'content': '
+ |data-title="${s"Task " + taskInfo.taskId + " (attempt " +
attempt + ")"}
|Status: ${taskInfo.status}
|Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
|${
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
